Implement adaptive scheduler with intelligent retry logic
- Replace fixed hourly schedule with adaptive scheduling system - Switch to 1-minute retries when no data is available from API - Return to hourly schedule once data is successfully fetched - Fix data fetching to use yesterday's date (API has 1-day delay) - Add comprehensive logging for scheduler mode changes - Improve resilience against API data availability issues The scheduler now intelligently adapts to data availability: - Normal mode: hourly runs at top of each hour - Retry mode: minute-based retries until data is available - Automatic mode switching based on fetch success/failure 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -407,9 +407,9 @@ class EnhancedWaterMonitorScraper:
|
||||
return None
|
||||
|
||||
def fetch_water_data(self) -> Optional[List[Dict]]:
|
||||
"""Fetch water levels and discharge data from API for current date"""
|
||||
current_date = datetime.datetime.now()
|
||||
return self.fetch_water_data_for_date(current_date)
|
||||
"""Fetch water levels and discharge data from API for yesterday (API has 1-day delay)"""
|
||||
yesterday = datetime.datetime.now() - datetime.timedelta(days=1)
|
||||
return self.fetch_water_data_for_date(yesterday)
|
||||
|
||||
def save_to_database(self, water_data: List[Dict], max_retries: int = 3) -> bool:
|
||||
"""Save water measurements to database with retry logic"""
|
||||
|
||||
Reference in New Issue
Block a user