feat: hourly HII/ThaiWater rainfall + backup water-level collection
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 37s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 10s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 37s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 10s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.
This commit is contained in:
@@ -118,6 +118,17 @@ async def lifespan(app: FastAPI):
|
||||
db_config = Config.get_database_config()
|
||||
app_state["scraper"] = EnhancedWaterMonitorScraper(db_config)
|
||||
|
||||
# Initialize HII/ThaiWater collector (rainfall + backup water level)
|
||||
try:
|
||||
from .hii_collector import create_collector_from_config
|
||||
|
||||
app_state["hii_collector"] = create_collector_from_config()
|
||||
if app_state["hii_collector"]:
|
||||
logger.info("HII collection enabled (Ping-basin rainfall + water level)")
|
||||
except Exception as e:
|
||||
app_state["hii_collector"] = None
|
||||
logger.error(f"HII collector initialization failed: {e}")
|
||||
|
||||
# Initialize health checks
|
||||
health_manager = HealthCheckManager()
|
||||
health_manager.add_check(DatabaseHealthCheck(app_state["scraper"].db_adapter))
|
||||
@@ -220,6 +231,22 @@ async def background_scraping_task():
|
||||
increment_counter("scraping_cycles_failed")
|
||||
logger.error(f"Background scraping cycle failed: {e}")
|
||||
|
||||
# HII rainfall/water-level snapshot (independent of RID cycle)
|
||||
hii_collector = app_state.get("hii_collector")
|
||||
if hii_collector:
|
||||
try:
|
||||
hii_counts = await asyncio.get_event_loop().run_in_executor(
|
||||
None, hii_collector.run_cycle
|
||||
)
|
||||
set_gauge(
|
||||
"hii_rainfall_rows_saved", hii_counts["rainfall"]
|
||||
)
|
||||
set_gauge(
|
||||
"hii_waterlevel_rows_saved", hii_counts["waterlevel"]
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"HII collection failed: {e}")
|
||||
|
||||
app_state["is_scraping"] = False
|
||||
|
||||
# Calculate next run time
|
||||
|
||||
Reference in New Issue
Block a user