7 Commits

Include renames

Author	SHA1	Message	Date
grabowski	d621aa9ce7	eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate ... Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_* caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.	2026-09-11 21:55:37 +02:00
grabowski	731f10910e	perf: stale-while-revalidate caching, bigger executor, cheap DB health probe ... Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine).	2026-08-12 12:32:38 +07:00
grabowski	039af8caac	perf: cache /measurements/latest; stale-on-error fallback for cached endpoints ... The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120).	2026-08-12 11:05:02 +07:00
grabowski	d27ca8bf40	perf: 120s TTL cache with single-flight on /api/hii/*/latest ... These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one.	2026-08-12 10:58:19 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00