20 Commits

Include renames

Author	SHA1	Message	Date
grabowski	701bb87d34	feat(p1): city flood stages and zones from CMU Water Center's 2024 flood map ... CMU Water Center's Chiang Mai flood hazard map (CMFlood) replaces the older municipal map. It is built on the October 2024 flood and has the Ping leave its banks above P.1 4.20 m, with five stages: 4.30, 4.50, 4.70, 5.00 and 5.30 m (the 2024 record). The old map's seven stages (3.70-4.60 m) said the city floods from 3.70 m; CMU calls that map out of date. - P1_FLOOD_STAGES: the five CMU stages, discharge on RID's 2024 rating. /forecast's P.1 `stages` now has five entries (same format). Thresholds are unchanged: 4.20 m danger is the bank top, 3.70 m warning half a metre below (as on CMU's gauge page). flood_config.json and the Rust parity fixtures regenerated (only the stages changed). - flood-zones-p1.geojson: CMU's KML (L1..L5) as five disjoint zones, zone k floods first at stage k; scripts/build_flood_zones.py rebuilds it. - Dashboard: CMU's colours and credit (map attribution); the stage grid follows the number of stages; zone lookup for markers honours holes. The city verdict uses statusOf's classes: flooding (over the banks) from 4.20 m, high water from 3.70 m. Zones show by themselves when the river is over its banks or stage 1 is >= 50 % likely. Alert, replay and verdict texts no longer call 3.70 m city flooding (EN + TH). - Docs: DATA_SOURCES 5.1 (CMFlood: the map, flood marks, poles, city gauges), FLOOD_FORECASTING, DASHBOARD_REDESIGN, NOTIFICATIONS, README credits.	2026-10-26 14:19:52 +07:00
grabowski	180d95c6bd	fix(forecast): no nightly gaps in the issued model forecasts ... The P.1 chart's "24 h peak, as issued" line had a gap every night and a dip at 21:00. Two causes: - A run forecasts from the newest reading only. RID publishes 22:00..06:00 in one batch in the morning (and now and then one hour late), so those hours were never forecast from. After each run the collector now forecasts the hours of the last 48 h (two weeks on the first run after a start) that have a RID reading but no model forecast, each as a run at that hour would have: gauges, HII fill and dam state up to that hour. Rain is fetched once, so a late hour's rain_fc24 partly holds rain that has since fallen (as in training). Only missing (hour, station) pairs are written. `ping-collector catch-up-forecasts --hours N` goes further back. - Overnight each hourly run re-issues the 21:00 reading, after 6 h as the heuristic, which overwrote the 21:00 model row. A heuristic row no longer replaces a model row in forecast_history. The dashboard draws only model rows on the issued line. Checked on a local copy of 18 days of RID data: the catch-up of an hour matches `ping-collector forecast --now <hour>`; 288 hours took 20 s (debug build). Tests for both database rules.	2026-10-20 15:21:48 +07:00
grabowski	cf10f2e4c8	feat(dashboard): console phase 4 - 2024 replay, simulation, replaces the old page ... The console is now the dashboard at / (src/static/dashboard.html); the old page and console.html are gone. - Replay of the October 2024 flood (flood-2024-all.json): the page's clock, verdict, list, markers, flood zones and panel follow the replay; the panel charts the season on fixed axes with the 2024 model's 24 h peak. Hourly while the model alerts or P.1 is above 3.50 m, faster through quiet days; pause and drag. Rain, basin stations and the 5-minute refresh are off during the replay (today's data). - ?demo_level= and ?demo_rise= simulate P.1. A violet REPLAY / amber SIMULATION bar, the top bar's KPI and the P.1 panel say so; the mode is drawn from state.mode alone, so a refresh or language switch cannot relabel either as live. - History: rows without discharge_percent are ThaiWater's copy of the gauge (night hours RID publishes late), with a different rating (P.1: 26 against RID's 54 m3/s); their discharge is not drawn for gauges with RID discharge. The discharge strip has headroom so its label stays clear of the line. - Swap: the old page's title, description, Open Graph and JSON-LD tags; no noindex; tests/test_dashboard.py = console tests + the old checks that still apply (one P.1 level, SEO files) + a replay/simulation labelling test. - Docs: README screenshot (docs/img/dashboard.jpg, replacing two unused images), CLAUDE.md, CONTRIBUTING.md, FLOOD_FORECASTING.md (where the model shows), DASHBOARD_REDESIGN.md (phase 4 done, mode rules). Deploy needs a restart: the server reads dashboard.html at start.	2026-10-20 13:47:15 +07:00
grabowski	7e642555b4	chore: remove the Python service; Python is the training stack only ... Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.	2026-10-18 17:15:07 +07:00
grabowski	6ae67dec63	feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night ... Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock).	2026-10-18 14:43:39 +07:00
grabowski	6d4beae65a	fix(ml): serve the forecast rain the model was trained with ... build_features cut the rain series to the gauge grid before the forward sum, and the grid ends at the newest reading, so the row every live forecast is made from had rain_fc24 = NaN. Training and all evaluations had the real Open-Meteo forecast there. The forward sum now runs over the grid extended by 24 h; earlier rows are unchanged. Harness (new opt-in variant rise_rain_fc24_missing = rise_rain with rain_fc24 blanked on test rows, five folds, --from-cache): P.1 and P.103 nearly unaffected, P.21 false alarms 10 -> 5 and MAE 0.232 -> 0.215 with the fix. Write-up in docs/FLOOD_FORECASTING.md section 5; results in models/eval_2026-10-18_fc24.json. Also: HII mirror rows in one hour now resolve to the latest reading via a stable sort on the raw time (was an unstable sort on the floored hour, i.e. row-order dependent). Needed for an exact Rust port.	2026-10-18 14:07:56 +07:00
grabowski	ce08312c0f	docs: public dashboard URL (water.buildfor.life) replaces the Tailscale IP everywhere	2026-09-11 22:38:13 +02:00
grabowski	d621aa9ce7	eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate ... Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_* caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.	2026-09-11 21:55:37 +02:00
grabowski	764764e07e	feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote ... train_all() now raises RainUnavailableError when use_rain=True and the Open-Meteo history cannot be loaded, instead of logging a warning and writing gauge-only (v2) bundles over the deployed v3 set -- which is what the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit way to get v2. CLI exits 2 with a one-line error. Three tests cover the guard, the opt-out, and the v3 happy path. scripts/retrain.sh trains into models/.staging, refuses to promote unless metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles into place (previous generation kept in models/.previous). No API restart: predict.py reloads by mtime on the hourly precompute. water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent, OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now does `uv sync` into .venv (one env rule; removes a stale venv/) and enables the timer. water-monitor.service in the repo matched neither the deployed unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).	2026-09-11 21:37:11 +02:00
grabowski	5dc5850df6	docs: source sweep — P.75 already is the Mae Ngat release signal ... Verified from a Thai ISP that lsim.rid.go.th is unreachable (not geo-blocked), then swept for any better-than-daily Mae Ngat source. Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has been a model feature since v1 — the model has always read the dam's actual outflow, hourly and directly. That is the likelier reason the daily reservoir table adds nothing, beyond its publication lag. Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with no archive, so they cannot retrain against past events; the HII collector accumulates them from 2026-08-11 for a post-monsoon revisit. Also documents api/dam (whole per-dam range in one request, vs the ~2,900-request per-day loop) and several cross-check mirrors.	2026-08-13 21:58:47 +07:00
grabowski	70e4da07a0	docs: lsim.rid.go.th is unreachable, not geo-blocked ... Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Correct the earlier note that assumed the timeout was a foreign-network block, and stop pointing the dam-feature follow-up at a host that cannot be reached.	2026-08-13 21:00:24 +07:00
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	21e9d2e114	feat: hgb-v2 — regression heads predict rise, recovering flood warning lead ... Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.	2026-08-12 15:43:29 +07:00
grabowski	0005f7dce1	feat: codified backtests, honest docs, belt-and-braces serving, perf fixes ... Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).	2026-08-12 10:46:00 +07:00
grabowski	27fa292e09	docs: September 2025 flood render - deployed config, 24 h warning, peak within 7 cm ... Hour-by-hour chart of the 25-28 Sep 2025 event as forecast by the exact deployed configuration (trained through 2024, event unseen): first alert 26 Sep 18:00, flooding began 27 Sep 18:00 (24 h lead), predicted peak 4.00 m vs actual 3.93 m. The near-miss 3.51 m crest on 26 Sep correctly never alerted.	2026-08-10 18:35:55 +07:00
grabowski	f0183faa62	feat: discharge-driven river animation speed, replay stat tiles, hourly doc render ... - river dash animation now flows at a speed continuously derived from each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via inline per-path animation-duration, so it updates live and per-frame during the replay (CSS speed classes removed - setStyle cannot change classes) - the replay drives the Combined discharge and Strongest flow stat tiles each frame (marked '2024 replay'), restored on finish - docs: hour-by-hour detection detail render (22-28 Sep 2024) showing the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the 24 h warning between them; embedded with commentary - Matrix alerts now link to https://water.buildfor.life/ (override via ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link	2026-08-10 18:29:01 +07:00
grabowski	6112c681a7	docs: render of actual vs predicted through the 2024 flood; adaptive replay speed ... docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead predicted peak issued at each hour by a model trained only on pre-flood data, with the warning-probability panel below (first alert 24 Sep 01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's headline-validation section with an honest reading, including the ~0.4 m peak under-prediction. Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when risk is elevated, 1 h (hour-by-hour) while the model is alerting or the river is near/above flood stage - so viewers can watch the detection sequence unfold.	2026-08-10 18:10:48 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00