8 Commits

Include renames

Author	SHA1	Message	Date
grabowski	5ad8e4eac3	ci: green pipelines that check what exists; one formatting contract ... The Test Suite job failed on every push since the black check was added because the tree had never been formatted, and pre-commit said 120 columns while CI ran black's default 88. pyproject.toml now carries [tool.black] / [tool.isort] (88, black profile) as the single source; pre-commit reads it; `make format` applied it (13 files, whitespace only, 146 insertions / 128 deletions, tests unchanged at 146 passed). ci.yml: lint (black, isort, flake8 hard errors) + pytest. The Docker registry push, VictoriaMetrics integration test, staging/production deploy and Apache-Bench jobs were template scaffolding for hosts and registries that do not exist; production is a systemd unit updated by git pull. Removed rather than left permanently skipped. docs.yml: the "Check markdown links" step curl'd every URL in every .md and failed on localhost examples and the Tailscale IP, and the Sphinx jobs built artifacts nobody read. Replaced by two checks that mean something: relative links/images in README, CONTRIBUTING and docs/ resolve inside the repo, and the FastAPI OpenAPI schema exports with the documented endpoints present (uploaded as an artifact).	2026-09-11 23:05:37 +02:00
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	21e9d2e114	feat: hgb-v2 — regression heads predict rise, recovering flood warning lead ... Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.	2026-08-12 15:43:29 +07:00
grabowski	0005f7dce1	feat: codified backtests, honest docs, belt-and-braces serving, perf fixes ... Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).	2026-08-12 10:46:00 +07:00
grabowski	ecd34177bb	fix: backtest/review findings in the flood-ML package ... From the adversarial review and threshold backtest (swarm verification): - predict.py: when a bundle's trained thresholds differ from the current config (deploy before retrain), skip its stale classifier heads and derive p_warning/p_danger from the regression + sigma against the CURRENT thresholds - the dashboard can no longer show contradictory old-threshold classifier output next to new-threshold stages - features.py: decouple the low-coverage regression-label rescue from the warning threshold (now the station's own p97.5 level); the old coupling silently dropped 34% of P.5's regression training rows and cost +46% MAE when its threshold rose - features.py: P.82 danger 3.80 -> 3.75 (3.80 was above the station's 8-year maximum of 3.78, so danger could never train or fire) - data.py / predict.py: anchor models/cache paths to the repo root; the relative paths silently returned zero rows when run from another CWD - annotate P.4A thresholds as low-confidence (11 supporting readings) 47 tests pass. Retrain required for the label-rescue and P.82 changes to reach the classifier heads.	2026-08-10 15:57:01 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00