Commit Graph
4 Commits
Author SHA1 Message Date
grabowski df0ae8cda3 feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
The rolling-origin harness (models/eval_rain.json) showed catchment rain
halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and
extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at
P.103). Ported: train_all loads the catchment-mean series (use_rain /
--no-rain to opt out; without it bundles train as v2), predict fetches
live rain hourly and passes an empty series on failure so rain-trained
bundles serve with NaN features instead of tripping the feature guard,
and the leader worker persists hourly per-point + catchment-mean rows to
a new openmeteo_rain table.

Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING
(alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h
acceptance gate PASSES for the first time. Journey on that crossing:
v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its
artifact +46h latch for a calibrated +2h with zero false alarms. P.1
MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.
2026-08-12 17:05:01 +07:00
grabowski 21e9d2e114 feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103;
results in models/eval_variants.json) showed the absolute-level target
alerting AT the crossing on essentially every event, while the rise
target (future max - current level, level added back at serving) gives
+6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than
weighted/quantile variants, and ~11% better MAE. Weighted and quantile
variants rejected: more false alarms, no Brier-score calibration gain.

Ported to production: train.py fits rise in both eval and refit passes
(sigma/metrics computed in absolute space), bundles stamped hgb-v2 with
regression_target='rise', predict.py adds the level back for v2 and
stays compatible with v1 bundles, backtest_render.py mirrors the same
math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h
BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and
the record-peak underprediction is gone (rise models can exceed the
training max). The >=12h acceptance gate still fails honestly at +6h —
closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at
6/12/24h; docs updated throughout.
2026-08-12 15:43:29 +07:00
grabowski 0005f7dce1 feat: codified backtests, honest docs, belt-and-braces serving, perf fixes
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the
flood backtests, now reproducible via scripts/backtest_render.py (renders
the three docs/img charts and gates on a >=12h 2024 first-alert lead —
currently failing by design and documented as such).

Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing
was 24 Sep 17:00 (8h earlier than recorded; confirmed against the
independent HII sensor), the historical 24h-warning claim was partly a
missing-data artifact, and retrained warn classifiers collapse on the
filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves
(11.3 -> 10.5 cm). Serving therefore becomes max(classifier,
sigmoid(regression)) so alerting is never worse than the regression path;
metrics table, head-gating tiers, honest-limits and runbook expectations
all updated to the current model (hgb-v1+d2d0e65).

Perf, from Locust load testing (scripts/locustfile.py + load_test.py):
single-flight lock around /forecast inference (concurrent cache misses
previously each ran ~18s inference and starved the shared thread pool;
200-user run after: 105 rps, 0.01% errors), and /measurements/latest +
/health moved off the event loop (synchronous DB/network calls in async
handlers were stalling every request under load).
2026-08-12 10:46:00 +07:00
grabowski f0183faa62 feat: discharge-driven river animation speed, replay stat tiles, hourly doc render
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 10s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 20s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
- river dash animation now flows at a speed continuously derived from
  each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via
  inline per-path animation-duration, so it updates live and per-frame
  during the replay (CSS speed classes removed - setStyle cannot change
  classes)
- the replay drives the Combined discharge and Strongest flow stat
  tiles each frame (marked '2024 replay'), restored on finish
- docs: hour-by-hour detection detail render (22-28 Sep 2024) showing
  the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the
  24 h warning between them; embedded with commentary
- Matrix alerts now link to https://water.buildfor.life/ (override via
  ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link
2026-08-10 18:29:01 +07:00