feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.
This commit is contained in:
+50
-32
@@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**:
|
||||
|
||||
| Head | Type | Target |
|
||||
|---|---|---|
|
||||
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] |
|
||||
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) |
|
||||
| `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
|
||||
| `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
|
||||
|
||||
@@ -261,14 +261,19 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93%
|
||||
between them, roughly doubling the test rows and collapsing the warning base
|
||||
rates).
|
||||
|
||||
**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB
|
||||
(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1:
|
||||
**Current model** `hgb-v2` (rise target), generated 2026-08-12 on the
|
||||
gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12.
|
||||
P.1:
|
||||
|
||||
| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
|
||||
|---|---|---|---|---|---|
|
||||
| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% |
|
||||
| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% |
|
||||
| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% |
|
||||
| 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% |
|
||||
| 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% |
|
||||
| 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% |
|
||||
|
||||
(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored
|
||||
5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved
|
||||
every regression figure; PR-AUC belongs to the unchanged classifier heads.)
|
||||
|
||||
Level accuracy improved; standalone classifier discrimination did not survive
|
||||
the data change (which is why serving is now `max(classifier, sigmoid)` — see
|
||||
@@ -327,6 +332,20 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
|
||||
> `scripts/backtest_render.py` regenerates all three charts and fails its
|
||||
> acceptance gate while the 2024 lead stays under 12 h — keeping this page
|
||||
> honest is now automatic.
|
||||
>
|
||||
> **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin,
|
||||
> event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon
|
||||
> 2021-2025) compared the absolute-level target against rise-target variants.
|
||||
> The rise target — regression predicts *future max minus current level*, the
|
||||
> level is added back at serving — won decisively and is now deployed as
|
||||
> `hgb-v2`: the regenerated charts below show the 2024 first alert moving from
|
||||
> 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert
|
||||
> from at-crossing to **45 h early**, the record-peak underprediction
|
||||
> eliminated (the model now slightly overshoots 5.30 m rather than capping
|
||||
> ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and
|
||||
> quantile variants were evaluated and rejected (more false alarms, no
|
||||
> calibration gain by Brier score). The ≥12 h acceptance gate still fails at
|
||||
> +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
|
||||
|
||||
### The September 2025 flood, as the deployed configuration saw it
|
||||
|
||||
@@ -334,14 +353,14 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
|
||||
through 2024](img/backtest-2025-p1.png)
|
||||
|
||||
This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
|
||||
saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the
|
||||
re-examination note above; the original one-off render, trained on the sparser
|
||||
data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On
|
||||
today's fuller dataset the retrained equivalent first alerts at **18:00 on
|
||||
27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The
|
||||
discrimination remains good: the near-miss 3.51 m crest on 26 September never
|
||||
triggers, the probability fires only for the real event, and it stands down as
|
||||
the water recedes.
|
||||
saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
|
||||
gap-filled data — see the re-examination note above for the history of these
|
||||
numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45
|
||||
hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is
|
||||
partly the twin-crest shape of this event (the near-miss 3.51 m crest of
|
||||
26 September keeps the 24 h-peak forecast near the line), so read it as
|
||||
"the model was correctly alarmed through the whole double crest", not as a
|
||||
general 45 h capability.
|
||||
|
||||
### Headline validation: the October 2024 record flood
|
||||
|
||||
@@ -353,29 +372,28 @@ followed. This is the closest thing to a real operational test available.
|
||||

|
||||
|
||||
The render above shows the whole event hour by hour *(regenerated 2026-08-11
|
||||
on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead
|
||||
predicted peak the model issued at each hour (amber, dashed) — the amber line
|
||||
tracks both flood waves but no longer clearly leads the first one. Bottom: the
|
||||
belt-and-braces probability of flooding within 24 h; on the fuller data its
|
||||
**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00
|
||||
24 September crossing**, then stays correctly alarmed through the October
|
||||
record wave. Also visible, honestly: the predicted peak tops out well short of
|
||||
the actual 5.30 m record (the extreme-compression limitation discussed below).
|
||||
The same historic model track drives the dashboard's "Replay Oct 2024 flood"
|
||||
feature.
|
||||
The render above shows the whole event hour by hour *(regenerated 2026-08-12
|
||||
with the hgb-v2 rise target)*. Top: the observed level (blue) against the
|
||||
24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the
|
||||
amber line now leads the blue one into both flood waves. Bottom: the
|
||||
belt-and-braces probability of flooding within 24 h; the **first alert comes
|
||||
at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays
|
||||
correctly alarmed through the October record wave. The predicted peak now
|
||||
slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the
|
||||
rise target removed the cannot-exceed-training-max ceiling. The same historic
|
||||
model track drives the dashboard's "Replay Oct 2024 flood" feature.
|
||||
|
||||

|
||||
|
||||
The hour-by-hour detail of the detection window *(regenerated 2026-08-11)*
|
||||
shows the corrected sequence: the river crosses 3.70 m at **17:00 on
|
||||
The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
|
||||
hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on
|
||||
24 September** (the hours recovered by gap-filling; independently confirmed by
|
||||
the HII sensor at the same bridge), while the retrained model's probability
|
||||
only crosses 0.5 at **11:00 on 25 September**. The original render — sparser
|
||||
data, different trained model — alerted at 01:00 on 24 September against an
|
||||
apparent 01:00 25 September crossing. Closing this real gap is what the
|
||||
rainfall features and rise-target work are for.
|
||||
the HII sensor at the same bridge), and the model's probability crosses 0.5 at
|
||||
**11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under
|
||||
the absolute-level target this alert came 18 hours *after* the crossing; the
|
||||
rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance
|
||||
gate demands is what the rainfall features are for.
|
||||
|
||||
The event bullets below quote the original (pre-gap-fill) evaluation of the
|
||||
deployed model and are kept for the historical record — see the re-examination
|
||||
|
||||
Reference in New Issue
Block a user