feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s

Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103;
results in models/eval_variants.json) showed the absolute-level target
alerting AT the crossing on essentially every event, while the rise
target (future max - current level, level added back at serving) gives
+6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than
weighted/quantile variants, and ~11% better MAE. Weighted and quantile
variants rejected: more false alarms, no Brier-score calibration gain.

Ported to production: train.py fits rise in both eval and refit passes
(sigma/metrics computed in absolute space), bundles stamped hgb-v2 with
regression_target='rise', predict.py adds the level back for v2 and
stays compatible with v1 bundles, backtest_render.py mirrors the same
math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h
BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and
the record-peak underprediction is gone (rise models can exceed the
training max). The >=12h acceptance gate still fails honestly at +6h —
closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at
6/12/24h; docs updated throughout.
This commit is contained in:
2026-08-12 15:43:29 +07:00
parent a0086086a2
commit 21e9d2e114
8 changed files with 814 additions and 44 deletions
+50 -32
View File
@@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**:
| Head | Type | Target |
|---|---|---|
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] |
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) |
| `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
| `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
@@ -261,14 +261,19 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93%
between them, roughly doubling the test rows and collapsing the warning base
rates).
**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB
(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1:
**Current model** `hgb-v2` (rise target), generated 2026-08-12 on the
gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12.
P.1:
| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
|---|---|---|---|---|---|
| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% |
| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% |
| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% |
| 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% |
| 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% |
| 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% |
(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored
5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved
every regression figure; PR-AUC belongs to the unchanged classifier heads.)
Level accuracy improved; standalone classifier discrimination did not survive
the data change (which is why serving is now `max(classifier, sigmoid)` — see
@@ -327,6 +332,20 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
> `scripts/backtest_render.py` regenerates all three charts and fails its
> acceptance gate while the 2024 lead stays under 12 h — keeping this page
> honest is now automatic.
>
> **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin,
> event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon
> 2021-2025) compared the absolute-level target against rise-target variants.
> The rise target — regression predicts *future max minus current level*, the
> level is added back at serving — won decisively and is now deployed as
> `hgb-v2`: the regenerated charts below show the 2024 first alert moving from
> 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert
> from at-crossing to **45 h early**, the record-peak underprediction
> eliminated (the model now slightly overshoots 5.30 m rather than capping
> ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and
> quantile variants were evaluated and rejected (more false alarms, no
> calibration gain by Brier score). The ≥12 h acceptance gate still fails at
> +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
### The September 2025 flood, as the deployed configuration saw it
@@ -334,14 +353,14 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
through 2024](img/backtest-2025-p1.png)
This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the
re-examination note above; the original one-off render, trained on the sparser
data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On
today's fuller dataset the retrained equivalent first alerts at **18:00 on
27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The
discrimination remains good: the near-miss 3.51 m crest on 26 September never
triggers, the probability fires only for the real event, and it stands down as
the water recedes.
saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
gap-filled data — see the re-examination note above for the history of these
numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45
hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is
partly the twin-crest shape of this event (the near-miss 3.51 m crest of
26 September keeps the 24 h-peak forecast near the line), so read it as
"the model was correctly alarmed through the whole double crest", not as a
general 45 h capability.
### Headline validation: the October 2024 record flood
@@ -353,29 +372,28 @@ followed. This is the closest thing to a real operational test available.
![Observed P.1 level vs the model's 24 h-ahead predicted peak through the
October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
The render above shows the whole event hour by hour *(regenerated 2026-08-11
on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead
predicted peak the model issued at each hour (amber, dashed) — the amber line
tracks both flood waves but no longer clearly leads the first one. Bottom: the
belt-and-braces probability of flooding within 24 h; on the fuller data its
**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00
24 September crossing**, then stays correctly alarmed through the October
record wave. Also visible, honestly: the predicted peak tops out well short of
the actual 5.30 m record (the extreme-compression limitation discussed below).
The same historic model track drives the dashboard's "Replay Oct 2024 flood"
feature.
The render above shows the whole event hour by hour *(regenerated 2026-08-12
with the hgb-v2 rise target)*. Top: the observed level (blue) against the
24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the
amber line now leads the blue one into both flood waves. Bottom: the
belt-and-braces probability of flooding within 24 h; the **first alert comes
at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays
correctly alarmed through the October record wave. The predicted peak now
slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the
rise target removed the cannot-exceed-training-max ceiling. The same historic
model track drives the dashboard's "Replay Oct 2024 flood" feature.
![Hour-by-hour detail of the detection window, 2228 September
2024](img/backtest-2024-p1-detail.png)
The hour-by-hour detail of the detection window *(regenerated 2026-08-11)*
shows the corrected sequence: the river crosses 3.70 m at **17:00 on
The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on
24 September** (the hours recovered by gap-filling; independently confirmed by
the HII sensor at the same bridge), while the retrained model's probability
only crosses 0.5 at **11:00 on 25 September**. The original render — sparser
data, different trained model — alerted at 01:00 on 24 September against an
apparent 01:00 25 September crossing. Closing this real gap is what the
rainfall features and rise-target work are for.
the HII sensor at the same bridge), and the model's probability crosses 0.5 at
**11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under
the absolute-level target this alert came 18 hours *after* the crossing; the
rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance
gate demands is what the rainfall features are for.
The event bullets below quote the original (pre-gap-fill) evaluation of the
deployed model and are kept for the historical record — see the re-examination