B4L/Northern-Thailand-Ping-River-Monitor
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Details
Documentation / Validate Documentation (push) Failing after 9s
Details
Documentation / Generate API Documentation (push) Successful in 8s
Details
Documentation / Build Sphinx Documentation (push) Successful in 17s
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Details
Documentation / Documentation Summary (push) Successful in 2s
Details

Browse Source 
Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103;
results in models/eval_variants.json) showed the absolute-level target
alerting AT the crossing on essentially every event, while the rise
target (future max - current level, level added back at serving) gives
+6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than
weighted/quantile variants, and ~11% better MAE. Weighted and quantile
variants rejected: more false alarms, no Brier-score calibration gain.

Ported to production: train.py fits rise in both eval and refit passes
(sigma/metrics computed in absolute space), bundles stamped hgb-v2 with
regression_target='rise', predict.py adds the level back for v2 and
stays compatible with v1 bundles, backtest_render.py mirrors the same
math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h
BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and
the record-peak underprediction is gone (rise models can exceed the
training max). The >=12h acceptance gate still fails honestly at +6h —
closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at
6/12/24h; docs updated throughout.
This commit is contained in:
grabowski
2026-08-12 15:43:29 +07:00
parent a0086086a2
commit 21e9d2e114
8 changed files with 814 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FLOOD_FORECASTING.md
+50 -32
View File
@@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**:
| Head | Type | Target |
|---|---|---|
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] |
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) |
| `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
| `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
@@ -261,14 +261,19 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93%
between them, roughly doubling the test rows and collapsing the warning base
rates).
**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB
(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1:
**Current model** `hgb-v2` (rise target), generated 2026-08-12 on the
gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12.
P.1:
| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
|---|---|---|---|---|---|
| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% |
| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% |
| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% |
| 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% |
| 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% |
| 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% |
(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored
5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved
every regression figure; PR-AUC belongs to the unchanged classifier heads.)
Level accuracy improved; standalone classifier discrimination did not survive
the data change (which is why serving is now `max(classifier, sigmoid)` — see
@@ -327,6 +332,20 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
> `scripts/backtest_render.py` regenerates all three charts and fails its
> acceptance gate while the 2024 lead stays under 12 h — keeping this page
> honest is now automatic.
>
> **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin,
> event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon
> 2021-2025) compared the absolute-level target against rise-target variants.
> The rise target — regression predicts *future max minus current level*, the
> level is added back at serving — won decisively and is now deployed as
> `hgb-v2`: the regenerated charts below show the 2024 first alert moving from
> 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert
> from at-crossing to **45 h early**, the record-peak underprediction
> eliminated (the model now slightly overshoots 5.30 m rather than capping
> ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and
> quantile variants were evaluated and rejected (more false alarms, no
> calibration gain by Brier score). The ≥12 h acceptance gate still fails at
> +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
### The September 2025 flood, as the deployed configuration saw it
@@ -334,14 +353,14 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
through 2024](img/backtest-2025-p1.png)
This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the
re-examination note above; the original one-off render, trained on the sparser
data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On
today's fuller dataset the retrained equivalent first alerts at **18:00 on
27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The
discrimination remains good: the near-miss 3.51 m crest on 26 September never
triggers, the probability fires only for the real event, and it stands down as
the water recedes.
saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
gap-filled data — see the re-examination note above for the history of these
numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45
hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is
partly the twin-crest shape of this event (the near-miss 3.51 m crest of
26 September keeps the 24 h-peak forecast near the line), so read it as
"the model was correctly alarmed through the whole double crest", not as a
general 45 h capability.
### Headline validation: the October 2024 record flood
@@ -353,29 +372,28 @@ followed. This is the closest thing to a real operational test available.
![Observed P.1 level vs the model's 24 h-ahead predicted peak through the
October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
The render above shows the whole event hour by hour *(regenerated 2026-08-11
on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead
predicted peak the model issued at each hour (amber, dashed) — the amber line
tracks both flood waves but no longer clearly leads the first one. Bottom: the
belt-and-braces probability of flooding within 24 h; on the fuller data its
**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00
24 September crossing**, then stays correctly alarmed through the October
record wave. Also visible, honestly: the predicted peak tops out well short of
the actual 5.30 m record (the extreme-compression limitation discussed below).
The same historic model track drives the dashboard's "Replay Oct 2024 flood"
feature.
The render above shows the whole event hour by hour *(regenerated 2026-08-12
with the hgb-v2 rise target)*. Top: the observed level (blue) against the
24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the
amber line now leads the blue one into both flood waves. Bottom: the
belt-and-braces probability of flooding within 24 h; the **first alert comes
at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays
correctly alarmed through the October record wave. The predicted peak now
slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the
rise target removed the cannot-exceed-training-max ceiling. The same historic
model track drives the dashboard's "Replay Oct 2024 flood" feature.
![Hour-by-hour detail of the detection window, 22–28 September
2024](img/backtest-2024-p1-detail.png)
The hour-by-hour detail of the detection window *(regenerated 2026-08-11)*
shows the corrected sequence: the river crosses 3.70 m at **17:00 on
The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on
24 September** (the hours recovered by gap-filling; independently confirmed by
the HII sensor at the same bridge), while the retrained model's probability
only crosses 0.5 at **11:00 on 25 September**. The original render — sparser
data, different trained model — alerted at 01:00 on 24 September against an
apparent 01:00 25 September crossing. Closing this real gap is what the
rainfall features and rise-target work are for.
the HII sensor at the same bridge), and the model's probability crosses 0.5 at
**11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under
the absolute-level target this alert came 18 hours *after* the crossing; the
rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance
gate demands is what the rainfall features are for.
The event bullets below quote the original (pre-gap-fill) evaluation of the
deployed model and are kept for the historical record — see the re-examination
docs/img/backtest-2024-p1-detail.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 118 KiB

After

Width:  |  Height:  |  Size: 108 KiB

docs/img/backtest-2024-p1.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 147 KiB

After

Width:  |  Height:  |  Size: 142 KiB

docs/img/backtest-2025-p1.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 141 KiB

After

Width:  |  Height:  |  Size: 138 KiB

models/eval_variants.json
+723
View File
@@ -0,0 +1,723 @@
[
{
"station": "P.1",
"warn_thr": 3.7,
"folds": [
{
"year": 2021,
"n_train": 20024,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.073069548799388,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.070970681677648,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.07394501467643348,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.07289422294875283,
"mae_above_2p5": null,
"brier_warn": 6.870660393452143e-17,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2022,
"n_train": 28782,
"n_test": 4392,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"peak_ts": "2022-10-03T15:00:00",
"peak_level": 4.65
}
],
"variants": {
"baseline_abs": {
"mae": 0.09128484356553154,
"mae_above_2p5": 0.2856025611084259,
"brier_warn": 0.005440790049399641,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 0.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.4
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.08489734638010188,
"mae_above_2p5": 0.24517428534843191,
"brier_warn": 0.004136576477574926,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 6.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.870617057762467
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.08451609397908402,
"mae_above_2p5": 0.24677443475763222,
"brier_warn": 0.0041231071254735656,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 6.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.9060484870307466
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.08412283956084679,
"mae_above_2p5": 0.2653552568120815,
"brier_warn": 0.004349073127074364,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 3.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.641048749261635
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2023,
"n_train": 37537,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.08161389876263064,
"mae_above_2p5": 0.1335546690803318,
"brier_warn": 3.284466170848282e-09,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.07647507344443248,
"mae_above_2p5": 0.12593127745781565,
"brier_warn": 3.19529506149809e-11,
"events": [],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.07633405851952908,
"mae_above_2p5": 0.12234064428755373,
"brier_warn": 9.093694748477163e-10,
"events": [],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.07460579730428984,
"mae_above_2p5": 0.11243109915383911,
"brier_warn": 8.148092046865233e-08,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2024,
"n_train": 46321,
"n_test": 4392,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"peak_ts": "2024-09-26T02:00:00",
"peak_level": 4.93
},
{
"crossing": "2024-10-03T09:00:00",
"peak_ts": "2024-10-05T12:00:00",
"peak_level": 5.3
}
],
"variants": {
"baseline_abs": {
"mae": 0.11011679980584997,
"mae_above_2p5": 0.28849258735632527,
"brier_warn": 0.010963910189916666,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 0.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.39
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 0.0,
"peak_level": 5.3,
"peak_pred_24h_before": 4.9
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.10075626719251345,
"mae_above_2p5": 0.26384939692286363,
"brier_warn": 0.011608221543810462,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 6.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.534709676862131
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 3.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.056619694027486
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.10644718990351541,
"mae_above_2p5": 0.2848978622398853,
"brier_warn": 0.013872775238121434,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 9.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.502016075573465
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 72.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.123356409745132
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.09617759284936986,
"mae_above_2p5": 0.271753261978628,
"brier_warn": 0.011660822672241142,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 4.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.532042588397363
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 2.0,
"peak_level": 5.3,
"peak_pred_24h_before": 4.956335200519836
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2025,
"n_train": 55081,
"n_test": 4392,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"peak_ts": "2025-09-27T22:00:00",
"peak_level": 3.93
}
],
"variants": {
"baseline_abs": {
"mae": 0.15985415338980086,
"mae_above_2p5": 0.277881996645181,
"brier_warn": 0.009824391159106445,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 0.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.2561560254201525
}
],
"false_alarm_episodes": 2
},
"rise": {
"mae": 0.13102686839694494,
"mae_above_2p5": 0.2729944665011028,
"brier_warn": 0.010889835530782944,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 46.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.232029710676319
}
],
"false_alarm_episodes": 1
},
"rise_weighted": {
"mae": 0.13121120517634968,
"mae_above_2p5": 0.27796411411759725,
"brier_warn": 0.009393099541680463,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 46.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 3
},
"rise_quantile": {
"mae": 0.11678707793162903,
"mae_above_2p5": 0.28528124941530186,
"brier_warn": 0.013009605253899563,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 45.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 3
}
}
}
]
},
{
"station": "P.103",
"warn_thr": 5.95,
"folds": [
{
"year": 2021,
"n_train": 20009,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.1517830652111977,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.15279228710793116,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.17632375724441426,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.13593873552532,
"mae_above_2p5": null,
"brier_warn": 1.6481640122270367e-09,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2022,
"n_train": 28769,
"n_test": 4392,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"peak_ts": "2022-08-14T08:00:00",
"peak_level": 6.09
},
{
"crossing": "2022-10-02T16:00:00",
"peak_ts": "2022-10-03T16:00:00",
"peak_level": 7.54
}
],
"variants": {
"baseline_abs": {
"mae": 0.15960776318568,
"mae_above_2p5": 0.5240325509357986,
"brier_warn": 0.010134410144044625,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 0.0,
"peak_level": 6.09,
"peak_pred_24h_before": 5.01930531142127
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 0.0,
"peak_level": 7.54,
"peak_pred_24h_before": 6.01
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.15389446229226025,
"mae_above_2p5": 0.41887094569113953,
"brier_warn": 0.007740077230532647,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 6.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.837975953559253
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 9.0,
"peak_level": 7.54,
"peak_pred_24h_before": 7.168225721504108
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.1498671765711928,
"mae_above_2p5": 0.40088103598066704,
"brier_warn": 0.00801963186023152,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 6.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.81
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 8.0,
"peak_level": 7.54,
"peak_pred_24h_before": 7.002000385945215
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.13833442255794418,
"mae_above_2p5": 0.4529809369770209,
"brier_warn": 0.009090606171021184,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 1.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.8698313890426705
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 5.0,
"peak_level": 7.54,
"peak_pred_24h_before": 6.650326949981244
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2023,
"n_train": 37524,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.1489843074454942,
"mae_above_2p5": null,
"brier_warn": 1.882373500761472e-08,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.16106550376855028,
"mae_above_2p5": null,
"brier_warn": 0.0001779564366636896,
"events": [],
"false_alarm_episodes": 1
},
"rise_weighted": {
"mae": 0.1688589840007776,
"mae_above_2p5": null,
"brier_warn": 0.0012753245446952602,
"events": [],
"false_alarm_episodes": 1
},
"rise_quantile": {
"mae": 0.1510624979748974,
"mae_above_2p5": null,
"brier_warn": 0.0005959001115900152,
"events": [],
"false_alarm_episodes": 1
}
}
},
{
"year": 2024,
"n_train": 46308,
"n_test": 4058,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"peak_ts": "2024-09-26T00:00:00",
"peak_level": 8.27
},
{
"crossing": "2024-09-30T03:00:00",
"peak_ts": "2024-09-30T06:00:00",
"peak_level": 5.99
},
{
"crossing": "2024-10-03T06:00:00",
"peak_ts": "2024-10-05T07:00:00",
"peak_level": 9.93
}
],
"variants": {
"baseline_abs": {
"mae": 0.193584781859895,
"mae_above_2p5": 0.5381754200484403,
"brier_warn": 0.018292133172974026,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 5.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.11
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 0.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.42
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 0.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.86
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.17314177863843333,
"mae_above_2p5": 0.48294427141283425,
"brier_warn": 0.016091510788709233,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 10.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.167139790234238
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 10.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.618725525893519
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 69.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.889819144742388
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.18833560689452924,
"mae_above_2p5": 0.4948752445774323,
"brier_warn": 0.01622076553986666,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 11.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.283961881370446
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 10.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.783590096006894
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 69.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.922098265471745
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.1644888402154109,
"mae_above_2p5": 0.46461119464587947,
"brier_warn": 0.013172046828124688,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 11.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.348489352370365
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 8.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.42
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 2.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.91343423984052
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2025,
"n_train": 54577,
"n_test": 4392,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"peak_ts": "2025-09-27T21:00:00",
"peak_level": 6.64
},
{
"crossing": "2025-10-03T06:00:00",
"peak_ts": "2025-10-03T12:00:00",
"peak_level": 6.14
}
],
"variants": {
"baseline_abs": {
"mae": 0.2036944006891808,
"mae_above_2p5": 0.40787978637525985,
"brier_warn": 0.014130122065259989,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 14.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 41.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.703778873833753
}
],
"false_alarm_episodes": 2
},
"rise": {
"mae": 0.21617454799997243,
"mae_above_2p5": 0.406245211140301,
"brier_warn": 0.012587550711857222,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 14.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 47.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.953951787788988
}
],
"false_alarm_episodes": 1
},
"rise_weighted": {
"mae": 0.21768864574559124,
"mae_above_2p5": 0.41071503357135775,
"brier_warn": 0.014605838473436519,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 13.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 49.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.606011827978066
}
],
"false_alarm_episodes": 1
},
"rise_quantile": {
"mae": 0.18196250028633115,
"mae_above_2p5": 0.3987262421172889,
"brier_warn": 0.011560937993250782,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 13.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 42.0,
"peak_level": 6.14,
"peak_pred_24h_before": 6.00597561680636
}
],
"false_alarm_episodes": 1
}
}
}
]
}
]
scripts/backtest_render.py
+14 -4
View File
@@ -46,14 +46,23 @@ RED = "#d9534f"
def fit_backtest_model(df_long: pd.DataFrame, train_end: str):
"""Train the 24 h regression + warning heads on rows <= train_end only."""
X, Y, _meta = features.build_matrix(df_long, STATION, (HORIZON,))
"""Train the 24 h regression + warning heads on rows <= train_end only.
Mirrors the deployed hgb-v2 pipeline: the regression head learns the RISE
over the current level (rolling-origin evaluation 2026-08-12 showed this
moves first-alert leads from ~0 h to +6..+46 h); label statistics are
bounded to the training cutoff.
"""
X, Y, _meta = features.build_matrix(
df_long, STATION, (HORIZON,), stats_end=train_end
)
train_mask = X.index <= pd.Timestamp(train_end)
X_train, Y_train = X.loc[train_mask], Y.loc[train_mask]
max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}"
reg_rows = Y_train[max_col].notna()
reg = _make_regressor().fit(X_train.loc[reg_rows], Y_train.loc[reg_rows, max_col])
rise = Y_train.loc[reg_rows, max_col] - X_train.loc[reg_rows, "level"]
reg = _make_regressor().fit(X_train.loc[reg_rows], rise)
warn_rows = Y_train[warn_col].notna()
clf = _make_classifier().fit(
X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int)
@@ -70,7 +79,8 @@ def event_series(df_long, X, reg, clf, window_start: str, window_end: str):
Xw = X.loc[window_start:window_end]
forecasts = pd.DataFrame(index=Xw.index)
forecasts["pred_max"] = reg.predict(Xw)
# reg predicts the rise; add the current level back (as serving does)
forecasts["pred_max"] = reg.predict(Xw) + Xw["level"].to_numpy()
# Belt-and-braces probability: the classifier OR the regression-sigmoid,
# whichever is more alarmed. The classifier alone proved unreliable on
# out-of-distribution extremes (silent on the 2024 record flood).
src/ml/predict.py
+5 -1
View File
@@ -161,7 +161,11 @@ def _model_forecast(
if reg is None:
results.append(None)
continue
predicted_max = max(float(reg.predict(feature_row)[0]), current_level)
raw_prediction = float(reg.predict(feature_row)[0])
if bundle.get("regression_target") == "rise":
# v2 bundles predict the rise over the current level
raw_prediction += current_level
predicted_max = max(raw_prediction, current_level)
sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
# Belt-and-braces: the classifier head OR the regression-sigmoid path,
src/ml/train.py
+22 -7
View File
@@ -126,7 +126,8 @@ def _p_warning_series(
"""Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
if head is not None:
return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
predicted_max = pd.Series(reg.predict(X), index=X.index)
# reg predicts the RISE over current level; add the level back
predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"]
return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
@@ -240,14 +241,22 @@ def train_station(
)
horizon_metrics: dict = {}
# --- regression head (max level) ---
# --- regression head (rise to future max) ---
# Target = future max MINUS current level ("rise"). Rises are far more
# stationary than absolute stages, which softens the cannot-exceed-
# training-max ceiling: on the rolling-origin harness (2026-08-12) the
# rise target moved P.1 first-alert leads from +0h to +6/+46h and cut
# the 2024 record-peak underprediction. Prediction = rise + level.
reg_labeled = eval_Y[max_col].notna()
reg = None
if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
rise_target = (
eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"]
)
reg = _safe_fit(
_make_regressor(hgb_overrides),
eval_X.loc[reg_labeled],
eval_Y.loc[reg_labeled, max_col],
rise_target,
f"max_{h}",
skipped_heads,
)
@@ -259,7 +268,10 @@ def train_station(
test_labeled = Y_test[max_col].notna()
if test_labeled.sum() > 0:
y_true = Y_test.loc[test_labeled, max_col]
y_pred = reg.predict(X_test.loc[test_labeled])
y_pred = (
reg.predict(X_test.loc[test_labeled])
+ X_test.loc[test_labeled, "level"].to_numpy()
)
residuals = y_true.to_numpy() - y_pred
sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
horizon_metrics["n_test"] = int(test_labeled.sum())
@@ -367,7 +379,7 @@ def train_station(
reg = _safe_fit(
_make_regressor(hgb_overrides),
X.loc[labeled],
Y.loc[labeled, max_col],
Y.loc[labeled, max_col] - X.loc[labeled, "level"], # rise target
head_key,
skipped_heads,
)
@@ -401,7 +413,10 @@ def train_station(
bundle = {
"station_code": station,
"model_version": f"hgb-v1+{_git_short_sha()}",
"model_version": f"hgb-v2+{_git_short_sha()}",
# v2: regression heads predict the RISE over the current level; the
# serving side must add the level back. Old v1 bundles lack this key.
"regression_target": "rise",
"trained_at": datetime.datetime.now().isoformat(),
"sklearn_version": sklearn.__version__,
"feature_names": feature_names,
@@ -428,7 +443,7 @@ def train_all(
"""Train and save every requested station's models. Returns the metrics.json payload."""
models_dir = Path(models_dir)
models_dir.mkdir(parents=True, exist_ok=True)
model_version = f"hgb-v1+{_git_short_sha()}"
model_version = f"hgb-v2+{_git_short_sha()}"
station_results: Dict[str, dict] = {}
for station in stations: