feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s

Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103;
results in models/eval_variants.json) showed the absolute-level target
alerting AT the crossing on essentially every event, while the rise
target (future max - current level, level added back at serving) gives
+6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than
weighted/quantile variants, and ~11% better MAE. Weighted and quantile
variants rejected: more false alarms, no Brier-score calibration gain.

Ported to production: train.py fits rise in both eval and refit passes
(sigma/metrics computed in absolute space), bundles stamped hgb-v2 with
regression_target='rise', predict.py adds the level back for v2 and
stays compatible with v1 bundles, backtest_render.py mirrors the same
math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h
BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and
the record-peak underprediction is gone (rise models can exceed the
training max). The >=12h acceptance gate still fails honestly at +6h —
closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at
6/12/24h; docs updated throughout.
This commit is contained in:
2026-08-12 15:43:29 +07:00
parent a0086086a2
commit 21e9d2e114
8 changed files with 814 additions and 44 deletions
+50 -32
View File
@@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**:
| Head | Type | Target | | Head | Type | Target |
|---|---|---| |---|---|---|
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] | | `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) |
| `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] | | `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
| `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] | | `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
@@ -261,14 +261,19 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93%
between them, roughly doubling the test rows and collapsing the warning base between them, roughly doubling the test rows and collapsing the warning base
rates). rates).
**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB **Current model** `hgb-v2` (rise target), generated 2026-08-12 on the
(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1: gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12.
P.1:
| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate | | Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% | | 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% |
| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% | | 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% |
| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% | | 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% |
(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored
5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved
every regression figure; PR-AUC belongs to the unchanged classifier heads.)
Level accuracy improved; standalone classifier discrimination did not survive Level accuracy improved; standalone classifier discrimination did not survive
the data change (which is why serving is now `max(classifier, sigmoid)` — see the data change (which is why serving is now `max(classifier, sigmoid)` — see
@@ -327,6 +332,20 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
> `scripts/backtest_render.py` regenerates all three charts and fails its > `scripts/backtest_render.py` regenerates all three charts and fails its
> acceptance gate while the 2024 lead stays under 12 h — keeping this page > acceptance gate while the 2024 lead stays under 12 h — keeping this page
> honest is now automatic. > honest is now automatic.
>
> **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin,
> event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon
> 2021-2025) compared the absolute-level target against rise-target variants.
> The rise target — regression predicts *future max minus current level*, the
> level is added back at serving — won decisively and is now deployed as
> `hgb-v2`: the regenerated charts below show the 2024 first alert moving from
> 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert
> from at-crossing to **45 h early**, the record-peak underprediction
> eliminated (the model now slightly overshoots 5.30 m rather than capping
> ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and
> quantile variants were evaluated and rejected (more false alarms, no
> calibration gain by Brier score). The ≥12 h acceptance gate still fails at
> +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
### The September 2025 flood, as the deployed configuration saw it ### The September 2025 flood, as the deployed configuration saw it
@@ -334,14 +353,14 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
through 2024](img/backtest-2025-p1.png) through 2024](img/backtest-2025-p1.png)
This uses the deployed configuration (train ≤ 2024-12-31) on an event it never This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
re-examination note above; the original one-off render, trained on the sparser gap-filled data — see the re-examination note above for the history of these
data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45
today's fuller dataset the retrained equivalent first alerts at **18:00 on hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is
27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The partly the twin-crest shape of this event (the near-miss 3.51 m crest of
discrimination remains good: the near-miss 3.51 m crest on 26 September never 26 September keeps the 24 h-peak forecast near the line), so read it as
triggers, the probability fires only for the real event, and it stands down as "the model was correctly alarmed through the whole double crest", not as a
the water recedes. general 45 h capability.
### Headline validation: the October 2024 record flood ### Headline validation: the October 2024 record flood
@@ -353,29 +372,28 @@ followed. This is the closest thing to a real operational test available.
![Observed P.1 level vs the model's 24 h-ahead predicted peak through the ![Observed P.1 level vs the model's 24 h-ahead predicted peak through the
October 2024 flood, with the warning probability below](img/backtest-2024-p1.png) October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
The render above shows the whole event hour by hour *(regenerated 2026-08-11 The render above shows the whole event hour by hour *(regenerated 2026-08-12
on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead with the hgb-v2 rise target)*. Top: the observed level (blue) against the
predicted peak the model issued at each hour (amber, dashed) — the amber line 24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the
tracks both flood waves but no longer clearly leads the first one. Bottom: the amber line now leads the blue one into both flood waves. Bottom: the
belt-and-braces probability of flooding within 24 h; on the fuller data its belt-and-braces probability of flooding within 24 h; the **first alert comes
**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00 at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays
24 September crossing**, then stays correctly alarmed through the October correctly alarmed through the October record wave. The predicted peak now
record wave. Also visible, honestly: the predicted peak tops out well short of slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the
the actual 5.30 m record (the extreme-compression limitation discussed below). rise target removed the cannot-exceed-training-max ceiling. The same historic
The same historic model track drives the dashboard's "Replay Oct 2024 flood" model track drives the dashboard's "Replay Oct 2024 flood" feature.
feature.
![Hour-by-hour detail of the detection window, 2228 September ![Hour-by-hour detail of the detection window, 2228 September
2024](img/backtest-2024-p1-detail.png) 2024](img/backtest-2024-p1-detail.png)
The hour-by-hour detail of the detection window *(regenerated 2026-08-11)* The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
shows the corrected sequence: the river crosses 3.70 m at **17:00 on hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on
24 September** (the hours recovered by gap-filling; independently confirmed by 24 September** (the hours recovered by gap-filling; independently confirmed by
the HII sensor at the same bridge), while the retrained model's probability the HII sensor at the same bridge), and the model's probability crosses 0.5 at
only crosses 0.5 at **11:00 on 25 September**. The original render — sparser **11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under
data, different trained model — alerted at 01:00 on 24 September against an the absolute-level target this alert came 18 hours *after* the crossing; the
apparent 01:00 25 September crossing. Closing this real gap is what the rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance
rainfall features and rise-target work are for. gate demands is what the rainfall features are for.
The event bullets below quote the original (pre-gap-fill) evaluation of the The event bullets below quote the original (pre-gap-fill) evaluation of the
deployed model and are kept for the historical record — see the re-examination deployed model and are kept for the historical record — see the re-examination
Binary file not shown.

Before

Width:  |  Height:  |  Size: 118 KiB

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 147 KiB

After

Width:  |  Height:  |  Size: 142 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 141 KiB

After

Width:  |  Height:  |  Size: 138 KiB

+723
View File
@@ -0,0 +1,723 @@
[
{
"station": "P.1",
"warn_thr": 3.7,
"folds": [
{
"year": 2021,
"n_train": 20024,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.073069548799388,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.070970681677648,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.07394501467643348,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.07289422294875283,
"mae_above_2p5": null,
"brier_warn": 6.870660393452143e-17,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2022,
"n_train": 28782,
"n_test": 4392,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"peak_ts": "2022-10-03T15:00:00",
"peak_level": 4.65
}
],
"variants": {
"baseline_abs": {
"mae": 0.09128484356553154,
"mae_above_2p5": 0.2856025611084259,
"brier_warn": 0.005440790049399641,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 0.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.4
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.08489734638010188,
"mae_above_2p5": 0.24517428534843191,
"brier_warn": 0.004136576477574926,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 6.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.870617057762467
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.08451609397908402,
"mae_above_2p5": 0.24677443475763222,
"brier_warn": 0.0041231071254735656,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 6.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.9060484870307466
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.08412283956084679,
"mae_above_2p5": 0.2653552568120815,
"brier_warn": 0.004349073127074364,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 3.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.641048749261635
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2023,
"n_train": 37537,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.08161389876263064,
"mae_above_2p5": 0.1335546690803318,
"brier_warn": 3.284466170848282e-09,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.07647507344443248,
"mae_above_2p5": 0.12593127745781565,
"brier_warn": 3.19529506149809e-11,
"events": [],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.07633405851952908,
"mae_above_2p5": 0.12234064428755373,
"brier_warn": 9.093694748477163e-10,
"events": [],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.07460579730428984,
"mae_above_2p5": 0.11243109915383911,
"brier_warn": 8.148092046865233e-08,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2024,
"n_train": 46321,
"n_test": 4392,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"peak_ts": "2024-09-26T02:00:00",
"peak_level": 4.93
},
{
"crossing": "2024-10-03T09:00:00",
"peak_ts": "2024-10-05T12:00:00",
"peak_level": 5.3
}
],
"variants": {
"baseline_abs": {
"mae": 0.11011679980584997,
"mae_above_2p5": 0.28849258735632527,
"brier_warn": 0.010963910189916666,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 0.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.39
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 0.0,
"peak_level": 5.3,
"peak_pred_24h_before": 4.9
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.10075626719251345,
"mae_above_2p5": 0.26384939692286363,
"brier_warn": 0.011608221543810462,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 6.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.534709676862131
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 3.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.056619694027486
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.10644718990351541,
"mae_above_2p5": 0.2848978622398853,
"brier_warn": 0.013872775238121434,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 9.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.502016075573465
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 72.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.123356409745132
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.09617759284936986,
"mae_above_2p5": 0.271753261978628,
"brier_warn": 0.011660822672241142,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 4.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.532042588397363
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 2.0,
"peak_level": 5.3,
"peak_pred_24h_before": 4.956335200519836
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2025,
"n_train": 55081,
"n_test": 4392,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"peak_ts": "2025-09-27T22:00:00",
"peak_level": 3.93
}
],
"variants": {
"baseline_abs": {
"mae": 0.15985415338980086,
"mae_above_2p5": 0.277881996645181,
"brier_warn": 0.009824391159106445,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 0.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.2561560254201525
}
],
"false_alarm_episodes": 2
},
"rise": {
"mae": 0.13102686839694494,
"mae_above_2p5": 0.2729944665011028,
"brier_warn": 0.010889835530782944,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 46.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.232029710676319
}
],
"false_alarm_episodes": 1
},
"rise_weighted": {
"mae": 0.13121120517634968,
"mae_above_2p5": 0.27796411411759725,
"brier_warn": 0.009393099541680463,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 46.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 3
},
"rise_quantile": {
"mae": 0.11678707793162903,
"mae_above_2p5": 0.28528124941530186,
"brier_warn": 0.013009605253899563,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 45.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 3
}
}
}
]
},
{
"station": "P.103",
"warn_thr": 5.95,
"folds": [
{
"year": 2021,
"n_train": 20009,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.1517830652111977,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.15279228710793116,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.17632375724441426,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.13593873552532,
"mae_above_2p5": null,
"brier_warn": 1.6481640122270367e-09,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2022,
"n_train": 28769,
"n_test": 4392,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"peak_ts": "2022-08-14T08:00:00",
"peak_level": 6.09
},
{
"crossing": "2022-10-02T16:00:00",
"peak_ts": "2022-10-03T16:00:00",
"peak_level": 7.54
}
],
"variants": {
"baseline_abs": {
"mae": 0.15960776318568,
"mae_above_2p5": 0.5240325509357986,
"brier_warn": 0.010134410144044625,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 0.0,
"peak_level": 6.09,
"peak_pred_24h_before": 5.01930531142127
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 0.0,
"peak_level": 7.54,
"peak_pred_24h_before": 6.01
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.15389446229226025,
"mae_above_2p5": 0.41887094569113953,
"brier_warn": 0.007740077230532647,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 6.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.837975953559253
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 9.0,
"peak_level": 7.54,
"peak_pred_24h_before": 7.168225721504108
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.1498671765711928,
"mae_above_2p5": 0.40088103598066704,
"brier_warn": 0.00801963186023152,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 6.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.81
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 8.0,
"peak_level": 7.54,
"peak_pred_24h_before": 7.002000385945215
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.13833442255794418,
"mae_above_2p5": 0.4529809369770209,
"brier_warn": 0.009090606171021184,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 1.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.8698313890426705
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 5.0,
"peak_level": 7.54,
"peak_pred_24h_before": 6.650326949981244
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2023,
"n_train": 37524,
"n_test": 4392,
"events": [],
"variants": {
"baseline_abs": {
"mae": 0.1489843074454942,
"mae_above_2p5": null,
"brier_warn": 1.882373500761472e-08,
"events": [],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.16106550376855028,
"mae_above_2p5": null,
"brier_warn": 0.0001779564366636896,
"events": [],
"false_alarm_episodes": 1
},
"rise_weighted": {
"mae": 0.1688589840007776,
"mae_above_2p5": null,
"brier_warn": 0.0012753245446952602,
"events": [],
"false_alarm_episodes": 1
},
"rise_quantile": {
"mae": 0.1510624979748974,
"mae_above_2p5": null,
"brier_warn": 0.0005959001115900152,
"events": [],
"false_alarm_episodes": 1
}
}
},
{
"year": 2024,
"n_train": 46308,
"n_test": 4058,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"peak_ts": "2024-09-26T00:00:00",
"peak_level": 8.27
},
{
"crossing": "2024-09-30T03:00:00",
"peak_ts": "2024-09-30T06:00:00",
"peak_level": 5.99
},
{
"crossing": "2024-10-03T06:00:00",
"peak_ts": "2024-10-05T07:00:00",
"peak_level": 9.93
}
],
"variants": {
"baseline_abs": {
"mae": 0.193584781859895,
"mae_above_2p5": 0.5381754200484403,
"brier_warn": 0.018292133172974026,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 5.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.11
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 0.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.42
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 0.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.86
}
],
"false_alarm_episodes": 0
},
"rise": {
"mae": 0.17314177863843333,
"mae_above_2p5": 0.48294427141283425,
"brier_warn": 0.016091510788709233,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 10.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.167139790234238
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 10.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.618725525893519
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 69.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.889819144742388
}
],
"false_alarm_episodes": 0
},
"rise_weighted": {
"mae": 0.18833560689452924,
"mae_above_2p5": 0.4948752445774323,
"brier_warn": 0.01622076553986666,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 11.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.283961881370446
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 10.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.783590096006894
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 69.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.922098265471745
}
],
"false_alarm_episodes": 0
},
"rise_quantile": {
"mae": 0.1644888402154109,
"mae_above_2p5": 0.46461119464587947,
"brier_warn": 0.013172046828124688,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 11.0,
"peak_level": 8.27,
"peak_pred_24h_before": 7.348489352370365
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 8.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.42
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 2.0,
"peak_level": 9.93,
"peak_pred_24h_before": 7.91343423984052
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2025,
"n_train": 54577,
"n_test": 4392,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"peak_ts": "2025-09-27T21:00:00",
"peak_level": 6.64
},
{
"crossing": "2025-10-03T06:00:00",
"peak_ts": "2025-10-03T12:00:00",
"peak_level": 6.14
}
],
"variants": {
"baseline_abs": {
"mae": 0.2036944006891808,
"mae_above_2p5": 0.40787978637525985,
"brier_warn": 0.014130122065259989,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 14.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 41.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.703778873833753
}
],
"false_alarm_episodes": 2
},
"rise": {
"mae": 0.21617454799997243,
"mae_above_2p5": 0.406245211140301,
"brier_warn": 0.012587550711857222,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 14.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 47.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.953951787788988
}
],
"false_alarm_episodes": 1
},
"rise_weighted": {
"mae": 0.21768864574559124,
"mae_above_2p5": 0.41071503357135775,
"brier_warn": 0.014605838473436519,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 13.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 49.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.606011827978066
}
],
"false_alarm_episodes": 1
},
"rise_quantile": {
"mae": 0.18196250028633115,
"mae_above_2p5": 0.3987262421172889,
"brier_warn": 0.011560937993250782,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 13.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 42.0,
"peak_level": 6.14,
"peak_pred_24h_before": 6.00597561680636
}
],
"false_alarm_episodes": 1
}
}
}
]
}
]
+14 -4
View File
@@ -46,14 +46,23 @@ RED = "#d9534f"
def fit_backtest_model(df_long: pd.DataFrame, train_end: str): def fit_backtest_model(df_long: pd.DataFrame, train_end: str):
"""Train the 24 h regression + warning heads on rows <= train_end only.""" """Train the 24 h regression + warning heads on rows <= train_end only.
X, Y, _meta = features.build_matrix(df_long, STATION, (HORIZON,))
Mirrors the deployed hgb-v2 pipeline: the regression head learns the RISE
over the current level (rolling-origin evaluation 2026-08-12 showed this
moves first-alert leads from ~0 h to +6..+46 h); label statistics are
bounded to the training cutoff.
"""
X, Y, _meta = features.build_matrix(
df_long, STATION, (HORIZON,), stats_end=train_end
)
train_mask = X.index <= pd.Timestamp(train_end) train_mask = X.index <= pd.Timestamp(train_end)
X_train, Y_train = X.loc[train_mask], Y.loc[train_mask] X_train, Y_train = X.loc[train_mask], Y.loc[train_mask]
max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}" max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}"
reg_rows = Y_train[max_col].notna() reg_rows = Y_train[max_col].notna()
reg = _make_regressor().fit(X_train.loc[reg_rows], Y_train.loc[reg_rows, max_col]) rise = Y_train.loc[reg_rows, max_col] - X_train.loc[reg_rows, "level"]
reg = _make_regressor().fit(X_train.loc[reg_rows], rise)
warn_rows = Y_train[warn_col].notna() warn_rows = Y_train[warn_col].notna()
clf = _make_classifier().fit( clf = _make_classifier().fit(
X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int) X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int)
@@ -70,7 +79,8 @@ def event_series(df_long, X, reg, clf, window_start: str, window_end: str):
Xw = X.loc[window_start:window_end] Xw = X.loc[window_start:window_end]
forecasts = pd.DataFrame(index=Xw.index) forecasts = pd.DataFrame(index=Xw.index)
forecasts["pred_max"] = reg.predict(Xw) # reg predicts the rise; add the current level back (as serving does)
forecasts["pred_max"] = reg.predict(Xw) + Xw["level"].to_numpy()
# Belt-and-braces probability: the classifier OR the regression-sigmoid, # Belt-and-braces probability: the classifier OR the regression-sigmoid,
# whichever is more alarmed. The classifier alone proved unreliable on # whichever is more alarmed. The classifier alone proved unreliable on
# out-of-distribution extremes (silent on the 2024 record flood). # out-of-distribution extremes (silent on the 2024 record flood).
+5 -1
View File
@@ -161,7 +161,11 @@ def _model_forecast(
if reg is None: if reg is None:
results.append(None) results.append(None)
continue continue
predicted_max = max(float(reg.predict(feature_row)[0]), current_level) raw_prediction = float(reg.predict(feature_row)[0])
if bundle.get("regression_target") == "rise":
# v2 bundles predict the rise over the current level
raw_prediction += current_level
predicted_max = max(raw_prediction, current_level)
sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA) sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
# Belt-and-braces: the classifier head OR the regression-sigmoid path, # Belt-and-braces: the classifier head OR the regression-sigmoid path,
+22 -7
View File
@@ -126,7 +126,8 @@ def _p_warning_series(
"""Model score if a classifier head exists, else the sigmoid-derived fallback probability.""" """Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
if head is not None: if head is not None:
return pd.Series(head.predict_proba(X)[:, 1], index=X.index) return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
predicted_max = pd.Series(reg.predict(X), index=X.index) # reg predicts the RISE over current level; add the level back
predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"]
return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma)) return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
@@ -240,14 +241,22 @@ def train_station(
) )
horizon_metrics: dict = {} horizon_metrics: dict = {}
# --- regression head (max level) --- # --- regression head (rise to future max) ---
# Target = future max MINUS current level ("rise"). Rises are far more
# stationary than absolute stages, which softens the cannot-exceed-
# training-max ceiling: on the rolling-origin harness (2026-08-12) the
# rise target moved P.1 first-alert leads from +0h to +6/+46h and cut
# the 2024 record-peak underprediction. Prediction = rise + level.
reg_labeled = eval_Y[max_col].notna() reg_labeled = eval_Y[max_col].notna()
reg = None reg = None
if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD: if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
rise_target = (
eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"]
)
reg = _safe_fit( reg = _safe_fit(
_make_regressor(hgb_overrides), _make_regressor(hgb_overrides),
eval_X.loc[reg_labeled], eval_X.loc[reg_labeled],
eval_Y.loc[reg_labeled, max_col], rise_target,
f"max_{h}", f"max_{h}",
skipped_heads, skipped_heads,
) )
@@ -259,7 +268,10 @@ def train_station(
test_labeled = Y_test[max_col].notna() test_labeled = Y_test[max_col].notna()
if test_labeled.sum() > 0: if test_labeled.sum() > 0:
y_true = Y_test.loc[test_labeled, max_col] y_true = Y_test.loc[test_labeled, max_col]
y_pred = reg.predict(X_test.loc[test_labeled]) y_pred = (
reg.predict(X_test.loc[test_labeled])
+ X_test.loc[test_labeled, "level"].to_numpy()
)
residuals = y_true.to_numpy() - y_pred residuals = y_true.to_numpy() - y_pred
sigma_h = max(float(np.std(residuals)), MIN_SIGMA) sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
horizon_metrics["n_test"] = int(test_labeled.sum()) horizon_metrics["n_test"] = int(test_labeled.sum())
@@ -367,7 +379,7 @@ def train_station(
reg = _safe_fit( reg = _safe_fit(
_make_regressor(hgb_overrides), _make_regressor(hgb_overrides),
X.loc[labeled], X.loc[labeled],
Y.loc[labeled, max_col], Y.loc[labeled, max_col] - X.loc[labeled, "level"], # rise target
head_key, head_key,
skipped_heads, skipped_heads,
) )
@@ -401,7 +413,10 @@ def train_station(
bundle = { bundle = {
"station_code": station, "station_code": station,
"model_version": f"hgb-v1+{_git_short_sha()}", "model_version": f"hgb-v2+{_git_short_sha()}",
# v2: regression heads predict the RISE over the current level; the
# serving side must add the level back. Old v1 bundles lack this key.
"regression_target": "rise",
"trained_at": datetime.datetime.now().isoformat(), "trained_at": datetime.datetime.now().isoformat(),
"sklearn_version": sklearn.__version__, "sklearn_version": sklearn.__version__,
"feature_names": feature_names, "feature_names": feature_names,
@@ -428,7 +443,7 @@ def train_all(
"""Train and save every requested station's models. Returns the metrics.json payload.""" """Train and save every requested station's models. Returns the metrics.json payload."""
models_dir = Path(models_dir) models_dir = Path(models_dir)
models_dir.mkdir(parents=True, exist_ok=True) models_dir.mkdir(parents=True, exist_ok=True)
model_version = f"hgb-v1+{_git_short_sha()}" model_version = f"hgb-v2+{_git_short_sha()}"
station_results: Dict[str, dict] = {} station_results: Dict[str, dict] = {}
for station in stations: for station in stations: