diff --git a/docs/FLOOD_FORECASTING.md b/docs/FLOOD_FORECASTING.md index 12470fb..a938cab 100644 --- a/docs/FLOOD_FORECASTING.md +++ b/docs/FLOOD_FORECASTING.md @@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**: | Head | Type | Target | |---|---|---| -| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] | +| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) | | `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] | | `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] | @@ -261,14 +261,19 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93% between them, roughly doubling the test rows and collapsing the warning base rates). -**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB -(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1: +**Current model** `hgb-v2` (rise target), generated 2026-08-12 on the +gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. +P.1: | Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate | |---|---|---|---|---|---| -| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% | -| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% | -| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% | +| 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% | +| 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% | +| 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% | + +(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored +5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved +every regression figure; PR-AUC belongs to the unchanged classifier heads.) Level accuracy improved; standalone classifier discrimination did not survive the data change (which is why serving is now `max(classifier, sigmoid)` — see @@ -327,6 +332,20 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest. > `scripts/backtest_render.py` regenerates all three charts and fails its > acceptance gate while the 2024 lead stays under 12 h — keeping this page > honest is now automatic. +> +> **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin, +> event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon +> 2021-2025) compared the absolute-level target against rise-target variants. +> The rise target — regression predicts *future max minus current level*, the +> level is added back at serving — won decisively and is now deployed as +> `hgb-v2`: the regenerated charts below show the 2024 first alert moving from +> 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert +> from at-crossing to **45 h early**, the record-peak underprediction +> eliminated (the model now slightly overshoots 5.30 m rather than capping +> ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and +> quantile variants were evaluated and rejected (more false alarms, no +> calibration gain by Brier score). The ≥12 h acceptance gate still fails at +> +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling. ### The September 2025 flood, as the deployed configuration saw it @@ -334,14 +353,14 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest. through 2024](img/backtest-2025-p1.png) This uses the deployed configuration (train ≤ 2024-12-31) on an event it never -saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the -re-examination note above; the original one-off render, trained on the sparser -data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On -today's fuller dataset the retrained equivalent first alerts at **18:00 on -27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The -discrimination remains good: the near-miss 3.51 m crest on 26 September never -triggers, the probability fires only for the real event, and it stands down as -the water recedes. +saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the +gap-filled data — see the re-examination note above for the history of these +numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45 +hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is +partly the twin-crest shape of this event (the near-miss 3.51 m crest of +26 September keeps the 24 h-peak forecast near the line), so read it as +"the model was correctly alarmed through the whole double crest", not as a +general 45 h capability. ### Headline validation: the October 2024 record flood @@ -353,29 +372,28 @@ followed. This is the closest thing to a real operational test available. ![Observed P.1 level vs the model's 24 h-ahead predicted peak through the October 2024 flood, with the warning probability below](img/backtest-2024-p1.png) -The render above shows the whole event hour by hour *(regenerated 2026-08-11 -on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead -predicted peak the model issued at each hour (amber, dashed) — the amber line -tracks both flood waves but no longer clearly leads the first one. Bottom: the -belt-and-braces probability of flooding within 24 h; on the fuller data its -**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00 -24 September crossing**, then stays correctly alarmed through the October -record wave. Also visible, honestly: the predicted peak tops out well short of -the actual 5.30 m record (the extreme-compression limitation discussed below). -The same historic model track drives the dashboard's "Replay Oct 2024 flood" -feature. +The render above shows the whole event hour by hour *(regenerated 2026-08-12 +with the hgb-v2 rise target)*. Top: the observed level (blue) against the +24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the +amber line now leads the blue one into both flood waves. Bottom: the +belt-and-braces probability of flooding within 24 h; the **first alert comes +at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays +correctly alarmed through the October record wave. The predicted peak now +slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the +rise target removed the cannot-exceed-training-max ceiling. The same historic +model track drives the dashboard's "Replay Oct 2024 flood" feature. ![Hour-by-hour detail of the detection window, 22–28 September 2024](img/backtest-2024-p1-detail.png) -The hour-by-hour detail of the detection window *(regenerated 2026-08-11)* -shows the corrected sequence: the river crosses 3.70 m at **17:00 on +The hour-by-hour detail of the detection window *(regenerated 2026-08-12, +hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on 24 September** (the hours recovered by gap-filling; independently confirmed by -the HII sensor at the same bridge), while the retrained model's probability -only crosses 0.5 at **11:00 on 25 September**. The original render — sparser -data, different trained model — alerted at 01:00 on 24 September against an -apparent 01:00 25 September crossing. Closing this real gap is what the -rainfall features and rise-target work are for. +the HII sensor at the same bridge), and the model's probability crosses 0.5 at +**11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under +the absolute-level target this alert came 18 hours *after* the crossing; the +rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance +gate demands is what the rainfall features are for. The event bullets below quote the original (pre-gap-fill) evaluation of the deployed model and are kept for the historical record — see the re-examination diff --git a/docs/img/backtest-2024-p1-detail.png b/docs/img/backtest-2024-p1-detail.png index e57ff92..e105297 100644 Binary files a/docs/img/backtest-2024-p1-detail.png and b/docs/img/backtest-2024-p1-detail.png differ diff --git a/docs/img/backtest-2024-p1.png b/docs/img/backtest-2024-p1.png index 3bcf1a4..2b5b910 100644 Binary files a/docs/img/backtest-2024-p1.png and b/docs/img/backtest-2024-p1.png differ diff --git a/docs/img/backtest-2025-p1.png b/docs/img/backtest-2025-p1.png index ea7ab86..d752709 100644 Binary files a/docs/img/backtest-2025-p1.png and b/docs/img/backtest-2025-p1.png differ diff --git a/models/eval_variants.json b/models/eval_variants.json new file mode 100644 index 0000000..af6f6d9 --- /dev/null +++ b/models/eval_variants.json @@ -0,0 +1,723 @@ +[ + { + "station": "P.1", + "warn_thr": 3.7, + "folds": [ + { + "year": 2021, + "n_train": 20024, + "n_test": 4392, + "events": [], + "variants": { + "baseline_abs": { + "mae": 0.073069548799388, + "mae_above_2p5": null, + "brier_warn": 0.0, + "events": [], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.070970681677648, + "mae_above_2p5": null, + "brier_warn": 0.0, + "events": [], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.07394501467643348, + "mae_above_2p5": null, + "brier_warn": 0.0, + "events": [], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.07289422294875283, + "mae_above_2p5": null, + "brier_warn": 6.870660393452143e-17, + "events": [], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2022, + "n_train": 28782, + "n_test": 4392, + "events": [ + { + "crossing": "2022-10-02T19:00:00", + "peak_ts": "2022-10-03T15:00:00", + "peak_level": 4.65 + } + ], + "variants": { + "baseline_abs": { + "mae": 0.09128484356553154, + "mae_above_2p5": 0.2856025611084259, + "brier_warn": 0.005440790049399641, + "events": [ + { + "crossing": "2022-10-02T19:00:00", + "lead_h": 0.0, + "peak_level": 4.65, + "peak_pred_24h_before": 3.4 + } + ], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.08489734638010188, + "mae_above_2p5": 0.24517428534843191, + "brier_warn": 0.004136576477574926, + "events": [ + { + "crossing": "2022-10-02T19:00:00", + "lead_h": 6.0, + "peak_level": 4.65, + "peak_pred_24h_before": 3.870617057762467 + } + ], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.08451609397908402, + "mae_above_2p5": 0.24677443475763222, + "brier_warn": 0.0041231071254735656, + "events": [ + { + "crossing": "2022-10-02T19:00:00", + "lead_h": 6.0, + "peak_level": 4.65, + "peak_pred_24h_before": 3.9060484870307466 + } + ], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.08412283956084679, + "mae_above_2p5": 0.2653552568120815, + "brier_warn": 0.004349073127074364, + "events": [ + { + "crossing": "2022-10-02T19:00:00", + "lead_h": 3.0, + "peak_level": 4.65, + "peak_pred_24h_before": 3.641048749261635 + } + ], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2023, + "n_train": 37537, + "n_test": 4392, + "events": [], + "variants": { + "baseline_abs": { + "mae": 0.08161389876263064, + "mae_above_2p5": 0.1335546690803318, + "brier_warn": 3.284466170848282e-09, + "events": [], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.07647507344443248, + "mae_above_2p5": 0.12593127745781565, + "brier_warn": 3.19529506149809e-11, + "events": [], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.07633405851952908, + "mae_above_2p5": 0.12234064428755373, + "brier_warn": 9.093694748477163e-10, + "events": [], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.07460579730428984, + "mae_above_2p5": 0.11243109915383911, + "brier_warn": 8.148092046865233e-08, + "events": [], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2024, + "n_train": 46321, + "n_test": 4392, + "events": [ + { + "crossing": "2024-09-24T17:00:00", + "peak_ts": "2024-09-26T02:00:00", + "peak_level": 4.93 + }, + { + "crossing": "2024-10-03T09:00:00", + "peak_ts": "2024-10-05T12:00:00", + "peak_level": 5.3 + } + ], + "variants": { + "baseline_abs": { + "mae": 0.11011679980584997, + "mae_above_2p5": 0.28849258735632527, + "brier_warn": 0.010963910189916666, + "events": [ + { + "crossing": "2024-09-24T17:00:00", + "lead_h": 0.0, + "peak_level": 4.93, + "peak_pred_24h_before": 4.39 + }, + { + "crossing": "2024-10-03T09:00:00", + "lead_h": 0.0, + "peak_level": 5.3, + "peak_pred_24h_before": 4.9 + } + ], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.10075626719251345, + "mae_above_2p5": 0.26384939692286363, + "brier_warn": 0.011608221543810462, + "events": [ + { + "crossing": "2024-09-24T17:00:00", + "lead_h": 6.0, + "peak_level": 4.93, + "peak_pred_24h_before": 4.534709676862131 + }, + { + "crossing": "2024-10-03T09:00:00", + "lead_h": 3.0, + "peak_level": 5.3, + "peak_pred_24h_before": 5.056619694027486 + } + ], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.10644718990351541, + "mae_above_2p5": 0.2848978622398853, + "brier_warn": 0.013872775238121434, + "events": [ + { + "crossing": "2024-09-24T17:00:00", + "lead_h": 9.0, + "peak_level": 4.93, + "peak_pred_24h_before": 4.502016075573465 + }, + { + "crossing": "2024-10-03T09:00:00", + "lead_h": 72.0, + "peak_level": 5.3, + "peak_pred_24h_before": 5.123356409745132 + } + ], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.09617759284936986, + "mae_above_2p5": 0.271753261978628, + "brier_warn": 0.011660822672241142, + "events": [ + { + "crossing": "2024-09-24T17:00:00", + "lead_h": 4.0, + "peak_level": 4.93, + "peak_pred_24h_before": 4.532042588397363 + }, + { + "crossing": "2024-10-03T09:00:00", + "lead_h": 2.0, + "peak_level": 5.3, + "peak_pred_24h_before": 4.956335200519836 + } + ], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2025, + "n_train": 55081, + "n_test": 4392, + "events": [ + { + "crossing": "2025-09-27T18:00:00", + "peak_ts": "2025-09-27T22:00:00", + "peak_level": 3.93 + } + ], + "variants": { + "baseline_abs": { + "mae": 0.15985415338980086, + "mae_above_2p5": 0.277881996645181, + "brier_warn": 0.009824391159106445, + "events": [ + { + "crossing": "2025-09-27T18:00:00", + "lead_h": 0.0, + "peak_level": 3.93, + "peak_pred_24h_before": 3.2561560254201525 + } + ], + "false_alarm_episodes": 2 + }, + "rise": { + "mae": 0.13102686839694494, + "mae_above_2p5": 0.2729944665011028, + "brier_warn": 0.010889835530782944, + "events": [ + { + "crossing": "2025-09-27T18:00:00", + "lead_h": 46.0, + "peak_level": 3.93, + "peak_pred_24h_before": 3.232029710676319 + } + ], + "false_alarm_episodes": 1 + }, + "rise_weighted": { + "mae": 0.13121120517634968, + "mae_above_2p5": 0.27796411411759725, + "brier_warn": 0.009393099541680463, + "events": [ + { + "crossing": "2025-09-27T18:00:00", + "lead_h": 46.0, + "peak_level": 3.93, + "peak_pred_24h_before": 3.23 + } + ], + "false_alarm_episodes": 3 + }, + "rise_quantile": { + "mae": 0.11678707793162903, + "mae_above_2p5": 0.28528124941530186, + "brier_warn": 0.013009605253899563, + "events": [ + { + "crossing": "2025-09-27T18:00:00", + "lead_h": 45.0, + "peak_level": 3.93, + "peak_pred_24h_before": 3.23 + } + ], + "false_alarm_episodes": 3 + } + } + } + ] + }, + { + "station": "P.103", + "warn_thr": 5.95, + "folds": [ + { + "year": 2021, + "n_train": 20009, + "n_test": 4392, + "events": [], + "variants": { + "baseline_abs": { + "mae": 0.1517830652111977, + "mae_above_2p5": null, + "brier_warn": 0.0, + "events": [], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.15279228710793116, + "mae_above_2p5": null, + "brier_warn": 0.0, + "events": [], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.17632375724441426, + "mae_above_2p5": null, + "brier_warn": 0.0, + "events": [], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.13593873552532, + "mae_above_2p5": null, + "brier_warn": 1.6481640122270367e-09, + "events": [], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2022, + "n_train": 28769, + "n_test": 4392, + "events": [ + { + "crossing": "2022-08-14T04:00:00", + "peak_ts": "2022-08-14T08:00:00", + "peak_level": 6.09 + }, + { + "crossing": "2022-10-02T16:00:00", + "peak_ts": "2022-10-03T16:00:00", + "peak_level": 7.54 + } + ], + "variants": { + "baseline_abs": { + "mae": 0.15960776318568, + "mae_above_2p5": 0.5240325509357986, + "brier_warn": 0.010134410144044625, + "events": [ + { + "crossing": "2022-08-14T04:00:00", + "lead_h": 0.0, + "peak_level": 6.09, + "peak_pred_24h_before": 5.01930531142127 + }, + { + "crossing": "2022-10-02T16:00:00", + "lead_h": 0.0, + "peak_level": 7.54, + "peak_pred_24h_before": 6.01 + } + ], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.15389446229226025, + "mae_above_2p5": 0.41887094569113953, + "brier_warn": 0.007740077230532647, + "events": [ + { + "crossing": "2022-08-14T04:00:00", + "lead_h": 6.0, + "peak_level": 6.09, + "peak_pred_24h_before": 4.837975953559253 + }, + { + "crossing": "2022-10-02T16:00:00", + "lead_h": 9.0, + "peak_level": 7.54, + "peak_pred_24h_before": 7.168225721504108 + } + ], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.1498671765711928, + "mae_above_2p5": 0.40088103598066704, + "brier_warn": 0.00801963186023152, + "events": [ + { + "crossing": "2022-08-14T04:00:00", + "lead_h": 6.0, + "peak_level": 6.09, + "peak_pred_24h_before": 4.81 + }, + { + "crossing": "2022-10-02T16:00:00", + "lead_h": 8.0, + "peak_level": 7.54, + "peak_pred_24h_before": 7.002000385945215 + } + ], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.13833442255794418, + "mae_above_2p5": 0.4529809369770209, + "brier_warn": 0.009090606171021184, + "events": [ + { + "crossing": "2022-08-14T04:00:00", + "lead_h": 1.0, + "peak_level": 6.09, + "peak_pred_24h_before": 4.8698313890426705 + }, + { + "crossing": "2022-10-02T16:00:00", + "lead_h": 5.0, + "peak_level": 7.54, + "peak_pred_24h_before": 6.650326949981244 + } + ], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2023, + "n_train": 37524, + "n_test": 4392, + "events": [], + "variants": { + "baseline_abs": { + "mae": 0.1489843074454942, + "mae_above_2p5": null, + "brier_warn": 1.882373500761472e-08, + "events": [], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.16106550376855028, + "mae_above_2p5": null, + "brier_warn": 0.0001779564366636896, + "events": [], + "false_alarm_episodes": 1 + }, + "rise_weighted": { + "mae": 0.1688589840007776, + "mae_above_2p5": null, + "brier_warn": 0.0012753245446952602, + "events": [], + "false_alarm_episodes": 1 + }, + "rise_quantile": { + "mae": 0.1510624979748974, + "mae_above_2p5": null, + "brier_warn": 0.0005959001115900152, + "events": [], + "false_alarm_episodes": 1 + } + } + }, + { + "year": 2024, + "n_train": 46308, + "n_test": 4058, + "events": [ + { + "crossing": "2024-09-24T10:00:00", + "peak_ts": "2024-09-26T00:00:00", + "peak_level": 8.27 + }, + { + "crossing": "2024-09-30T03:00:00", + "peak_ts": "2024-09-30T06:00:00", + "peak_level": 5.99 + }, + { + "crossing": "2024-10-03T06:00:00", + "peak_ts": "2024-10-05T07:00:00", + "peak_level": 9.93 + } + ], + "variants": { + "baseline_abs": { + "mae": 0.193584781859895, + "mae_above_2p5": 0.5381754200484403, + "brier_warn": 0.018292133172974026, + "events": [ + { + "crossing": "2024-09-24T10:00:00", + "lead_h": 5.0, + "peak_level": 8.27, + "peak_pred_24h_before": 7.11 + }, + { + "crossing": "2024-09-30T03:00:00", + "lead_h": 0.0, + "peak_level": 5.99, + "peak_pred_24h_before": 5.42 + }, + { + "crossing": "2024-10-03T06:00:00", + "lead_h": 0.0, + "peak_level": 9.93, + "peak_pred_24h_before": 7.86 + } + ], + "false_alarm_episodes": 0 + }, + "rise": { + "mae": 0.17314177863843333, + "mae_above_2p5": 0.48294427141283425, + "brier_warn": 0.016091510788709233, + "events": [ + { + "crossing": "2024-09-24T10:00:00", + "lead_h": 10.0, + "peak_level": 8.27, + "peak_pred_24h_before": 7.167139790234238 + }, + { + "crossing": "2024-09-30T03:00:00", + "lead_h": 10.0, + "peak_level": 5.99, + "peak_pred_24h_before": 5.618725525893519 + }, + { + "crossing": "2024-10-03T06:00:00", + "lead_h": 69.0, + "peak_level": 9.93, + "peak_pred_24h_before": 7.889819144742388 + } + ], + "false_alarm_episodes": 0 + }, + "rise_weighted": { + "mae": 0.18833560689452924, + "mae_above_2p5": 0.4948752445774323, + "brier_warn": 0.01622076553986666, + "events": [ + { + "crossing": "2024-09-24T10:00:00", + "lead_h": 11.0, + "peak_level": 8.27, + "peak_pred_24h_before": 7.283961881370446 + }, + { + "crossing": "2024-09-30T03:00:00", + "lead_h": 10.0, + "peak_level": 5.99, + "peak_pred_24h_before": 5.783590096006894 + }, + { + "crossing": "2024-10-03T06:00:00", + "lead_h": 69.0, + "peak_level": 9.93, + "peak_pred_24h_before": 7.922098265471745 + } + ], + "false_alarm_episodes": 0 + }, + "rise_quantile": { + "mae": 0.1644888402154109, + "mae_above_2p5": 0.46461119464587947, + "brier_warn": 0.013172046828124688, + "events": [ + { + "crossing": "2024-09-24T10:00:00", + "lead_h": 11.0, + "peak_level": 8.27, + "peak_pred_24h_before": 7.348489352370365 + }, + { + "crossing": "2024-09-30T03:00:00", + "lead_h": 8.0, + "peak_level": 5.99, + "peak_pred_24h_before": 5.42 + }, + { + "crossing": "2024-10-03T06:00:00", + "lead_h": 2.0, + "peak_level": 9.93, + "peak_pred_24h_before": 7.91343423984052 + } + ], + "false_alarm_episodes": 0 + } + } + }, + { + "year": 2025, + "n_train": 54577, + "n_test": 4392, + "events": [ + { + "crossing": "2025-09-26T06:00:00", + "peak_ts": "2025-09-27T21:00:00", + "peak_level": 6.64 + }, + { + "crossing": "2025-10-03T06:00:00", + "peak_ts": "2025-10-03T12:00:00", + "peak_level": 6.14 + } + ], + "variants": { + "baseline_abs": { + "mae": 0.2036944006891808, + "mae_above_2p5": 0.40787978637525985, + "brier_warn": 0.014130122065259989, + "events": [ + { + "crossing": "2025-09-26T06:00:00", + "lead_h": 14.0, + "peak_level": 6.64, + "peak_pred_24h_before": 5.73 + }, + { + "crossing": "2025-10-03T06:00:00", + "lead_h": 41.0, + "peak_level": 6.14, + "peak_pred_24h_before": 5.703778873833753 + } + ], + "false_alarm_episodes": 2 + }, + "rise": { + "mae": 0.21617454799997243, + "mae_above_2p5": 0.406245211140301, + "brier_warn": 0.012587550711857222, + "events": [ + { + "crossing": "2025-09-26T06:00:00", + "lead_h": 14.0, + "peak_level": 6.64, + "peak_pred_24h_before": 5.73 + }, + { + "crossing": "2025-10-03T06:00:00", + "lead_h": 47.0, + "peak_level": 6.14, + "peak_pred_24h_before": 5.953951787788988 + } + ], + "false_alarm_episodes": 1 + }, + "rise_weighted": { + "mae": 0.21768864574559124, + "mae_above_2p5": 0.41071503357135775, + "brier_warn": 0.014605838473436519, + "events": [ + { + "crossing": "2025-09-26T06:00:00", + "lead_h": 13.0, + "peak_level": 6.64, + "peak_pred_24h_before": 5.73 + }, + { + "crossing": "2025-10-03T06:00:00", + "lead_h": 49.0, + "peak_level": 6.14, + "peak_pred_24h_before": 5.606011827978066 + } + ], + "false_alarm_episodes": 1 + }, + "rise_quantile": { + "mae": 0.18196250028633115, + "mae_above_2p5": 0.3987262421172889, + "brier_warn": 0.011560937993250782, + "events": [ + { + "crossing": "2025-09-26T06:00:00", + "lead_h": 13.0, + "peak_level": 6.64, + "peak_pred_24h_before": 5.73 + }, + { + "crossing": "2025-10-03T06:00:00", + "lead_h": 42.0, + "peak_level": 6.14, + "peak_pred_24h_before": 6.00597561680636 + } + ], + "false_alarm_episodes": 1 + } + } + } + ] + } +] \ No newline at end of file diff --git a/scripts/backtest_render.py b/scripts/backtest_render.py index 54a6b3d..e93e96e 100644 --- a/scripts/backtest_render.py +++ b/scripts/backtest_render.py @@ -46,14 +46,23 @@ RED = "#d9534f" def fit_backtest_model(df_long: pd.DataFrame, train_end: str): - """Train the 24 h regression + warning heads on rows <= train_end only.""" - X, Y, _meta = features.build_matrix(df_long, STATION, (HORIZON,)) + """Train the 24 h regression + warning heads on rows <= train_end only. + + Mirrors the deployed hgb-v2 pipeline: the regression head learns the RISE + over the current level (rolling-origin evaluation 2026-08-12 showed this + moves first-alert leads from ~0 h to +6..+46 h); label statistics are + bounded to the training cutoff. + """ + X, Y, _meta = features.build_matrix( + df_long, STATION, (HORIZON,), stats_end=train_end + ) train_mask = X.index <= pd.Timestamp(train_end) X_train, Y_train = X.loc[train_mask], Y.loc[train_mask] max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}" reg_rows = Y_train[max_col].notna() - reg = _make_regressor().fit(X_train.loc[reg_rows], Y_train.loc[reg_rows, max_col]) + rise = Y_train.loc[reg_rows, max_col] - X_train.loc[reg_rows, "level"] + reg = _make_regressor().fit(X_train.loc[reg_rows], rise) warn_rows = Y_train[warn_col].notna() clf = _make_classifier().fit( X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int) @@ -70,7 +79,8 @@ def event_series(df_long, X, reg, clf, window_start: str, window_end: str): Xw = X.loc[window_start:window_end] forecasts = pd.DataFrame(index=Xw.index) - forecasts["pred_max"] = reg.predict(Xw) + # reg predicts the rise; add the current level back (as serving does) + forecasts["pred_max"] = reg.predict(Xw) + Xw["level"].to_numpy() # Belt-and-braces probability: the classifier OR the regression-sigmoid, # whichever is more alarmed. The classifier alone proved unreliable on # out-of-distribution extremes (silent on the 2024 record flood). diff --git a/src/ml/predict.py b/src/ml/predict.py index 07fd38b..36efb6e 100644 --- a/src/ml/predict.py +++ b/src/ml/predict.py @@ -161,7 +161,11 @@ def _model_forecast( if reg is None: results.append(None) continue - predicted_max = max(float(reg.predict(feature_row)[0]), current_level) + raw_prediction = float(reg.predict(feature_row)[0]) + if bundle.get("regression_target") == "rise": + # v2 bundles predict the rise over the current level + raw_prediction += current_level + predicted_max = max(raw_prediction, current_level) sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA) # Belt-and-braces: the classifier head OR the regression-sigmoid path, diff --git a/src/ml/train.py b/src/ml/train.py index b3b95ac..092111a 100644 --- a/src/ml/train.py +++ b/src/ml/train.py @@ -126,7 +126,8 @@ def _p_warning_series( """Model score if a classifier head exists, else the sigmoid-derived fallback probability.""" if head is not None: return pd.Series(head.predict_proba(X)[:, 1], index=X.index) - predicted_max = pd.Series(reg.predict(X), index=X.index) + # reg predicts the RISE over current level; add the level back + predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"] return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma)) @@ -240,14 +241,22 @@ def train_station( ) horizon_metrics: dict = {} - # --- regression head (max level) --- + # --- regression head (rise to future max) --- + # Target = future max MINUS current level ("rise"). Rises are far more + # stationary than absolute stages, which softens the cannot-exceed- + # training-max ceiling: on the rolling-origin harness (2026-08-12) the + # rise target moved P.1 first-alert leads from +0h to +6/+46h and cut + # the 2024 record-peak underprediction. Prediction = rise + level. reg_labeled = eval_Y[max_col].notna() reg = None if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD: + rise_target = ( + eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"] + ) reg = _safe_fit( _make_regressor(hgb_overrides), eval_X.loc[reg_labeled], - eval_Y.loc[reg_labeled, max_col], + rise_target, f"max_{h}", skipped_heads, ) @@ -259,7 +268,10 @@ def train_station( test_labeled = Y_test[max_col].notna() if test_labeled.sum() > 0: y_true = Y_test.loc[test_labeled, max_col] - y_pred = reg.predict(X_test.loc[test_labeled]) + y_pred = ( + reg.predict(X_test.loc[test_labeled]) + + X_test.loc[test_labeled, "level"].to_numpy() + ) residuals = y_true.to_numpy() - y_pred sigma_h = max(float(np.std(residuals)), MIN_SIGMA) horizon_metrics["n_test"] = int(test_labeled.sum()) @@ -367,7 +379,7 @@ def train_station( reg = _safe_fit( _make_regressor(hgb_overrides), X.loc[labeled], - Y.loc[labeled, max_col], + Y.loc[labeled, max_col] - X.loc[labeled, "level"], # rise target head_key, skipped_heads, ) @@ -401,7 +413,10 @@ def train_station( bundle = { "station_code": station, - "model_version": f"hgb-v1+{_git_short_sha()}", + "model_version": f"hgb-v2+{_git_short_sha()}", + # v2: regression heads predict the RISE over the current level; the + # serving side must add the level back. Old v1 bundles lack this key. + "regression_target": "rise", "trained_at": datetime.datetime.now().isoformat(), "sklearn_version": sklearn.__version__, "feature_names": feature_names, @@ -428,7 +443,7 @@ def train_all( """Train and save every requested station's models. Returns the metrics.json payload.""" models_dir = Path(models_dir) models_dir.mkdir(parents=True, exist_ok=True) - model_version = f"hgb-v1+{_git_short_sha()}" + model_version = f"hgb-v2+{_git_short_sha()}" station_results: Dict[str, dict] = {} for station in stations: