feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.
This commit is contained in:
+50
-32
@@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**:
|
|||||||
|
|
||||||
| Head | Type | Target |
|
| Head | Type | Target |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] |
|
| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) |
|
||||||
| `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
|
| `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
|
||||||
| `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
|
| `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
|
||||||
|
|
||||||
@@ -261,14 +261,19 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93%
|
|||||||
between them, roughly doubling the test rows and collapsing the warning base
|
between them, roughly doubling the test rows and collapsing the warning base
|
||||||
rates).
|
rates).
|
||||||
|
|
||||||
**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB
|
**Current model** `hgb-v2` (rise target), generated 2026-08-12 on the
|
||||||
(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1:
|
gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12.
|
||||||
|
P.1:
|
||||||
|
|
||||||
| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
|
| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% |
|
| 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% |
|
||||||
| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% |
|
| 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% |
|
||||||
| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% |
|
| 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% |
|
||||||
|
|
||||||
|
(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored
|
||||||
|
5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved
|
||||||
|
every regression figure; PR-AUC belongs to the unchanged classifier heads.)
|
||||||
|
|
||||||
Level accuracy improved; standalone classifier discrimination did not survive
|
Level accuracy improved; standalone classifier discrimination did not survive
|
||||||
the data change (which is why serving is now `max(classifier, sigmoid)` — see
|
the data change (which is why serving is now `max(classifier, sigmoid)` — see
|
||||||
@@ -327,6 +332,20 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
|
|||||||
> `scripts/backtest_render.py` regenerates all three charts and fails its
|
> `scripts/backtest_render.py` regenerates all three charts and fails its
|
||||||
> acceptance gate while the 2024 lead stays under 12 h — keeping this page
|
> acceptance gate while the 2024 lead stays under 12 h — keeping this page
|
||||||
> honest is now automatic.
|
> honest is now automatic.
|
||||||
|
>
|
||||||
|
> **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin,
|
||||||
|
> event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon
|
||||||
|
> 2021-2025) compared the absolute-level target against rise-target variants.
|
||||||
|
> The rise target — regression predicts *future max minus current level*, the
|
||||||
|
> level is added back at serving — won decisively and is now deployed as
|
||||||
|
> `hgb-v2`: the regenerated charts below show the 2024 first alert moving from
|
||||||
|
> 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert
|
||||||
|
> from at-crossing to **45 h early**, the record-peak underprediction
|
||||||
|
> eliminated (the model now slightly overshoots 5.30 m rather than capping
|
||||||
|
> ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and
|
||||||
|
> quantile variants were evaluated and rejected (more false alarms, no
|
||||||
|
> calibration gain by Brier score). The ≥12 h acceptance gate still fails at
|
||||||
|
> +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
|
||||||
|
|
||||||
### The September 2025 flood, as the deployed configuration saw it
|
### The September 2025 flood, as the deployed configuration saw it
|
||||||
|
|
||||||
@@ -334,14 +353,14 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
|
|||||||
through 2024](img/backtest-2025-p1.png)
|
through 2024](img/backtest-2025-p1.png)
|
||||||
|
|
||||||
This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
|
This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
|
||||||
saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the
|
saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
|
||||||
re-examination note above; the original one-off render, trained on the sparser
|
gap-filled data — see the re-examination note above for the history of these
|
||||||
data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On
|
numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45
|
||||||
today's fuller dataset the retrained equivalent first alerts at **18:00 on
|
hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is
|
||||||
27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The
|
partly the twin-crest shape of this event (the near-miss 3.51 m crest of
|
||||||
discrimination remains good: the near-miss 3.51 m crest on 26 September never
|
26 September keeps the 24 h-peak forecast near the line), so read it as
|
||||||
triggers, the probability fires only for the real event, and it stands down as
|
"the model was correctly alarmed through the whole double crest", not as a
|
||||||
the water recedes.
|
general 45 h capability.
|
||||||
|
|
||||||
### Headline validation: the October 2024 record flood
|
### Headline validation: the October 2024 record flood
|
||||||
|
|
||||||
@@ -353,29 +372,28 @@ followed. This is the closest thing to a real operational test available.
|
|||||||

|
October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
|
||||||
|
|
||||||
The render above shows the whole event hour by hour *(regenerated 2026-08-11
|
The render above shows the whole event hour by hour *(regenerated 2026-08-12
|
||||||
on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead
|
with the hgb-v2 rise target)*. Top: the observed level (blue) against the
|
||||||
predicted peak the model issued at each hour (amber, dashed) — the amber line
|
24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the
|
||||||
tracks both flood waves but no longer clearly leads the first one. Bottom: the
|
amber line now leads the blue one into both flood waves. Bottom: the
|
||||||
belt-and-braces probability of flooding within 24 h; on the fuller data its
|
belt-and-braces probability of flooding within 24 h; the **first alert comes
|
||||||
**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00
|
at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays
|
||||||
24 September crossing**, then stays correctly alarmed through the October
|
correctly alarmed through the October record wave. The predicted peak now
|
||||||
record wave. Also visible, honestly: the predicted peak tops out well short of
|
slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the
|
||||||
the actual 5.30 m record (the extreme-compression limitation discussed below).
|
rise target removed the cannot-exceed-training-max ceiling. The same historic
|
||||||
The same historic model track drives the dashboard's "Replay Oct 2024 flood"
|
model track drives the dashboard's "Replay Oct 2024 flood" feature.
|
||||||
feature.
|
|
||||||
|
|
||||||

|
2024](img/backtest-2024-p1-detail.png)
|
||||||
|
|
||||||
The hour-by-hour detail of the detection window *(regenerated 2026-08-11)*
|
The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
|
||||||
shows the corrected sequence: the river crosses 3.70 m at **17:00 on
|
hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on
|
||||||
24 September** (the hours recovered by gap-filling; independently confirmed by
|
24 September** (the hours recovered by gap-filling; independently confirmed by
|
||||||
the HII sensor at the same bridge), while the retrained model's probability
|
the HII sensor at the same bridge), and the model's probability crosses 0.5 at
|
||||||
only crosses 0.5 at **11:00 on 25 September**. The original render — sparser
|
**11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under
|
||||||
data, different trained model — alerted at 01:00 on 24 September against an
|
the absolute-level target this alert came 18 hours *after* the crossing; the
|
||||||
apparent 01:00 25 September crossing. Closing this real gap is what the
|
rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance
|
||||||
rainfall features and rise-target work are for.
|
gate demands is what the rainfall features are for.
|
||||||
|
|
||||||
The event bullets below quote the original (pre-gap-fill) evaluation of the
|
The event bullets below quote the original (pre-gap-fill) evaluation of the
|
||||||
deployed model and are kept for the historical record — see the re-examination
|
deployed model and are kept for the historical record — see the re-examination
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 118 KiB After Width: | Height: | Size: 108 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 147 KiB After Width: | Height: | Size: 142 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 141 KiB After Width: | Height: | Size: 138 KiB |
@@ -0,0 +1,723 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"station": "P.1",
|
||||||
|
"warn_thr": 3.7,
|
||||||
|
"folds": [
|
||||||
|
{
|
||||||
|
"year": 2021,
|
||||||
|
"n_train": 20024,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.073069548799388,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.070970681677648,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.07394501467643348,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.07289422294875283,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 6.870660393452143e-17,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2022,
|
||||||
|
"n_train": 28782,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T19:00:00",
|
||||||
|
"peak_ts": "2022-10-03T15:00:00",
|
||||||
|
"peak_level": 4.65
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.09128484356553154,
|
||||||
|
"mae_above_2p5": 0.2856025611084259,
|
||||||
|
"brier_warn": 0.005440790049399641,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T19:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 4.65,
|
||||||
|
"peak_pred_24h_before": 3.4
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.08489734638010188,
|
||||||
|
"mae_above_2p5": 0.24517428534843191,
|
||||||
|
"brier_warn": 0.004136576477574926,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T19:00:00",
|
||||||
|
"lead_h": 6.0,
|
||||||
|
"peak_level": 4.65,
|
||||||
|
"peak_pred_24h_before": 3.870617057762467
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.08451609397908402,
|
||||||
|
"mae_above_2p5": 0.24677443475763222,
|
||||||
|
"brier_warn": 0.0041231071254735656,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T19:00:00",
|
||||||
|
"lead_h": 6.0,
|
||||||
|
"peak_level": 4.65,
|
||||||
|
"peak_pred_24h_before": 3.9060484870307466
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.08412283956084679,
|
||||||
|
"mae_above_2p5": 0.2653552568120815,
|
||||||
|
"brier_warn": 0.004349073127074364,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T19:00:00",
|
||||||
|
"lead_h": 3.0,
|
||||||
|
"peak_level": 4.65,
|
||||||
|
"peak_pred_24h_before": 3.641048749261635
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2023,
|
||||||
|
"n_train": 37537,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.08161389876263064,
|
||||||
|
"mae_above_2p5": 0.1335546690803318,
|
||||||
|
"brier_warn": 3.284466170848282e-09,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.07647507344443248,
|
||||||
|
"mae_above_2p5": 0.12593127745781565,
|
||||||
|
"brier_warn": 3.19529506149809e-11,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.07633405851952908,
|
||||||
|
"mae_above_2p5": 0.12234064428755373,
|
||||||
|
"brier_warn": 9.093694748477163e-10,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.07460579730428984,
|
||||||
|
"mae_above_2p5": 0.11243109915383911,
|
||||||
|
"brier_warn": 8.148092046865233e-08,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2024,
|
||||||
|
"n_train": 46321,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T17:00:00",
|
||||||
|
"peak_ts": "2024-09-26T02:00:00",
|
||||||
|
"peak_level": 4.93
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T09:00:00",
|
||||||
|
"peak_ts": "2024-10-05T12:00:00",
|
||||||
|
"peak_level": 5.3
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.11011679980584997,
|
||||||
|
"mae_above_2p5": 0.28849258735632527,
|
||||||
|
"brier_warn": 0.010963910189916666,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T17:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 4.93,
|
||||||
|
"peak_pred_24h_before": 4.39
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T09:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 5.3,
|
||||||
|
"peak_pred_24h_before": 4.9
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.10075626719251345,
|
||||||
|
"mae_above_2p5": 0.26384939692286363,
|
||||||
|
"brier_warn": 0.011608221543810462,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T17:00:00",
|
||||||
|
"lead_h": 6.0,
|
||||||
|
"peak_level": 4.93,
|
||||||
|
"peak_pred_24h_before": 4.534709676862131
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T09:00:00",
|
||||||
|
"lead_h": 3.0,
|
||||||
|
"peak_level": 5.3,
|
||||||
|
"peak_pred_24h_before": 5.056619694027486
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.10644718990351541,
|
||||||
|
"mae_above_2p5": 0.2848978622398853,
|
||||||
|
"brier_warn": 0.013872775238121434,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T17:00:00",
|
||||||
|
"lead_h": 9.0,
|
||||||
|
"peak_level": 4.93,
|
||||||
|
"peak_pred_24h_before": 4.502016075573465
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T09:00:00",
|
||||||
|
"lead_h": 72.0,
|
||||||
|
"peak_level": 5.3,
|
||||||
|
"peak_pred_24h_before": 5.123356409745132
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.09617759284936986,
|
||||||
|
"mae_above_2p5": 0.271753261978628,
|
||||||
|
"brier_warn": 0.011660822672241142,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T17:00:00",
|
||||||
|
"lead_h": 4.0,
|
||||||
|
"peak_level": 4.93,
|
||||||
|
"peak_pred_24h_before": 4.532042588397363
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T09:00:00",
|
||||||
|
"lead_h": 2.0,
|
||||||
|
"peak_level": 5.3,
|
||||||
|
"peak_pred_24h_before": 4.956335200519836
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2025,
|
||||||
|
"n_train": 55081,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-27T18:00:00",
|
||||||
|
"peak_ts": "2025-09-27T22:00:00",
|
||||||
|
"peak_level": 3.93
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.15985415338980086,
|
||||||
|
"mae_above_2p5": 0.277881996645181,
|
||||||
|
"brier_warn": 0.009824391159106445,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-27T18:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 3.93,
|
||||||
|
"peak_pred_24h_before": 3.2561560254201525
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 2
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.13102686839694494,
|
||||||
|
"mae_above_2p5": 0.2729944665011028,
|
||||||
|
"brier_warn": 0.010889835530782944,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-27T18:00:00",
|
||||||
|
"lead_h": 46.0,
|
||||||
|
"peak_level": 3.93,
|
||||||
|
"peak_pred_24h_before": 3.232029710676319
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.13121120517634968,
|
||||||
|
"mae_above_2p5": 0.27796411411759725,
|
||||||
|
"brier_warn": 0.009393099541680463,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-27T18:00:00",
|
||||||
|
"lead_h": 46.0,
|
||||||
|
"peak_level": 3.93,
|
||||||
|
"peak_pred_24h_before": 3.23
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 3
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.11678707793162903,
|
||||||
|
"mae_above_2p5": 0.28528124941530186,
|
||||||
|
"brier_warn": 0.013009605253899563,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-27T18:00:00",
|
||||||
|
"lead_h": 45.0,
|
||||||
|
"peak_level": 3.93,
|
||||||
|
"peak_pred_24h_before": 3.23
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 3
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"station": "P.103",
|
||||||
|
"warn_thr": 5.95,
|
||||||
|
"folds": [
|
||||||
|
{
|
||||||
|
"year": 2021,
|
||||||
|
"n_train": 20009,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.1517830652111977,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.15279228710793116,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.17632375724441426,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.13593873552532,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 1.6481640122270367e-09,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2022,
|
||||||
|
"n_train": 28769,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-08-14T04:00:00",
|
||||||
|
"peak_ts": "2022-08-14T08:00:00",
|
||||||
|
"peak_level": 6.09
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T16:00:00",
|
||||||
|
"peak_ts": "2022-10-03T16:00:00",
|
||||||
|
"peak_level": 7.54
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.15960776318568,
|
||||||
|
"mae_above_2p5": 0.5240325509357986,
|
||||||
|
"brier_warn": 0.010134410144044625,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-08-14T04:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 6.09,
|
||||||
|
"peak_pred_24h_before": 5.01930531142127
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T16:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 7.54,
|
||||||
|
"peak_pred_24h_before": 6.01
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.15389446229226025,
|
||||||
|
"mae_above_2p5": 0.41887094569113953,
|
||||||
|
"brier_warn": 0.007740077230532647,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-08-14T04:00:00",
|
||||||
|
"lead_h": 6.0,
|
||||||
|
"peak_level": 6.09,
|
||||||
|
"peak_pred_24h_before": 4.837975953559253
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T16:00:00",
|
||||||
|
"lead_h": 9.0,
|
||||||
|
"peak_level": 7.54,
|
||||||
|
"peak_pred_24h_before": 7.168225721504108
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.1498671765711928,
|
||||||
|
"mae_above_2p5": 0.40088103598066704,
|
||||||
|
"brier_warn": 0.00801963186023152,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-08-14T04:00:00",
|
||||||
|
"lead_h": 6.0,
|
||||||
|
"peak_level": 6.09,
|
||||||
|
"peak_pred_24h_before": 4.81
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T16:00:00",
|
||||||
|
"lead_h": 8.0,
|
||||||
|
"peak_level": 7.54,
|
||||||
|
"peak_pred_24h_before": 7.002000385945215
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.13833442255794418,
|
||||||
|
"mae_above_2p5": 0.4529809369770209,
|
||||||
|
"brier_warn": 0.009090606171021184,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2022-08-14T04:00:00",
|
||||||
|
"lead_h": 1.0,
|
||||||
|
"peak_level": 6.09,
|
||||||
|
"peak_pred_24h_before": 4.8698313890426705
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2022-10-02T16:00:00",
|
||||||
|
"lead_h": 5.0,
|
||||||
|
"peak_level": 7.54,
|
||||||
|
"peak_pred_24h_before": 6.650326949981244
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2023,
|
||||||
|
"n_train": 37524,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.1489843074454942,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 1.882373500761472e-08,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.16106550376855028,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0001779564366636896,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.1688589840007776,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0012753245446952602,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.1510624979748974,
|
||||||
|
"mae_above_2p5": null,
|
||||||
|
"brier_warn": 0.0005959001115900152,
|
||||||
|
"events": [],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2024,
|
||||||
|
"n_train": 46308,
|
||||||
|
"n_test": 4058,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T10:00:00",
|
||||||
|
"peak_ts": "2024-09-26T00:00:00",
|
||||||
|
"peak_level": 8.27
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-30T03:00:00",
|
||||||
|
"peak_ts": "2024-09-30T06:00:00",
|
||||||
|
"peak_level": 5.99
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T06:00:00",
|
||||||
|
"peak_ts": "2024-10-05T07:00:00",
|
||||||
|
"peak_level": 9.93
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.193584781859895,
|
||||||
|
"mae_above_2p5": 0.5381754200484403,
|
||||||
|
"brier_warn": 0.018292133172974026,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T10:00:00",
|
||||||
|
"lead_h": 5.0,
|
||||||
|
"peak_level": 8.27,
|
||||||
|
"peak_pred_24h_before": 7.11
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-30T03:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 5.99,
|
||||||
|
"peak_pred_24h_before": 5.42
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T06:00:00",
|
||||||
|
"lead_h": 0.0,
|
||||||
|
"peak_level": 9.93,
|
||||||
|
"peak_pred_24h_before": 7.86
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.17314177863843333,
|
||||||
|
"mae_above_2p5": 0.48294427141283425,
|
||||||
|
"brier_warn": 0.016091510788709233,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T10:00:00",
|
||||||
|
"lead_h": 10.0,
|
||||||
|
"peak_level": 8.27,
|
||||||
|
"peak_pred_24h_before": 7.167139790234238
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-30T03:00:00",
|
||||||
|
"lead_h": 10.0,
|
||||||
|
"peak_level": 5.99,
|
||||||
|
"peak_pred_24h_before": 5.618725525893519
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T06:00:00",
|
||||||
|
"lead_h": 69.0,
|
||||||
|
"peak_level": 9.93,
|
||||||
|
"peak_pred_24h_before": 7.889819144742388
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.18833560689452924,
|
||||||
|
"mae_above_2p5": 0.4948752445774323,
|
||||||
|
"brier_warn": 0.01622076553986666,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T10:00:00",
|
||||||
|
"lead_h": 11.0,
|
||||||
|
"peak_level": 8.27,
|
||||||
|
"peak_pred_24h_before": 7.283961881370446
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-30T03:00:00",
|
||||||
|
"lead_h": 10.0,
|
||||||
|
"peak_level": 5.99,
|
||||||
|
"peak_pred_24h_before": 5.783590096006894
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T06:00:00",
|
||||||
|
"lead_h": 69.0,
|
||||||
|
"peak_level": 9.93,
|
||||||
|
"peak_pred_24h_before": 7.922098265471745
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.1644888402154109,
|
||||||
|
"mae_above_2p5": 0.46461119464587947,
|
||||||
|
"brier_warn": 0.013172046828124688,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-24T10:00:00",
|
||||||
|
"lead_h": 11.0,
|
||||||
|
"peak_level": 8.27,
|
||||||
|
"peak_pred_24h_before": 7.348489352370365
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-09-30T03:00:00",
|
||||||
|
"lead_h": 8.0,
|
||||||
|
"peak_level": 5.99,
|
||||||
|
"peak_pred_24h_before": 5.42
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2024-10-03T06:00:00",
|
||||||
|
"lead_h": 2.0,
|
||||||
|
"peak_level": 9.93,
|
||||||
|
"peak_pred_24h_before": 7.91343423984052
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"year": 2025,
|
||||||
|
"n_train": 54577,
|
||||||
|
"n_test": 4392,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-26T06:00:00",
|
||||||
|
"peak_ts": "2025-09-27T21:00:00",
|
||||||
|
"peak_level": 6.64
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2025-10-03T06:00:00",
|
||||||
|
"peak_ts": "2025-10-03T12:00:00",
|
||||||
|
"peak_level": 6.14
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"variants": {
|
||||||
|
"baseline_abs": {
|
||||||
|
"mae": 0.2036944006891808,
|
||||||
|
"mae_above_2p5": 0.40787978637525985,
|
||||||
|
"brier_warn": 0.014130122065259989,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-26T06:00:00",
|
||||||
|
"lead_h": 14.0,
|
||||||
|
"peak_level": 6.64,
|
||||||
|
"peak_pred_24h_before": 5.73
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2025-10-03T06:00:00",
|
||||||
|
"lead_h": 41.0,
|
||||||
|
"peak_level": 6.14,
|
||||||
|
"peak_pred_24h_before": 5.703778873833753
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 2
|
||||||
|
},
|
||||||
|
"rise": {
|
||||||
|
"mae": 0.21617454799997243,
|
||||||
|
"mae_above_2p5": 0.406245211140301,
|
||||||
|
"brier_warn": 0.012587550711857222,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-26T06:00:00",
|
||||||
|
"lead_h": 14.0,
|
||||||
|
"peak_level": 6.64,
|
||||||
|
"peak_pred_24h_before": 5.73
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2025-10-03T06:00:00",
|
||||||
|
"lead_h": 47.0,
|
||||||
|
"peak_level": 6.14,
|
||||||
|
"peak_pred_24h_before": 5.953951787788988
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
},
|
||||||
|
"rise_weighted": {
|
||||||
|
"mae": 0.21768864574559124,
|
||||||
|
"mae_above_2p5": 0.41071503357135775,
|
||||||
|
"brier_warn": 0.014605838473436519,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-26T06:00:00",
|
||||||
|
"lead_h": 13.0,
|
||||||
|
"peak_level": 6.64,
|
||||||
|
"peak_pred_24h_before": 5.73
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2025-10-03T06:00:00",
|
||||||
|
"lead_h": 49.0,
|
||||||
|
"peak_level": 6.14,
|
||||||
|
"peak_pred_24h_before": 5.606011827978066
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
},
|
||||||
|
"rise_quantile": {
|
||||||
|
"mae": 0.18196250028633115,
|
||||||
|
"mae_above_2p5": 0.3987262421172889,
|
||||||
|
"brier_warn": 0.011560937993250782,
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"crossing": "2025-09-26T06:00:00",
|
||||||
|
"lead_h": 13.0,
|
||||||
|
"peak_level": 6.64,
|
||||||
|
"peak_pred_24h_before": 5.73
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"crossing": "2025-10-03T06:00:00",
|
||||||
|
"lead_h": 42.0,
|
||||||
|
"peak_level": 6.14,
|
||||||
|
"peak_pred_24h_before": 6.00597561680636
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"false_alarm_episodes": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -46,14 +46,23 @@ RED = "#d9534f"
|
|||||||
|
|
||||||
|
|
||||||
def fit_backtest_model(df_long: pd.DataFrame, train_end: str):
|
def fit_backtest_model(df_long: pd.DataFrame, train_end: str):
|
||||||
"""Train the 24 h regression + warning heads on rows <= train_end only."""
|
"""Train the 24 h regression + warning heads on rows <= train_end only.
|
||||||
X, Y, _meta = features.build_matrix(df_long, STATION, (HORIZON,))
|
|
||||||
|
Mirrors the deployed hgb-v2 pipeline: the regression head learns the RISE
|
||||||
|
over the current level (rolling-origin evaluation 2026-08-12 showed this
|
||||||
|
moves first-alert leads from ~0 h to +6..+46 h); label statistics are
|
||||||
|
bounded to the training cutoff.
|
||||||
|
"""
|
||||||
|
X, Y, _meta = features.build_matrix(
|
||||||
|
df_long, STATION, (HORIZON,), stats_end=train_end
|
||||||
|
)
|
||||||
train_mask = X.index <= pd.Timestamp(train_end)
|
train_mask = X.index <= pd.Timestamp(train_end)
|
||||||
X_train, Y_train = X.loc[train_mask], Y.loc[train_mask]
|
X_train, Y_train = X.loc[train_mask], Y.loc[train_mask]
|
||||||
|
|
||||||
max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}"
|
max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}"
|
||||||
reg_rows = Y_train[max_col].notna()
|
reg_rows = Y_train[max_col].notna()
|
||||||
reg = _make_regressor().fit(X_train.loc[reg_rows], Y_train.loc[reg_rows, max_col])
|
rise = Y_train.loc[reg_rows, max_col] - X_train.loc[reg_rows, "level"]
|
||||||
|
reg = _make_regressor().fit(X_train.loc[reg_rows], rise)
|
||||||
warn_rows = Y_train[warn_col].notna()
|
warn_rows = Y_train[warn_col].notna()
|
||||||
clf = _make_classifier().fit(
|
clf = _make_classifier().fit(
|
||||||
X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int)
|
X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int)
|
||||||
@@ -70,7 +79,8 @@ def event_series(df_long, X, reg, clf, window_start: str, window_end: str):
|
|||||||
|
|
||||||
Xw = X.loc[window_start:window_end]
|
Xw = X.loc[window_start:window_end]
|
||||||
forecasts = pd.DataFrame(index=Xw.index)
|
forecasts = pd.DataFrame(index=Xw.index)
|
||||||
forecasts["pred_max"] = reg.predict(Xw)
|
# reg predicts the rise; add the current level back (as serving does)
|
||||||
|
forecasts["pred_max"] = reg.predict(Xw) + Xw["level"].to_numpy()
|
||||||
# Belt-and-braces probability: the classifier OR the regression-sigmoid,
|
# Belt-and-braces probability: the classifier OR the regression-sigmoid,
|
||||||
# whichever is more alarmed. The classifier alone proved unreliable on
|
# whichever is more alarmed. The classifier alone proved unreliable on
|
||||||
# out-of-distribution extremes (silent on the 2024 record flood).
|
# out-of-distribution extremes (silent on the 2024 record flood).
|
||||||
|
|||||||
+5
-1
@@ -161,7 +161,11 @@ def _model_forecast(
|
|||||||
if reg is None:
|
if reg is None:
|
||||||
results.append(None)
|
results.append(None)
|
||||||
continue
|
continue
|
||||||
predicted_max = max(float(reg.predict(feature_row)[0]), current_level)
|
raw_prediction = float(reg.predict(feature_row)[0])
|
||||||
|
if bundle.get("regression_target") == "rise":
|
||||||
|
# v2 bundles predict the rise over the current level
|
||||||
|
raw_prediction += current_level
|
||||||
|
predicted_max = max(raw_prediction, current_level)
|
||||||
sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
|
sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
|
||||||
|
|
||||||
# Belt-and-braces: the classifier head OR the regression-sigmoid path,
|
# Belt-and-braces: the classifier head OR the regression-sigmoid path,
|
||||||
|
|||||||
+22
-7
@@ -126,7 +126,8 @@ def _p_warning_series(
|
|||||||
"""Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
|
"""Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
|
||||||
if head is not None:
|
if head is not None:
|
||||||
return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
|
return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
|
||||||
predicted_max = pd.Series(reg.predict(X), index=X.index)
|
# reg predicts the RISE over current level; add the level back
|
||||||
|
predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"]
|
||||||
return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
|
return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
|
||||||
|
|
||||||
|
|
||||||
@@ -240,14 +241,22 @@ def train_station(
|
|||||||
)
|
)
|
||||||
horizon_metrics: dict = {}
|
horizon_metrics: dict = {}
|
||||||
|
|
||||||
# --- regression head (max level) ---
|
# --- regression head (rise to future max) ---
|
||||||
|
# Target = future max MINUS current level ("rise"). Rises are far more
|
||||||
|
# stationary than absolute stages, which softens the cannot-exceed-
|
||||||
|
# training-max ceiling: on the rolling-origin harness (2026-08-12) the
|
||||||
|
# rise target moved P.1 first-alert leads from +0h to +6/+46h and cut
|
||||||
|
# the 2024 record-peak underprediction. Prediction = rise + level.
|
||||||
reg_labeled = eval_Y[max_col].notna()
|
reg_labeled = eval_Y[max_col].notna()
|
||||||
reg = None
|
reg = None
|
||||||
if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
|
if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
|
||||||
|
rise_target = (
|
||||||
|
eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"]
|
||||||
|
)
|
||||||
reg = _safe_fit(
|
reg = _safe_fit(
|
||||||
_make_regressor(hgb_overrides),
|
_make_regressor(hgb_overrides),
|
||||||
eval_X.loc[reg_labeled],
|
eval_X.loc[reg_labeled],
|
||||||
eval_Y.loc[reg_labeled, max_col],
|
rise_target,
|
||||||
f"max_{h}",
|
f"max_{h}",
|
||||||
skipped_heads,
|
skipped_heads,
|
||||||
)
|
)
|
||||||
@@ -259,7 +268,10 @@ def train_station(
|
|||||||
test_labeled = Y_test[max_col].notna()
|
test_labeled = Y_test[max_col].notna()
|
||||||
if test_labeled.sum() > 0:
|
if test_labeled.sum() > 0:
|
||||||
y_true = Y_test.loc[test_labeled, max_col]
|
y_true = Y_test.loc[test_labeled, max_col]
|
||||||
y_pred = reg.predict(X_test.loc[test_labeled])
|
y_pred = (
|
||||||
|
reg.predict(X_test.loc[test_labeled])
|
||||||
|
+ X_test.loc[test_labeled, "level"].to_numpy()
|
||||||
|
)
|
||||||
residuals = y_true.to_numpy() - y_pred
|
residuals = y_true.to_numpy() - y_pred
|
||||||
sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
|
sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
|
||||||
horizon_metrics["n_test"] = int(test_labeled.sum())
|
horizon_metrics["n_test"] = int(test_labeled.sum())
|
||||||
@@ -367,7 +379,7 @@ def train_station(
|
|||||||
reg = _safe_fit(
|
reg = _safe_fit(
|
||||||
_make_regressor(hgb_overrides),
|
_make_regressor(hgb_overrides),
|
||||||
X.loc[labeled],
|
X.loc[labeled],
|
||||||
Y.loc[labeled, max_col],
|
Y.loc[labeled, max_col] - X.loc[labeled, "level"], # rise target
|
||||||
head_key,
|
head_key,
|
||||||
skipped_heads,
|
skipped_heads,
|
||||||
)
|
)
|
||||||
@@ -401,7 +413,10 @@ def train_station(
|
|||||||
|
|
||||||
bundle = {
|
bundle = {
|
||||||
"station_code": station,
|
"station_code": station,
|
||||||
"model_version": f"hgb-v1+{_git_short_sha()}",
|
"model_version": f"hgb-v2+{_git_short_sha()}",
|
||||||
|
# v2: regression heads predict the RISE over the current level; the
|
||||||
|
# serving side must add the level back. Old v1 bundles lack this key.
|
||||||
|
"regression_target": "rise",
|
||||||
"trained_at": datetime.datetime.now().isoformat(),
|
"trained_at": datetime.datetime.now().isoformat(),
|
||||||
"sklearn_version": sklearn.__version__,
|
"sklearn_version": sklearn.__version__,
|
||||||
"feature_names": feature_names,
|
"feature_names": feature_names,
|
||||||
@@ -428,7 +443,7 @@ def train_all(
|
|||||||
"""Train and save every requested station's models. Returns the metrics.json payload."""
|
"""Train and save every requested station's models. Returns the metrics.json payload."""
|
||||||
models_dir = Path(models_dir)
|
models_dir = Path(models_dir)
|
||||||
models_dir.mkdir(parents=True, exist_ok=True)
|
models_dir.mkdir(parents=True, exist_ok=True)
|
||||||
model_version = f"hgb-v1+{_git_short_sha()}"
|
model_version = f"hgb-v2+{_git_short_sha()}"
|
||||||
|
|
||||||
station_results: Dict[str, dict] = {}
|
station_results: Dict[str, dict] = {}
|
||||||
for station in stations:
|
for station in stations:
|
||||||
|
|||||||
Reference in New Issue
Block a user