feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate

The rolling-origin harness (models/eval_rain.json) showed catchment rain
halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and
extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at
P.103). Ported: train_all loads the catchment-mean series (use_rain /
--no-rain to opt out; without it bundles train as v2), predict fetches
live rain hourly and passes an empty series on failure so rain-trained
bundles serve with NaN features instead of tripping the feature guard,
and the leader worker persists hourly per-point + catchment-mean rows to
a new openmeteo_rain table.

Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING
(alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h
acceptance gate PASSES for the first time. Journey on that crossing:
v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its
artifact +46h latch for a calibrated +2h with zero false alarms. P.1
MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.

docs/DATA_SOURCES.md

@@ -204,7 +204,7 @@ Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
 
 | Source | What | Access |
 |---|---|---|
-| **Open-Meteo** (<https://open-meteo.com>) | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
+| **Open-Meteo** (<https://open-meteo.com>) — ✅ **INGESTED** since 2026-08-12 (`src/ml/rain.py`): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the `openmeteo_rain` table | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
 | TMD NWP API (`https://data.tmd.go.th/nwpapi/v1/forecast/location/...`) | WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (`/nwpapi/doc/main/`) |
 | GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |
 

docs/FLOOD_FORECASTING.md

@@ -261,19 +261,21 @@ next section explains why (the hourly grid was gap-filled from ~56% to ~93%
 between them, roughly doubling the test rows and collapsing the warning base
 rates).
 
-**Current model** `hgb-v2` (rise target), generated 2026-08-12 on the
-gap-filled DB (~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12.
-P.1:
+**Current model** `hgb-v3` (rise target + Open-Meteo rain features),
+generated 2026-08-12 on the gap-filled DB (~976k rows). Train ≤ 2024-12-31,
+test 2025-01-01 → 2026-08-12. P.1:
 
 | Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
 |---|---|---|---|---|---|
-| 6 h | 0.783 | 5.0 cm | 5.2 cm | 14,034 | 0.12% |
-| 12 h | 0.508 | 7.2 cm | 10.9 cm | 14,028 | 0.16% |
-| 24 h | 0.288 | 9.4 cm | 20.0 cm | 14,034 | 0.25% |
+| 6 h | 0.783 | 4.9 cm | 5.0 cm | 14,034 | 0.12% |
+| 12 h | 0.508 | 7.2 cm | 10.7 cm | 14,028 | 0.16% |
+| 24 h | 0.288 | 8.7 cm | 18.1 cm | 14,034 | 0.25% |
 
-(The prior absolute-target run of the same day, `hgb-v1+d2d0e65`, scored
-5.5/8.1/10.5 cm MAE and 7.9/14.6/24.0 cm above 2 m — the rise target improved
-every regression figure; PR-AUC belongs to the unchanged classifier heads.)
+(Progression across the same day's runs — v1 absolute target:
+5.5/8.1/10.5 cm MAE; v2 rise: 5.0/7.2/9.4; v3 rise+rain: 4.9/7.2/8.7 —
+with above-2 m MAE falling 24.0 → 20.0 → 18.1 cm at 24 h. PR-AUC belongs to
+the unchanged classifier heads; serving is belt-and-braces so alerting uses
+the improved regression path regardless.)
 
 Level accuracy improved; standalone classifier discrimination did not survive
 the data change (which is why serving is now `max(classifier, sigmoid)` — see
@@ -346,6 +348,21 @@ genuine out-of-distribution weakness (see the backtest sections) does the rest.
 > quantile variants were evaluated and rejected (more false alarms, no
 > calibration gain by Brier score). The ≥12 h acceptance gate still fails at
 > +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
+>
+> **2026-08-12 follow-up 2 — hgb-v3 (rain features): the gate passes.**
+> Open-Meteo catchment rainfall (five upper-Ping points, forecast-model
+> archive 2021+, `src/ml/rain.py`) added four features: trailing 6/24/72 h
+> rain sums and `rain_fc24`, the forward-24 h forecast sum — the first input
+> that can act before water reaches any gauge. On the rolling-origin harness
+> (`models/eval_rain.json`) rain roughly halved flood-year Brier scores, cut
+> flood-regime MAE 20–40%, and moved the hard 2024 leads from +6 h to +11 h
+> (P.1) and +10 to +19 h (P.103); the marginal 2025 double-crest event trades
+> its artifact +46 h "lead" for a calibrated +2 h with zero false alarms. The
+> regenerated backtest below now shows a **13-hour warning for the 2024
+> record flood (alert 04:00, crossing 17:00) — the ≥12 h acceptance gate
+> passes for the first time**. P.1 MAE improves again to 4.9/7.2/8.7 cm at
+> 6/12/24 h. Serving fetches live rain hourly and degrades to NaN features
+> (never a crash) if Open-Meteo is unreachable.
 
 ### The September 2025 flood, as the deployed configuration saw it
 
@@ -355,12 +372,13 @@ through 2024](img/backtest-2025-p1.png)
 This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
 saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
 gap-filled data — see the re-examination note above for the history of these
-numbers.)* The v2 model first alerts at **21:00 on 25 September 2025 — 45
-hours before the river crosses 3.70 m** at 18:00 on the 27th. The long lead is
-partly the twin-crest shape of this event (the near-miss 3.51 m crest of
-26 September keeps the 24 h-peak forecast near the line), so read it as
-"the model was correctly alarmed through the whole double crest", not as a
-general 45 h capability.
+numbers.)* The v3 model first alerts at **16:00 on 27 September 2025 — 2 hours before
+the river crosses 3.70 m** at 18:00. This is a shorter lead than v2's 45 h,
+and deliberately so: v2's long "lead" was an alarm that latched through the
+near-miss 3.51 m crest of the 26th; v3's rain-informed probabilities are far
+better calibrated on this marginal event (Brier halved, zero false-alarm
+episodes on the season) and fire when exceedance actually becomes likely. A
+barely-over-threshold crest is intrinsically a short-notice event.
 
 ### Headline validation: the October 2024 record flood
 
@@ -373,27 +391,29 @@ followed. This is the closest thing to a real operational test available.
 October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
 
 The render above shows the whole event hour by hour *(regenerated 2026-08-12
-with the hgb-v2 rise target)*. Top: the observed level (blue) against the
-24 h-ahead predicted peak the model issued at each hour (amber, dashed) — the
-amber line now leads the blue one into both flood waves. Bottom: the
+with the hgb-v3 rise + rain configuration)*. Top: the observed level (blue)
+against the 24 h-ahead predicted peak the model issued at each hour (amber,
+dashed) — the amber line leads the blue one into both flood waves. Bottom: the
 belt-and-braces probability of flooding within 24 h; the **first alert comes
-at 11:00 on 24 September, 6 hours before the true 17:00 crossing**, and stays
-correctly alarmed through the October record wave. The predicted peak now
-slightly overshoots the 5.30 m record instead of capping ~0.4 m below it — the
-rise target removed the cannot-exceed-training-max ceiling. The same historic
-model track drives the dashboard's "Replay Oct 2024 flood" feature.
+at 04:00 on 24 September, 13 hours before the true 17:00 crossing**, while
+the river in town still read 2.9 m — the rain features react to upstream
+precipitation before any gauge rises. The rise target removed the
+cannot-exceed-training-max ceiling, so the record 5.30 m peak is tracked
+rather than capped. The same historic model track drives the dashboard's
+"Replay Oct 2024 flood" feature.
 
 ![Hour-by-hour detail of the detection window, 22–28 September
 2024](img/backtest-2024-p1-detail.png)
 
 The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
-hgb-v2)* shows the sequence: the river crosses 3.70 m at **17:00 on
+hgb-v3)* shows the sequence: the river crosses 3.70 m at **17:00 on
 24 September** (the hours recovered by gap-filling; independently confirmed by
 the HII sensor at the same bridge), and the model's probability crosses 0.5 at
-**11:00 — a 6-hour warning** delivered while the river stood at 3.4 m. Under
-the absolute-level target this alert came 18 hours *after* the crossing; the
-rise target recovered the lead. Extending 6 h toward the 12+ h the acceptance
-gate demands is what the rainfall features are for.
+**04:00 — a 13-hour warning** delivered while the river stood at 2.9 m. The
+same alert under the absolute-level target came 18 hours *after* the crossing
+(v1), and 6 hours before it with the rise target alone (v2); catchment
+rainfall closed the rest. This clears the ≥12 h acceptance gate in
+`scripts/backtest_render.py`.
 
 The event bullets below quote the original (pre-gap-fill) evaluation of the
 deployed model and are kept for the historical record — see the re-examination
@@ -423,14 +443,16 @@ note above for why the lead times no longer reproduce:
 time into P.1 is 17 h (P.20), and the strongest predictors are much closer:
 P.103 at 1 h, P.67 at 7 h, P.21 at 9 h. Once a 24 h forecast reaches past roughly
 17 h, there is no observation that has "already happened" to inform it — the model
-is extrapolating basin state and season, not routing a wave. The regenerated
-backtests bear this out — harder than first documented (see the re-examination
-note in section 7): on the gap-filled data the retrained configuration alerts
-the 27 September 2025 crossing *as it happens* and the 24 September 2024
-crossing ~18 h *late*. **Genuine gauge-only lead for P.1 is at best ~7–17 h,
-and for fast rises can be zero.** Extending it requires rainfall inputs and
-Mae Ngat/Mae Kuang dam release data, plus the rise-target/quantile modelling
-work — rainfall collection began 2026-08-11 (see `docs/DATA_SOURCES.md`).
+is extrapolating basin state and season — unless it has rainfall. That is no
+longer hypothetical: hgb-v3's Open-Meteo features (see the re-examination
+notes in section 7) took the 2024 record-flood lead from 18 h late (v1
+gauge-only, absolute target) to 13 h early, precisely because catchment rain
+acts before any gauge rises, and `rain_fc24` — a weather *forecast* — acts
+before the rain itself falls. **Remaining honest limits:** marginal
+just-over-threshold crests (2025: +2 h) are intrinsically short-notice; the
+rain series only exists from 2021-03, so older training rows are rain-blind;
+forecast-rain quality bounds what the feature can add; and Mae Ngat/Mae Kuang
+dam releases remain uningested (see `docs/DATA_SOURCES.md`).
 
 **Danger-level skill at P.1 is unproven.** P.1 never crossed 4.5 m in the
 2025-01-01 → 2026-08-10 test span (`base_rate_danger` is 0.0, so every danger