feat: codified backtests, honest docs, belt-and-braces serving, perf fixes

Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the
flood backtests, now reproducible via scripts/backtest_render.py (renders
the three docs/img charts and gates on a >=12h 2024 first-alert lead —
currently failing by design and documented as such).

Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing
was 24 Sep 17:00 (8h earlier than recorded; confirmed against the
independent HII sensor), the historical 24h-warning claim was partly a
missing-data artifact, and retrained warn classifiers collapse on the
filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves
(11.3 -> 10.5 cm). Serving therefore becomes max(classifier,
sigmoid(regression)) so alerting is never worse than the regression path;
metrics table, head-gating tiers, honest-limits and runbook expectations
all updated to the current model (hgb-v1+d2d0e65).

Perf, from Locust load testing (scripts/locustfile.py + load_test.py):
single-flight lock around /forecast inference (concurrent cache misses
previously each ran ~18s inference and starved the shared thread pool;
200-user run after: 105 rps, 0.01% errors), and /measurements/latest +
/health moved off the event loop (synchronous DB/network calls in async
handlers were stalling every request under load).

docs/FLOOD_FORECASTING.md

@@ -203,16 +203,17 @@ section 6).
 
 The system degrades in tiers rather than failing:
 
-1. **Classifier head**, when the training span contains at least
-   `MIN_POSITIVES_FOR_CLASSIFIER = 30` positive examples. Below that, a
-   classifier would be fitting noise, and the head is recorded in
-   `skipped_heads` with its reason.
-2. **Sigmoid on the regression head**, when the classifier is absent.
-   `p = 1/(1 + exp(−(predicted_max − threshold)/σ))`, where σ is the standard
-   deviation of the regressor's test residuals (floor `MIN_SIGMA = 0.15` m). This
-   turns the peak-level prediction into a calibrated-ish probability that widens
-   correctly when the regressor is less accurate at that horizon — at P.1, σ is
-   0.15 m at 6 and 12 h but 0.166 m at 24 h.
+1. **Belt-and-braces probability** *(since 2026-08-11 — see the re-examination
+   note in section 7)*: the sigmoid-of-regression probability
+   `p = 1/(1 + exp(−(predicted_max − threshold)/σ))` is always computed (σ =
+   the regressor's test-residual std, floor `MIN_SIGMA = 0.15` m), and when a
+   classifier head exists — trained only if the span had at least
+   `MIN_POSITIVES_FOR_CLASSIFIER = 30` positives — the served probability is
+   `max(classifier, sigmoid)`. The classifier can raise the alarm but never
+   silence it: on the gap-filled data a trained classifier stayed near zero
+   through the 2024 record crossing while the regression tracked it.
+2. **Sigmoid only**, when the classifier head is absent or skipped
+   (recorded in `skipped_heads` with its reason).
 3. **Persistence heuristic** (`predict._heuristic_forecast`), when there is no
    model file at all, or the station's newest reading is more than
    `STALE_AFTER_H = 6` hours old. It extrapolates the last 3 h rate of rise
@@ -255,47 +256,92 @@ invalidates the cache without a restart.
 
 ### Holdout metrics (`models/metrics.json`)
 
-Model version `hgb-v1+49a3de0`, generated 2026-08-10. Train ≤ 2024-12-31, test
-2025-01-01 → 2026-08-10 — the test span is entirely unseen future data relative
-to training.
+Two evaluations exist and they differ sharply — the re-examination note in the
+next section explains why (the hourly grid was gap-filled from ~56% to ~93%
+between them, roughly doubling the test rows and collapsing the warning base
+rates).
 
-P.1 (Nawarat Bridge), the station that matters most:
+**Current model** `hgb-v1+d2d0e65`, generated 2026-08-12 on the gap-filled DB
+(~976k rows). Train ≤ 2024-12-31, test 2025-01-01 → 2026-08-12. P.1:
 
-| Horizon | Warning PR-AUC | Recall @1% FAR | Recall @5% FAR | MAE | MAE above 2 m | Test rows | Base rate |
-|---|---|---|---|---|---|---|---|
-| 6 h | 0.974 | 98.3% | 100% | 6.1 cm | 9.2 cm | 8,536 | 1.36% |
-| 12 h | 0.904 | 93.8% | 97.7% | 9.0 cm | 15.0 cm | 7,932 | 1.61% |
-| 24 h | 0.900 | 90.1% | 93.4% | 11.3 cm | 24.5 cm | 8,572 | 1.77% |
+| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
+|---|---|---|---|---|---|
+| 6 h | 0.783 | 5.5 cm | 7.9 cm | 14,034 | 0.12% |
+| 12 h | 0.508 | 8.1 cm | 14.6 cm | 14,028 | 0.16% |
+| 24 h | 0.288 | 10.5 cm | 24.0 cm | 14,034 | 0.25% |
 
-Read PR-AUC against the base rate — 0.974 versus a 1.36% positive rate is a wide
-margin over chance. "Recall at 1% false-alarm rate" is the operationally honest
-number: at a threshold that fires on 1% of quiet hours, the 6 h model still
-catches 98.3% of warning exceedances.
+Level accuracy improved; standalone classifier discrimination did not survive
+the data change (which is why serving is now `max(classifier, sigmoid)` — see
+"Head gating"). Recall-at-FAR is null at all horizons on this run. Across
+stations the 6 h warning PR-AUC now spans 0.987 (P.77) / 0.982 (P.5) / 0.956
+(P.85) / 0.950 (P.67) down to 0.436 (P.84), and danger heads are now evaluable
+at nine stations — strongest P.5 (0.958/0.883/0.811 at 6/12/24 h) and P.77
+(0.942/0.863/0.786); P.103's danger metrics, previously the highlight, are null
+on this span.
 
-P.103 (Ring Bridge 3) is the only station with enough danger-level events to
-evaluate a danger head on the 2025–26 span (base rate 5.7–7.4%): PR-AUC 0.979 /
-0.953 / 0.892 and recall at 1% FAR of 97.9% / 89.9% / 79.5% at 6 / 12 / 24 h.
+**Historical evaluation** (`hgb-v1+49a3de0`, 2026-08-10, pre-gap-fill DB —
+kept for the record; these numbers described the sparser 56%-filled grid and do
+not reproduce on today's data):
 
-Across the other stations the 6 h warning PR-AUC spans 0.996 (P.5) down to 0.302
-(P.82), and tracks almost exactly with how many exceedances that station saw. The
-strong ones are the frequently-flooded gauges — P.5 0.996, P.81 0.992, P.77 0.968,
-P.85 0.953, P.75 0.927 — and the weak ones are un-routed western tributaries with
-almost no positives (P.84 0.570, P.82 0.302 on 0.22% of test hours). P.92 and P.20
-have no evaluable warning metric at all: neither crossed 3.0 m often enough in the
-test span (P.92 not once, P.20 in 0.09% of hours) to score.
+| Horizon | Warning PR-AUC | Recall @1% FAR | MAE | Test rows | Base rate |
+|---|---|---|---|---|---|
+| 6 h | 0.974 | 98.3% | 6.1 cm | 8,536 | 1.36% |
+| 12 h | 0.904 | 93.8% | 9.0 cm | 7,932 | 1.61% |
+| 24 h | 0.900 | 90.1% | 11.3 cm | 8,572 | 1.77% |
+
+The dramatic PR-AUC difference is mostly the base rate: the filled grid adds
+~5,500 quiet test hours per horizon while the number of positive hours barely
+changes, so the same ranking quality scores far lower — and the classifier's
+genuine out-of-distribution weakness (see the backtest sections) does the rest.
+
+### 2026-08-11 re-examination: fuller data changes the backtest story
+
+> **Read this before the two backtest sections below.** On 2026-08-11 the
+> backtests were codified into `scripts/backtest_render.py` (previously they
+> were one-off runs) and re-run after the database grew from 592k to ~976k
+> rows (a `--fill-gaps all` pass repaired most of the missing 44% of the
+> hourly grid). Three things changed:
+>
+> 1. **The 2024 crossing was 8 hours earlier than documented.** The recovered
+>    hours show P.1 crossing 3.70 m at **17:00 on 24 September 2024**, not
+>    01:00 on 25 September — confirmed independently by the HII sensor at
+>    Nawarat Bridge (hii_waterlevel, station 3226: 3.73 m at 17:00). The
+>    originally celebrated "24-hour warning" was therefore ~16 hours measured
+>    against the real river.
+> 2. **Retraining on the fuller data improves level accuracy but degrades the
+>    warning classifiers.** P.1 24 h MAE improved (11.3 → 10.5 cm), but the
+>    warning-head PR-AUC collapsed (0.900 → 0.288 at 24 h): with the filled
+>    grid the classifier trains on many more dry-season rows and now stays
+>    silent through the September 2024 record crossing while the regression
+>    head tracks it. Serving was changed to belt-and-braces —
+>    `max(classifier, sigmoid(regression))` — so alerting can never be worse
+>    than the regression path.
+> 3. **Honest current lead times, from the regenerated charts below:** the
+>    retrained configuration first alerts ~18 h *after* the true 24 Sep 2024
+>    crossing and roughly *at* the 27 Sep 2025 crossing. The earlier, better
+>    numbers came from models trained and evaluated on the sparser data. The
+>    conclusion is not that the old system was better — it is that gauge-only
+>    features fundamentally lack lead time for fast rises, which is exactly
+>    the rainfall-input and rise-target work now queued (see "Honest limits").
+>
+> `scripts/backtest_render.py` regenerates all three charts and fails its
+> acceptance gate while the 2024 lead stays under 12 h — keeping this page
+> honest is now automatic.
 
 ### The September 2025 flood, as the deployed configuration saw it
 
 ![Observed vs predicted through the September 2025 flood — model trained only
 through 2024](img/backtest-2025-p1.png)
 
-This is the strongest single piece of evidence, because it uses the exact
-deployed configuration (train ≤ 2024-12-31) on an event it never saw: the
-model's **first alert came 26 September 2025 at 18:00, exactly 24 hours before
-the river crossed 3.70 m**, and it predicted a 4.00 m peak against an actual
-3.93 m — within 7 cm. Note the discrimination: the near-miss crest of 3.51 m on
-26 September never triggered an alert, the probability fires only for the real
-event, and it stands down as the water recedes.
+This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
+saw. *(Chart regenerated 2026-08-11 on the gap-filled data — see the
+re-examination note above; the original one-off render, trained on the sparser
+data, alerted 24 h ahead and predicted the 3.93 m peak within 7 cm.)* On
+today's fuller dataset the retrained equivalent first alerts at **18:00 on
+27 September 2025 — as the river crosses 3.70 m**, not a day ahead. The
+discrimination remains good: the near-miss 3.51 m crest on 26 September never
+triggers, the probability fires only for the real event, and it stands down as
+the water recedes.
 
 ### Headline validation: the October 2024 record flood
 
@@ -307,27 +353,33 @@ followed. This is the closest thing to a real operational test available.
 ![Observed P.1 level vs the model's 24 h-ahead predicted peak through the
 October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
 
-The render above shows the whole event hour by hour. Top: the observed level
-(blue) against the 24 h-ahead predicted peak the model issued at each hour
-(amber, dashed) — the amber line leads the blue one into both flood waves,
-which is the entire point of the system. Bottom: the model's probability of
-flooding within 24 h; it fires its **first alert at 01:00 on 24 September, a
-full day before the river crossed the 3.70 m flooding line**, stays pinned near
-1.0 through both waves, and stands down between and after them. Also visible,
-honestly: the predicted peak tops out ~0.4 m short of the actual 5.30 m record
-(the extreme-compression limitation discussed below), and the prediction is
-noisier on the recession limbs. The same model track drives the dashboard's
-"Replay Oct 2024 flood" feature, so this chart can be watched live on the map.
+The render above shows the whole event hour by hour *(regenerated 2026-08-11
+on the gap-filled data)*. Top: the observed level (blue) against the 24 h-ahead
+predicted peak the model issued at each hour (amber, dashed) — the amber line
+tracks both flood waves but no longer clearly leads the first one. Bottom: the
+belt-and-braces probability of flooding within 24 h; on the fuller data its
+**first alert comes at 11:00 on 25 September, ~18 hours after the true 17:00
+24 September crossing**, then stays correctly alarmed through the October
+record wave. Also visible, honestly: the predicted peak tops out well short of
+the actual 5.30 m record (the extreme-compression limitation discussed below).
+The same historic model track drives the dashboard's "Replay Oct 2024 flood"
+feature.
 
 ![Hour-by-hour detail of the detection window, 22–28 September
 2024](img/backtest-2024-p1-detail.png)
 
-The hour-by-hour detail of the detection window shows the sequence exactly: the
-predicted 24 h peak (amber) starts pulling away from the observed level late on
-23 September as upstream gauges rise, the warning probability snaps from ~0 to
-1.0 at **01:00 on 24 September**, and the river crosses 3.70 m at **01:00 on
-25 September** — a clean 24-hour warning, delivered while the river in town
-still looked normal at 2.8 m.
+The hour-by-hour detail of the detection window *(regenerated 2026-08-11)*
+shows the corrected sequence: the river crosses 3.70 m at **17:00 on
+24 September** (the hours recovered by gap-filling; independently confirmed by
+the HII sensor at the same bridge), while the retrained model's probability
+only crosses 0.5 at **11:00 on 25 September**. The original render — sparser
+data, different trained model — alerted at 01:00 on 24 September against an
+apparent 01:00 25 September crossing. Closing this real gap is what the
+rainfall features and rise-target work are for.
+
+The event bullets below quote the original (pre-gap-fill) evaluation of the
+deployed model and are kept for the historical record — see the re-examination
+note above for why the lead times no longer reproduce:
 
 - **25 September cold start.** P.1's first warning crossing of the episode was
   alerted **24–26 hours ahead**. This is the genuinely impressive case: the river
@@ -353,11 +405,14 @@ still looked normal at 2.8 m.
 time into P.1 is 17 h (P.20), and the strongest predictors are much closer:
 P.103 at 1 h, P.67 at 7 h, P.21 at 9 h. Once a 24 h forecast reaches past roughly
 17 h, there is no observation that has "already happened" to inform it — the model
-is extrapolating basin state and season, not routing a wave. The 2025–26 test
-events bear this out: the 25 September 2025 cold-start crossing was called 7 h
-ahead by the 12 h model and 9 h ahead by the 24 h model. **Practical lead for P.1
-is ~7–17 h.** Extending it requires rainfall forecasts and Mae Ngat/Mae Kuang dam
-release data, neither of which this system currently ingests.
+is extrapolating basin state and season, not routing a wave. The regenerated
+backtests bear this out — harder than first documented (see the re-examination
+note in section 7): on the gap-filled data the retrained configuration alerts
+the 27 September 2025 crossing *as it happens* and the 24 September 2024
+crossing ~18 h *late*. **Genuine gauge-only lead for P.1 is at best ~7–17 h,
+and for fast rises can be zero.** Extending it requires rainfall inputs and
+Mae Ngat/Mae Kuang dam release data, plus the rise-target/quantile modelling
+work — rainfall collection began 2026-08-11 (see `docs/DATA_SOURCES.md`).
 
 **Danger-level skill at P.1 is unproven.** P.1 never crossed 4.5 m in the
 2025-01-01 → 2026-08-10 test span (`base_rate_danger` is 0.0, so every danger
@@ -617,10 +672,12 @@ print({h: (d.get('pr_auc_warn'), d.get('mae')) for h, d in m['stations']['P.1'][
 ```
 
 Expect fifteen `trained` and one `heuristic` (P.4A). A station that reports
-`failed` names its reason in the same payload. If P.1's 6 h warning PR-AUC has
-dropped materially below ~0.97 or its MAE has risen well above ~6 cm, investigate
-before deploying — that usually means a data problem (a gauge that went quiet, or
-a bad backfill) rather than a modelling one.
+`failed` names its reason in the same payload. Compare against the *previous
+run's* `metrics.json`, not an absolute bar: after the 2026-08-11 gap-fill the
+expected baseline is P.1 6 h warning PR-AUC ≈ 0.78 and MAE ≈ 5.5 cm (the
+historical ~0.97 figure belonged to the sparse pre-fill grid — see section 5).
+A *material drop from the previous run* usually means a data problem (a gauge
+that went quiet, or a bad backfill) rather than a modelling one.
 
 **Run the tests** (synthetic data only, no database or network required):