feat: codified backtests, honest docs, belt-and-braces serving, perf fixes

Browse Source

Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the
flood backtests, now reproducible via scripts/backtest_render.py (renders
the three docs/img charts and gates on a >=12h 2024 first-alert lead —
currently failing by design and documented as such).

Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing
was 24 Sep 17:00 (8h earlier than recorded; confirmed against the
independent HII sensor), the historical 24h-warning claim was partly a
missing-data artifact, and retrained warn classifiers collapse on the
filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves
(11.3 -> 10.5 cm). Serving therefore becomes max(classifier,
sigmoid(regression)) so alerting is never worse than the regression path;
metrics table, head-gating tiers, honest-limits and runbook expectations
all updated to the current model (hgb-v1+d2d0e65).

Perf, from Locust load testing (scripts/locustfile.py + load_test.py):
single-flight lock around /forecast inference (concurrent cache misses
previously each ran ~18s inference and starved the shared thread pool;
200-user run after: 105 rps, 0.01% errors), and /measurements/latest +
/health moved off the event loop (synchronous DB/network calls in async
handlers were stalling every request under load).

...

This commit is contained in:

grabowski

2026-08-12 10:46:00 +07:00

parent d2d0e655aa

commit 0005f7dce1

9 changed files with 647 additions and 75 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

docs/img/backtest-2024-p1-detail.png

BIN

Binary file not shown.

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 109 KiB After Width:  |  Height:  |  Size: 118 KiB

docs/img/backtest-2024-p1.png

BIN

Binary file not shown.

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 119 KiB After Width:  |  Height:  |  Size: 147 KiB

docs/img/backtest-2025-p1.png

BIN

Binary file not shown.

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 127 KiB After Width:  |  Height:  |  Size: 141 KiB