Commit Graph
2 Commits
Author SHA1 Message Date
grabowski 0005f7dce1 feat: codified backtests, honest docs, belt-and-braces serving, perf fixes
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the
flood backtests, now reproducible via scripts/backtest_render.py (renders
the three docs/img charts and gates on a >=12h 2024 first-alert lead —
currently failing by design and documented as such).

Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing
was 24 Sep 17:00 (8h earlier than recorded; confirmed against the
independent HII sensor), the historical 24h-warning claim was partly a
missing-data artifact, and retrained warn classifiers collapse on the
filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves
(11.3 -> 10.5 cm). Serving therefore becomes max(classifier,
sigmoid(regression)) so alerting is never worse than the regression path;
metrics table, head-gating tiers, honest-limits and runbook expectations
all updated to the current model (hgb-v1+d2d0e65).

Perf, from Locust load testing (scripts/locustfile.py + load_test.py):
single-flight lock around /forecast inference (concurrent cache misses
previously each ran ~18s inference and starved the shared thread pool;
200-user run after: 105 rps, 0.01% errors), and /measurements/latest +
/health moved off the event loop (synchronous DB/network calls in async
handlers were stalling every request under load).
2026-08-12 10:46:00 +07:00
grabowski 6112c681a7 docs: render of actual vs predicted through the 2024 flood; adaptive replay speed
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead
predicted peak issued at each hour by a model trained only on pre-flood
data, with the warning-probability panel below (first alert 24 Sep
01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's
headline-validation section with an honest reading, including the
~0.4 m peak under-prediction.

Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when
risk is elevated, 1 h (hour-by-hour) while the model is alerting or the
river is near/above flood stage - so viewers can watch the detection
sequence unfold.
2026-08-10 18:10:48 +07:00