feat: codified backtests, honest docs, belt-and-braces serving, perf fixes
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s

Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the
flood backtests, now reproducible via scripts/backtest_render.py (renders
the three docs/img charts and gates on a >=12h 2024 first-alert lead —
currently failing by design and documented as such).

Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing
was 24 Sep 17:00 (8h earlier than recorded; confirmed against the
independent HII sensor), the historical 24h-warning claim was partly a
missing-data artifact, and retrained warn classifiers collapse on the
filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves
(11.3 -> 10.5 cm). Serving therefore becomes max(classifier,
sigmoid(regression)) so alerting is never worse than the regression path;
metrics table, head-gating tiers, honest-limits and runbook expectations
all updated to the current model (hgb-v1+d2d0e65).

Perf, from Locust load testing (scripts/locustfile.py + load_test.py):
single-flight lock around /forecast inference (concurrent cache misses
previously each ran ~18s inference and starved the shared thread pool;
200-user run after: 105 rps, 0.01% errors), and /measurements/latest +
/health moved off the event loop (synchronous DB/network calls in async
handlers were stalling every request under load).
This commit is contained in:
2026-08-12 10:46:00 +07:00
parent d2d0e655aa
commit 0005f7dce1
9 changed files with 647 additions and 75 deletions
+9 -6
View File
@@ -164,21 +164,24 @@ def _model_forecast(
predicted_max = max(float(reg.predict(feature_row)[0]), current_level)
sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
# Belt-and-braces: the classifier head OR the regression-sigmoid path,
# whichever is more alarmed. The 2026-08-11 backtest showed a trained
# classifier staying silent through the 2024 record flood while the
# regression head tracked it — alerting must never be worse than the
# regression fallback.
warn_head = (
None if thresholds_stale else bundle["heads"].get(f"warn_{horizon_h}")
)
p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
if warn_head is not None:
p_warning = float(warn_head.predict_proba(feature_row)[0][1])
else:
p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
p_warning = max(p_warning, float(warn_head.predict_proba(feature_row)[0][1]))
danger_head = (
None if thresholds_stale else bundle["heads"].get(f"danger_{horizon_h}")
)
p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
if danger_head is not None:
p_danger = float(danger_head.predict_proba(feature_row)[0][1])
else:
p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
p_danger = max(p_danger, float(danger_head.predict_proba(feature_row)[0][1]))
p_warning = _clip_probability(p_warning)
p_danger = min(_clip_probability(p_danger), p_warning)