feat: codified backtests, honest docs, belt-and-braces serving, perf fixes
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).
This commit is contained in:
+9
-6
@@ -164,21 +164,24 @@ def _model_forecast(
|
||||
predicted_max = max(float(reg.predict(feature_row)[0]), current_level)
|
||||
sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
|
||||
|
||||
# Belt-and-braces: the classifier head OR the regression-sigmoid path,
|
||||
# whichever is more alarmed. The 2026-08-11 backtest showed a trained
|
||||
# classifier staying silent through the 2024 record flood while the
|
||||
# regression head tracked it — alerting must never be worse than the
|
||||
# regression fallback.
|
||||
warn_head = (
|
||||
None if thresholds_stale else bundle["heads"].get(f"warn_{horizon_h}")
|
||||
)
|
||||
p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
|
||||
if warn_head is not None:
|
||||
p_warning = float(warn_head.predict_proba(feature_row)[0][1])
|
||||
else:
|
||||
p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
|
||||
p_warning = max(p_warning, float(warn_head.predict_proba(feature_row)[0][1]))
|
||||
|
||||
danger_head = (
|
||||
None if thresholds_stale else bundle["heads"].get(f"danger_{horizon_h}")
|
||||
)
|
||||
p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
|
||||
if danger_head is not None:
|
||||
p_danger = float(danger_head.predict_proba(feature_row)[0][1])
|
||||
else:
|
||||
p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
|
||||
p_danger = max(p_danger, float(danger_head.predict_proba(feature_row)[0][1]))
|
||||
|
||||
p_warning = _clip_probability(p_warning)
|
||||
p_danger = min(_clip_probability(p_danger), p_warning)
|
||||
|
||||
Reference in New Issue
Block a user