eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 41s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Documentation / Validate Documentation (push) Failing after 16s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 41s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Documentation / Validate Documentation (push) Failing after 16s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_* caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.
This commit is contained in:
@@ -536,6 +536,54 @@ exists alongside its flood events.
|
||||
(`fill_from_hii`, +9,341 h at P.81, +682 h at P.92, +810 h at P.20) is
|
||||
lead-neutral — the gate holds at 13 h with fill on — and ships enabled.
|
||||
|
||||
### 2026-09-12: three candidates on top of hgb-v3 — two rejected, one deferred
|
||||
|
||||
Same rolling-origin harness (`src/ml/evaluate.py`, five monsoon folds
|
||||
2021–2025, P.1 and P.103), all variants run from the identical
|
||||
`models/cache/` snapshot (`--from-cache`), results in
|
||||
`models/eval_2026-09-12*.json`, tables via `scripts/summarize_eval.py`.
|
||||
Baseline is `rise_rain`, the deployed configuration.
|
||||
|
||||
**Quantile regression heads (`rise_rain_quantile`, `_uw`) — rejected.** The
|
||||
August result that quantile loss beat L2 on MAE held with rain in the model
|
||||
(P.1 0.083/0.081 vs 0.087; P.103 0.143/0.124 vs 0.152), and Brier improved a
|
||||
hair, but the operational numbers went the wrong way: at P.103 the 2022-08-14
|
||||
crossing dropped from +6 h to +1 h lead, 2022-10-02 from +9 h to +5/+3 h, and
|
||||
the 2024-09-30 event from +9 h to +4 h; at P.1 2022 dropped +5 → +3/+2 h and
|
||||
2025 +2 → +1 h, with one false-alarm episode where the baseline had none. A
|
||||
median predicts the *typical* rise, and on the run-up to a crossing the typical
|
||||
rise is not the one that matters. MAE is not the objective; lead is.
|
||||
|
||||
**Quantile heads for sigma only (`rise_rain_qsigma`) — no effect.** The
|
||||
hybrid keeps the L2 point prediction (so every lead is identical to the
|
||||
baseline by construction — p≥0.5 alerts are sigma-independent) and derives a
|
||||
per-row sigma from q90−q50. Brier moved 0.0031 → 0.0029 at P.1 and
|
||||
0.0061 → 0.0060 at P.103, i.e. within noise, at the cost of three fitted
|
||||
heads per horizon instead of one. Per-row uncertainty from this family of
|
||||
models is not informative enough here to be worth the training time; the
|
||||
0.15 m floor stays.
|
||||
|
||||
**Forward-48 h forecast rain (`rise_rain_fc48`) — deferred.** Adding the
|
||||
`(t, t+48]` Open-Meteo sum alongside `rain_fc24` left MAE, Brier and false
|
||||
alarms unchanged and every event lead within ±1 h of baseline, *except* the
|
||||
2024-10-03 P.1 record crossing, which went from +21 h to +72 h (and +55 → +69 h
|
||||
at P.103). That is one event with the highest stakes in the record, on the
|
||||
same feature family that already produced the 2024 gain, but n=1 is not
|
||||
evidence: the P.103 2025-09-26 event lost 2 h in the same run. Rerun after the
|
||||
2026 season adds events; if the 48 h window still moves only the biggest
|
||||
onsets, promote it. Serving would need no new data source (`fetch_forecast`
|
||||
already pulls `forecast_days=2`).
|
||||
|
||||
**HII gauge rain — not evaluable yet.** `hii_rainfall` (~130 gauges in the
|
||||
upper-Ping box, DWR/FOP/HII/RID/TMD) is the obvious independent rain source,
|
||||
but the table only exists since 2026-08-11 and the api-v3 archive endpoint
|
||||
ignores its date range (see `docs/DATA_SOURCES.md` §2.1), so every training
|
||||
row before that is NaN and no fold in the harness has gauge data in its test
|
||||
span. `src/ml/hii_rain.py` builds the catchment mean and
|
||||
`GET /api/hii/rainfall/catchment` exposes it next to the Open-Meteo series with
|
||||
a 24 h-sum bias/MAE/correlation, so the two sources' relationship is on record
|
||||
by the time the 2027 fold (train ≤ 2027-04-30, test Jun–Nov 2027) can test it.
|
||||
|
||||
## 6. Deployment
|
||||
|
||||
### API
|
||||
|
||||
Reference in New Issue
Block a user