grabowski
d621aa9ce7
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 41s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Documentation / Validate Documentation (push) Failing after 16s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_* caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.
63 lines
2.9 KiB
Python
63 lines
2.9 KiB
Python
"""Summarise rolling-origin harness output side by side.
|
|
|
|
Usage:
|
|
uv run python scripts/summarize_eval.py models/eval_2026-09-12.json [more.json ...]
|
|
|
|
Aggregates each (station, variant) across folds: mean MAE, mean flood-regime
|
|
MAE, mean Brier, total false-alarm episodes, and every warning event with its
|
|
first-alert lead and the 24 h-ahead peak error -- the operational numbers that
|
|
decide whether a variant ships.
|
|
"""
|
|
|
|
import json
|
|
import statistics
|
|
import sys
|
|
from collections import OrderedDict
|
|
|
|
|
|
def summarize(paths):
|
|
for path in paths:
|
|
results = json.load(open(path, encoding="utf-8"))
|
|
print(f"\n##### {path}")
|
|
for station in results:
|
|
print(f"\n=== {station['station']} (warn {station['warn_thr']:.2f} m) ===")
|
|
agg = OrderedDict()
|
|
for fold in station["folds"]:
|
|
for name, m in fold["variants"].items():
|
|
a = agg.setdefault(
|
|
name, {"mae": [], "mae_hi": [], "brier": [], "fa": 0, "events": []}
|
|
)
|
|
a["mae"].append(m["mae"])
|
|
if m.get("mae_above_2p5") is not None:
|
|
a["mae_hi"].append(m["mae_above_2p5"])
|
|
if m.get("brier_warn") is not None:
|
|
a["brier"].append(m["brier_warn"])
|
|
a["fa"] += m["false_alarm_episodes"]
|
|
for e in m["events"]:
|
|
err = (
|
|
None
|
|
if e["peak_pred_24h_before"] is None
|
|
else e["peak_pred_24h_before"] - e["peak_level"]
|
|
)
|
|
a["events"].append((fold["year"], e["crossing"][:10], e["lead_h"], e["peak_level"], err))
|
|
print(f"{'variant':22} {'MAE':>6} {'MAE_hi':>7} {'Brier':>7} {'FA':>3} events: year crossing lead_h peak(err24h)")
|
|
for name, a in agg.items():
|
|
ev = " ".join(
|
|
f"{y} {d} {'—' if l is None else format(l, '+.0f')}h {p:.2f}({'—' if err is None else format(err, '+.2f')})"
|
|
for y, d, l, p, err in a["events"]
|
|
)
|
|
leads = [l for *_, l, _, _ in a["events"] if l is not None]
|
|
print(
|
|
f"{name:22} {statistics.mean(a['mae']):6.3f} "
|
|
f"{statistics.mean(a['mae_hi']) if a['mae_hi'] else float('nan'):7.3f} "
|
|
f"{statistics.mean(a['brier']) if a['brier'] else float('nan'):7.4f} "
|
|
f"{a['fa']:>3} {ev}"
|
|
)
|
|
if leads:
|
|
print(f"{'':22} lead: mean {statistics.mean(leads):+.1f} h, min {min(leads):+.0f} h, "
|
|
f"missed {sum(1 for *_, l, _, _ in a['events'] if l is None)}/{len(a['events'])}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
summarize(sys.argv[1:] or ["models/eval_variants.json"])
|