feat: "Is the model getting better?" - live verification per model version
CI / Test suite (push) Successful in 22s
CI / Format & lint (push) Successful in 16s
Docs / Validate documentation (push) Successful in 10s
Security / Dependency vulnerabilities (push) Successful in 1m34s
Security / Static analysis (push) Successful in 10s
Security / License report (push) Successful in 13s
CI / Test suite (push) Successful in 22s
CI / Format & lint (push) Successful in 16s
Docs / Validate documentation (push) Successful in 10s
Security / Dependency vulnerabilities (push) Successful in 1m34s
Security / Static analysis (push) Successful in 10s
Security / License report (push) Successful in 13s
src/ml/skill.py joins forecast_history (what each deployed version predicted for the 24 h peak, hourly) to water_measurements (what the river did) and reports per version: verified hours, peak MAE, bias, the persistence baseline (peak = current level), skill = 1 - MAE/persistence, and the same MAE restricted to observed peaks >= 2 m. Only forecasts whose window has elapsed with >= 75 % of hours observed count; a version needs 24 verified hours before it is compared. GET /api/forecast/skill?station_code=P.1&horizon=24 returns it (SWR cached, 15 min). The dashboard's forecast card gains a panel with a one-line verdict (current vs previous version), the per-version table, and a caveat that quiet weeks measure quiet-river accuracy only: the model is judged on flood-onset lead, which the backtests cover. EN + TH. On today's production data: hgb-v3+28b62e5 (369 h, Aug 13 - Sep 1) MAE 15.2 cm, skill -0.05; hgb-v2+f6570ac (224 h, Sep 1 - 11) MAE 12.3 cm, skill 0.36 - the "worse" v2 model scores better on a quieter fortnight, which is exactly why the panel shows the >= 2 m column and the caveat. Tests: 3, sqlite, synthetic. scripts/dev_proxy.py: DEV_PROXY_LOCAL lets a not-yet-deployed endpoint be answered from a local JSON file while everything else goes to prod.
This commit is contained in:
@@ -3,6 +3,7 @@ live server, so browser-side changes can be checked against real data
|
||||
before deploy. Usage: python scripts/dev_proxy.py [port]"""
|
||||
|
||||
import http.server
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
@@ -17,6 +18,12 @@ class Handler(http.server.BaseHTTPRequestHandler):
|
||||
body = (STATIC / "dashboard.html").read_bytes()
|
||||
self._send(200, "text/html; charset=utf-8", body)
|
||||
return
|
||||
# Local overrides for endpoints not yet deployed: DEV_PROXY_LOCAL=/api/x=file.json,...
|
||||
for pair in filter(None, os.environ.get("DEV_PROXY_LOCAL", "").split(",")):
|
||||
prefix, file = pair.split("=", 1)
|
||||
if self.path.split("?")[0] == prefix:
|
||||
self._send(200, "application/json", Path(file).read_bytes())
|
||||
return
|
||||
if self.path.startswith("/static/"):
|
||||
f = STATIC / self.path[len("/static/"):].split("?")[0]
|
||||
if f.is_file():
|
||||
|
||||
Reference in New Issue
Block a user