feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote

train_all() now raises RainUnavailableError when use_rain=True and the
Open-Meteo history cannot be loaded, instead of logging a warning and
writing gauge-only (v2) bundles over the deployed v3 set -- which is what
the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit
way to get v2. CLI exits 2 with a one-line error. Three tests cover the
guard, the opt-out, and the v3 happy path.

scripts/retrain.sh trains into models/.staging, refuses to promote unless
metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles
into place (previous generation kept in models/.previous). No API restart:
predict.py reloads by mtime on the hourly precompute.

water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent,
OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now
does `uv sync` into .venv (one env rule; removes a stale venv/) and enables
the timer. water-monitor.service in the repo matched neither the deployed
unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).
This commit is contained in:
2026-09-11 21:37:11 +02:00
parent 0a4bf843ff
commit 764764e07e
11 changed files with 326 additions and 26 deletions
+32 -4
View File
@@ -733,6 +733,32 @@ timestamp in every bundle. Both are echoed in every `/forecast` row, so you can
tell from the API response alone which code produced a forecast and how old the
model is.
**Scheduled retrain (since 2026-09-12).** `scripts/water-monitor-retrain.timer`
fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
time (`Persistent=true`, so a missed run catches up at boot). The unit runs
`scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
1. trains all stations into `models/.staging/` (the API keeps serving the old
bundles throughout);
2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
least 14 trained stations (exit 3, staging discarded, old models untouched);
3. renames the new bundles into `models/`, moving the previous generation to
`models/.previous/` for rollback.
No API restart: `predict.py` reloads bundles by mtime on the next hourly
precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
(`scripts/install.sh`) enables the timer.
**Why the trainer refuses to run without rain (since 2026-09-12).** On
2026-09-01 the server retrain could not reach the Open-Meteo archive on a
checkout with no `models/cache/`, logged a warning, and quietly overwrote the
v3 bundles with gauge-only v2 ones — the 13-hour early warning on the 2024 flood
became an 18-hour late one and nothing on the dashboard said so. `train_all()`
now raises `RainUnavailableError` (CLI exit 2) in that situation. Gauge-only
bundles are still available, but only by asking for them: `--no-rain`.
## 8. Operations runbook
All commands assume the project virtualenv is active (`.venv` locally).
@@ -786,10 +812,12 @@ that went quiet, or a bad backfill) rather than a modelling one.
python -m pytest tests/test_flood_forecast.py -v
```
Seven tests covering leakage, label alignment, the coverage gate, forward-fill and
staleness, a train/predict round trip, the heuristic fallback, and feature-name
stability. The whole suite runs in about 8 seconds, so there is no excuse for
skipping it before a deploy.
Tests cover leakage, label alignment, the coverage gate, forward-fill and
staleness, a train/predict round trip, the heuristic fallback, feature-name
stability, and the rain-downgrade guard (no rain series → `RainUnavailableError`,
nothing written; `--no-rain` → v2; rain present → v3 with the rain columns in
`feature_names`). The file runs in well under a minute, so there is no excuse
for skipping it before a deploy.
**Understanding graceful degradation.** Three things can make a forecast row
non-model-backed, and all of them are visible in the payload: