feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote
train_all() now raises RainUnavailableError when use_rain=True and the
Open-Meteo history cannot be loaded, instead of logging a warning and
writing gauge-only (v2) bundles over the deployed v3 set -- which is what
the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit
way to get v2. CLI exits 2 with a one-line error. Three tests cover the
guard, the opt-out, and the v3 happy path.
scripts/retrain.sh trains into models/.staging, refuses to promote unless
metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles
into place (previous generation kept in models/.previous). No API restart:
predict.py reloads by mtime on the hourly precompute.
water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent,
OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now
does `uv sync` into .venv (one env rule; removes a stale venv/) and enables
the timer. water-monitor.service in the repo matched neither the deployed
unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).
This commit is contained in:
@@ -733,6 +733,32 @@ timestamp in every bundle. Both are echoed in every `/forecast` row, so you can
|
||||
tell from the API response alone which code produced a forecast and how old the
|
||||
model is.
|
||||
|
||||
**Scheduled retrain (since 2026-09-12).** `scripts/water-monitor-retrain.timer`
|
||||
fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
|
||||
time (`Persistent=true`, so a missed run catches up at boot). The unit runs
|
||||
`scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
|
||||
|
||||
1. trains all stations into `models/.staging/` (the API keeps serving the old
|
||||
bundles throughout);
|
||||
2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
|
||||
least 14 trained stations (exit 3, staging discarded, old models untouched);
|
||||
3. renames the new bundles into `models/`, moving the previous generation to
|
||||
`models/.previous/` for rollback.
|
||||
|
||||
No API restart: `predict.py` reloads bundles by mtime on the next hourly
|
||||
precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
|
||||
run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
|
||||
flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
|
||||
(`scripts/install.sh`) enables the timer.
|
||||
|
||||
**Why the trainer refuses to run without rain (since 2026-09-12).** On
|
||||
2026-09-01 the server retrain could not reach the Open-Meteo archive on a
|
||||
checkout with no `models/cache/`, logged a warning, and quietly overwrote the
|
||||
v3 bundles with gauge-only v2 ones — the 13-hour early warning on the 2024 flood
|
||||
became an 18-hour late one and nothing on the dashboard said so. `train_all()`
|
||||
now raises `RainUnavailableError` (CLI exit 2) in that situation. Gauge-only
|
||||
bundles are still available, but only by asking for them: `--no-rain`.
|
||||
|
||||
## 8. Operations runbook
|
||||
|
||||
All commands assume the project virtualenv is active (`.venv` locally).
|
||||
@@ -786,10 +812,12 @@ that went quiet, or a bad backfill) rather than a modelling one.
|
||||
python -m pytest tests/test_flood_forecast.py -v
|
||||
```
|
||||
|
||||
Seven tests covering leakage, label alignment, the coverage gate, forward-fill and
|
||||
staleness, a train/predict round trip, the heuristic fallback, and feature-name
|
||||
stability. The whole suite runs in about 8 seconds, so there is no excuse for
|
||||
skipping it before a deploy.
|
||||
Tests cover leakage, label alignment, the coverage gate, forward-fill and
|
||||
staleness, a train/predict round trip, the heuristic fallback, feature-name
|
||||
stability, and the rain-downgrade guard (no rain series → `RainUnavailableError`,
|
||||
nothing written; `--no-rain` → v2; rain present → v3 with the rain columns in
|
||||
`feature_names`). The file runs in well under a minute, so there is no excuse
|
||||
for skipping it before a deploy.
|
||||
|
||||
**Understanding graceful degradation.** Three things can make a forecast row
|
||||
non-model-backed, and all of them are visible in the payload:
|
||||
|
||||
Reference in New Issue
Block a user