feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote

train_all() now raises RainUnavailableError when use_rain=True and the
Open-Meteo history cannot be loaded, instead of logging a warning and
writing gauge-only (v2) bundles over the deployed v3 set -- which is what
the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit
way to get v2. CLI exits 2 with a one-line error. Three tests cover the
guard, the opt-out, and the v3 happy path.

scripts/retrain.sh trains into models/.staging, refuses to promote unless
metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles
into place (previous generation kept in models/.previous). No API restart:
predict.py reloads by mtime on the hourly precompute.

water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent,
OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now
does `uv sync` into .venv (one env rule; removes a stale venv/) and enables
the timer. water-monitor.service in the repo matched neither the deployed
unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).
This commit is contained in:
2026-09-11 21:37:11 +02:00
parent 0a4bf843ff
commit 764764e07e
11 changed files with 326 additions and 26 deletions
+3 -3
View File
@@ -288,9 +288,9 @@ before starting if the script reports it is missing.
```bash
sudo useradd --system --no-create-home --shell /usr/sbin/nologin water-monitor
sudo cp scripts/water-monitor.service /etc/systemd/system/
sudo systemctl enable water-monitor.service
sudo systemctl start water-monitor.service
uv sync --python 3.11 # creates .venv, the interpreter both units run
sudo cp scripts/water-monitor.service scripts/water-monitor-retrain.service scripts/water-monitor-retrain.timer /etc/systemd/system/
sudo systemctl enable --now water-monitor.service water-monitor-retrain.timer
```
</details>