train_all() now raises RainUnavailableError when use_rain=True and the
Open-Meteo history cannot be loaded, instead of logging a warning and
writing gauge-only (v2) bundles over the deployed v3 set -- which is what
the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit
way to get v2. CLI exits 2 with a one-line error. Three tests cover the
guard, the opt-out, and the v3 happy path.
scripts/retrain.sh trains into models/.staging, refuses to promote unless
metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles
into place (previous generation kept in models/.previous). No API restart:
predict.py reloads by mtime on the hourly precompute.
water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent,
OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now
does `uv sync` into .venv (one env rule; removes a stale venv/) and enables
the timer. water-monitor.service in the repo matched neither the deployed
unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).
- .env now chmod 0600 and APP_DIR chmod 0750 after chown, so the Matrix token
and DB credentials are not world-readable.
- uv auto-install (curl | sh as root) is now opt-in via AUTO_INSTALL_UV=1 and
pins a specific uv version; otherwise the script requires uv to be
pre-installed and fails with instructions, avoiding unattended remote code
execution as root.
- scripts/install.sh: one-command hardened deploy (creates the water-monitor
system user, deploys to /opt, builds a uv-managed venv, installs and enables
the systemd unit). Idempotent; excludes .env/*.db/stations.json from sync so
runtime state is preserved.
- Fix placeholder Documentation= URL in water-monitor.service.
- README: document the script as the primary systemd install path, with manual
steps kept as a fallback.
grabowski