grabowski
764764e07e
train_all() now raises RainUnavailableError when use_rain=True and the
Open-Meteo history cannot be loaded, instead of logging a warning and
writing gauge-only (v2) bundles over the deployed v3 set -- which is what
the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit
way to get v2. CLI exits 2 with a one-line error. Three tests cover the
guard, the opt-out, and the v3 happy path.
scripts/retrain.sh trains into models/.staging, refuses to promote unless
metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles
into place (previous generation kept in models/.previous). No API restart:
predict.py reloads by mtime on the hourly precompute.
water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent,
OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now
does `uv sync` into .venv (one env rule; removes a stale venv/) and enables
the timer. water-monitor.service in the repo matched neither the deployed
unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).
40 lines
1.5 KiB
Desktop File
40 lines
1.5 KiB
Desktop File
[Unit]
|
|
Description=Retrain the Ping River flood forecast models
|
|
Documentation=https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/-/blob/master/docs/FLOOD_FORECASTING.md
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
User=water-monitor
|
|
Group=water-monitor
|
|
WorkingDirectory=/opt/thailand-water-monitor
|
|
EnvironmentFile=/opt/thailand-water-monitor/.env
|
|
# Same interpreter as water-monitor.service -- the uv-managed .venv.
|
|
# scripts/retrain.sh trains into models/.staging, refuses to promote anything
|
|
# that is not a rain-enabled (hgb-v3) set covering the expected stations, then
|
|
# renames the bundles into place. The API reloads them on its next hourly
|
|
# precompute; no restart, so a failed run leaves the old models serving.
|
|
ExecStart=/bin/bash /opt/thailand-water-monitor/scripts/retrain.sh
|
|
# HistGradientBoosting is CPU-bound; cap threads so training cannot starve
|
|
# the API (docs/FLOOD_FORECASTING.md section 6 measured 4 as the sweet spot).
|
|
Environment=OMP_NUM_THREADS=4
|
|
Environment=PYTHONPATH=/opt/thailand-water-monitor
|
|
Environment=PYTHONUNBUFFERED=1
|
|
Nice=15
|
|
IOSchedulingClass=idle
|
|
# 15 stations at ~50 s each plus data load: 12 min observed on 2026-09-12.
|
|
TimeoutStartSec=45min
|
|
|
|
# Same sandbox as the API unit.
|
|
NoNewPrivileges=true
|
|
PrivateTmp=true
|
|
ProtectSystem=strict
|
|
ProtectHome=true
|
|
ReadWritePaths=/opt/thailand-water-monitor
|
|
CapabilityBoundingSet=
|
|
|
|
StandardOutput=journal
|
|
StandardError=journal
|
|
SyslogIdentifier=water-monitor-retrain
|