grabowski
7e642555b4
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
92 lines
3.6 KiB
Bash
92 lines
3.6 KiB
Bash
#!/usr/bin/env bash
|
|
#
|
|
# Retrain the flood forecast models safely. Run by water-monitor-retrain.timer
|
|
# (monthly) or by hand: sudo systemctl start water-monitor-retrain.service
|
|
#
|
|
# Why a script rather than ExecStart=train_flood_model.py:
|
|
# * train.py writes each station's bundle straight into models/ over ~12 min,
|
|
# and ping-collector reloads bundles by mtime on every forecast run.
|
|
# Training into a staging dir and mv-ing (atomic on one filesystem) means
|
|
# it never sees a half-written bundle or a mixed old/new set.
|
|
# * A run that produced gauge-only (v2) bundles, or trained too few stations,
|
|
# must NOT replace the deployed models. train.py already aborts on a
|
|
# missing rain series; this script re-checks the written metrics anyway.
|
|
# * No restart is needed: ping-collector reloads changed bundles
|
|
# (models/flood_*.json.gz, written by train.py next to each .joblib) on
|
|
# its next forecast run, so the new models are live within an hour.
|
|
#
|
|
# Exit codes: 0 ok, 2 training refused (see log), 3 verification failed.
|
|
set -euo pipefail
|
|
|
|
APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
|
|
PYTHON="${PYTHON:-${APP_DIR}/.venv/bin/python}"
|
|
MODELS_DIR="${APP_DIR}/models"
|
|
STAGE_DIR="${MODELS_DIR}/.staging"
|
|
# P.4A is NOT_TRAINABLE by design (17% fill); 15 of 16 is the normal outcome.
|
|
MIN_TRAINED="${MIN_TRAINED:-14}"
|
|
EXPECT_VERSION_PREFIX="${EXPECT_VERSION_PREFIX:-hgb-v3+}"
|
|
|
|
log() { printf '%s retrain: %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }
|
|
|
|
cd "${APP_DIR}"
|
|
[ -x "${PYTHON}" ] || { log "no interpreter at ${PYTHON} (run uv sync)"; exit 3; }
|
|
|
|
rm -rf "${STAGE_DIR}"
|
|
mkdir -p "${STAGE_DIR}"
|
|
log "training into ${STAGE_DIR} (python=${PYTHON}, OMP_NUM_THREADS=${OMP_NUM_THREADS:-unset})"
|
|
|
|
# train_flood_model.py exits 2 on a missing rain series (RainUnavailableError)
|
|
# instead of silently writing v2 bundles -- propagate that unchanged.
|
|
set +e
|
|
"${PYTHON}" scripts/train_flood_model.py --stations all --models-dir "${STAGE_DIR}" "$@"
|
|
rc=$?
|
|
set -e
|
|
if [ "${rc}" -ne 0 ]; then
|
|
log "training failed (exit ${rc}); deployed models untouched"
|
|
rm -rf "${STAGE_DIR}"
|
|
exit "${rc}"
|
|
fi
|
|
|
|
# Verify before promoting. Reads metrics.json from the stage dir.
|
|
VERSION="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
|
|
import json, sys
|
|
m = json.load(open(sys.argv[1]))
|
|
print(m["model_version"])
|
|
PY
|
|
)"
|
|
TRAINED="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
|
|
import json, sys
|
|
m = json.load(open(sys.argv[1]))
|
|
print(sum(1 for s in m["stations"].values() if s.get("status") == "trained"))
|
|
PY
|
|
)"
|
|
log "staged model_version=${VERSION} trained_stations=${TRAINED}"
|
|
|
|
case "${VERSION}" in
|
|
"${EXPECT_VERSION_PREFIX}"*) ;;
|
|
*)
|
|
log "REFUSING to deploy: version '${VERSION}' does not start with '${EXPECT_VERSION_PREFIX}'"
|
|
rm -rf "${STAGE_DIR}"
|
|
exit 3
|
|
;;
|
|
esac
|
|
if [ "${TRAINED}" -lt "${MIN_TRAINED}" ]; then
|
|
log "REFUSING to deploy: only ${TRAINED} stations trained (< ${MIN_TRAINED})"
|
|
rm -rf "${STAGE_DIR}"
|
|
exit 3
|
|
fi
|
|
|
|
# Promote: per-file rename is atomic; readers see either the old or the new
|
|
# bundle, never a partial one. Keep one previous generation for rollback.
|
|
mkdir -p "${MODELS_DIR}/.previous"
|
|
# flood_*.json.gz is the Rust forecaster's export of the same bundle.
|
|
for f in "${STAGE_DIR}"/flood_*.joblib "${STAGE_DIR}"/flood_*.json.gz "${STAGE_DIR}/metrics.json"; do
|
|
name="$(basename "${f}")"
|
|
if [ -f "${MODELS_DIR}/${name}" ]; then
|
|
mv -f "${MODELS_DIR}/${name}" "${MODELS_DIR}/.previous/${name}"
|
|
fi
|
|
mv -f "${f}" "${MODELS_DIR}/${name}"
|
|
done
|
|
rm -rf "${STAGE_DIR}"
|
|
log "deployed ${VERSION} (${TRAINED} stations); previous generation in models/.previous. ping-collector loads it on its next forecast run."
|