feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night

Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared
.env) ping-collector, after each cycle that brought newer RID or HII readings
and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the
rain forecast, runs every exported model (rust/forecast), stores
forecast_history and forecast_current, and sends the ntfy notifications.
The Python API then serves forecast_current as /forecast (computing in-process
only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor
notifies.

ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics,
texts, hysteresis, capacity guard and notification_state keys, checked word
for word on 751 messages generated by Python (testdata/gen_notify_cases.py).
The change: for stations whose ThaiWater twin is the same sensor, a reading
newer than RID's drives the level alerts (marked as ThaiWater telemetry), and
the feed counts as stale only when neither source has a reading for 3 h. RID
alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the
nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports.

Commands: `forecast [--json] [--save] [--now] [--rain-file]`,
`notify --dry-run`. scripts/forecast_parity.py compares Python and the
collector on the live database (same moment, same rain); locally 48/48 rows
identical. install_collector.sh: models export + that check on every install,
--cutover-forecast (refuses on a failed check) and --rollback-forecast.

Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140).
Forecast values are written to NUMERIC as shortest round-trip text, so they
round as psycopg2's do. DB test schemas get a counter (parallel tests collided
on Windows' coarse clock).

scripts/forecast_parity.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Compare the Python forecast with ping-collector's on the live database.
+
+Run before setting FORECAST_IN_COLLECTOR=true, and after any change to src/ml
+or rust/forecast. On the server, as the service user from the app dir:
+
+    sudo -u water-monitor .venv/bin/python scripts/forecast_parity.py
+
+Both sides get the same moment and the same Open-Meteo rain series (fetched
+once here and handed to the collector as a file), so a difference is a porting
+bug, not timing. Exit 0 when every row matches: station, horizon, as_of,
+source, model and every level exactly; probabilities within a few units in the
+last place (numpy's and libm's exp may differ in the last bit).
+"""
+
+import argparse
+import datetime
+import json
+import math
+import os
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(REPO))
+
+ULPS = 4
+EXACT = (
+    "station_code",
+    "horizon_hours",
+    "as_of",
+    "source",
+    "model_version",
+    "trained_at",
+    "current_level",
+    "predicted_max_level",
+    "threshold_warning",
+    "threshold_danger",
+)
+
+
+def ulps(a: float, b: float) -> float:
+    if a == b:
+        return 0
+    return abs(a - b) / math.ulp(max(abs(a), abs(b)))
+
+
+def compare(py: list, rs: list) -> list:
+    problems = []
+    if len(py) != len(rs):
+        problems.append(f"row count: python {len(py)}, collector {len(rs)}")
+    for p, r in zip(py, rs):
+        where = f"{p.get('station_code')} {p.get('horizon_hours')}h"
+        for k in EXACT:
+            if p.get(k) != r.get(k):
+                problems.append(
+                    f"{where} {k}: python {p.get(k)!r}, collector {r.get(k)!r}"
+                )
+        for k in ("p_warning", "p_danger"):
+            d = ulps(float(p[k]), float(r[k]))
+            if d > ULPS:
+                problems.append(
+                    f"{where} {k}: python {p[k]!r}, collector {r[k]!r} ({d:.0f} ulp)"
+                )
+        ps, rs_ = p.get("stages"), r.get("stages")
+        if (ps is None) != (rs_ is None) or (ps and len(ps) != len(rs_)):
+            problems.append(f"{where} stages differ")
+        elif ps:
+            for a, b in zip(ps, rs_):
+                if a["stage"] != b["stage"] or a["level"] != b["level"]:
+                    problems.append(f"{where} stage {a['stage']} differs")
+                elif ulps(a["p_exceed"], b["p_exceed"]) > ULPS:
+                    problems.append(
+                        f"{where} stage {a['stage']} p_exceed {a['p_exceed']} vs {b['p_exceed']}"
+                    )
+    return problems
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument("--collector", default=str(REPO / "bin" / "ping-collector"))
+    parser.add_argument("--models-dir", default=str(REPO / "models"))
+    parser.add_argument(
+        "--db-url", default=None, help="default: the app's configuration"
+    )
+    parser.add_argument("--now", default=None, help="local time, default the clock")
+    args = parser.parse_args()
+
+    from src.ml import predict
+    from src.ml import rain as rain_mod
+    from src.ml.export import json_path_for
+
+    models = Path(args.models_dir)
+    stale = [
+        p.name
+        for p in models.glob("flood_*.joblib")
+        if not json_path_for(p).exists()
+        or json_path_for(p).stat().st_mtime < p.stat().st_mtime
+    ]
+    if stale:
+        print(
+            f"not exported for the collector: {', '.join(stale)}; run `python -m src.ml.export`"
+        )
+        return 2
+
+    now = (
+        datetime.datetime.fromisoformat(args.now)
+        if args.now
+        else datetime.datetime.now()
+    )
+    rain = rain_mod.serving_series()
+    if rain is None:
+        import pandas as pd
+
+        rain = pd.Series(dtype=float)
+    py = predict.get_latest_forecasts(
+        db_url=args.db_url, models_dir=models, now=now, rain=rain
+    )
+    py = json.loads(json.dumps(py, default=float))
+
+    with tempfile.TemporaryDirectory() as tmp:
+        rain_file = Path(tmp) / "rain.json"
+        rain_file.write_text(
+            json.dumps(
+                {
+                    "timestamps": [t.strftime("%Y-%m-%dT%H:%M:%S") for t in rain.index],
+                    "values": [None if math.isnan(v) else float(v) for v in rain],
+                }
+            )
+        )
+        env = dict(os.environ, MODELS_DIR=str(models), RUST_LOG="error")
+        if args.db_url:
+            env["POSTGRES_CONNECTION_STRING"] = args.db_url
+        out = subprocess.run(
+            [
+                args.collector,
+                "forecast",
+                "--json",
+                "--now",
+                now.isoformat(),
+                "--rain-file",
+                str(rain_file),
+            ],
+            capture_output=True,
+            text=True,
+            env=env,
+            cwd=REPO,
+            check=False,
+        )
+    if out.returncode != 0:
+        print(f"ping-collector forecast failed:\n{out.stderr}")
+        return 2
+    rs = json.loads(out.stdout)
+
+    problems = compare(py, rs)
+    models_used = sorted({r["model_version"] for r in py})
+    n_model = sum(r["source"] == "model" for r in py)
+    print(
+        f"as of {now:%Y-%m-%d %H:%M:%S}: {len(py)} rows ({n_model} from models {models_used}), "
+        f"rain {int(rain.notna().sum())} h"
+    )
+    if problems:
+        print(f"{len(problems)} DIFFERENCES:")
+        for p in problems[:40]:
+            print("  " + p)
+        return 1
+    exact = sum(p[k] == r[k] for p, r in zip(py, rs) for k in ("p_warning", "p_danger"))
+    print(
+        f"identical: every level and field; probabilities {exact}/{2 * len(py)} bit-identical, "
+        f"the rest within {ULPS} ulp"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/install_collector.sh

@@ -1,12 +1,18 @@
 #!/usr/bin/env bash
-# Install the Rust collector (rust/collector, `ping-collector`) and switch
-# data collection over to it, or back.
+# Install the Rust collector (rust/collector, `ping-collector`) and move work
+# from the Python app to it, or back.
 #
-#   sudo bash scripts/install_collector.sh             # install binary + unit, dry-run check
-#   sudo bash scripts/install_collector.sh --cutover   # install, then Rust collects:
-#                                                      #   COLLECTION_ENABLED=false, start
-#                                                      #   ping-collector, restart water-monitor
-#   sudo bash scripts/install_collector.sh --rollback  # Python collects again
+#   sudo bash scripts/install_collector.sh                     # install binary + unit; checks only
+#   sudo bash scripts/install_collector.sh --cutover           # Rust collects (COLLECTION_ENABLED=false)
+#   sudo bash scripts/install_collector.sh --cutover-forecast  # Rust also forecasts and sends the
+#                                                              # ntfy notifications (FORECAST_IN_COLLECTOR=true)
+#   sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + notifications back to Python
+#   sudo bash scripts/install_collector.sh --rollback          # Python does everything again
+#
+# Every install also exports the trained models for the Rust forecaster and
+# compares its forecast with Python's on the live data
+# (scripts/forecast_parity.py); --cutover-forecast refuses to switch when they
+# differ (FORCE=1 overrides).
 #
 # The binary comes from BIN=/path/to/ping-collector (e.g. the "ping-collector-
 # linux-x86_64" CI artifact) or from rust/target/release/ping-collector, built
@@ -20,10 +26,12 @@ APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
 SERVICE_USER="${SERVICE_USER:-water-monitor}" # as in scripts/ping-collector.service
 ENV_FILE="${APP_DIR}/.env"
 STATUS_FILE="${APP_DIR}/collector-status.json"
+PYTHON="${APP_DIR}/.venv/bin/python"
 REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 MODE="${1:-install}"
 
 log() { printf '\033[1;32m==>\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[warn]\033[0m %s\n' "$*" >&2; }
 die() { printf '\033[1;31m[error]\033[0m %s\n' "$*" >&2; exit 1; }
 
 [ "$(id -u)" -eq 0 ] || die "run as root (sudo)"
@@ -38,15 +46,44 @@ set_env() {
     fi
 }
 
-if [ "$MODE" = "--rollback" ]; then
-    log "Stopping ping-collector; the Python app collects again"
-    systemctl disable --now ping-collector.service 2>/dev/null || true
-    set_env COLLECTION_ENABLED true
-    systemctl restart water-monitor.service
-    log "Done. Check: journalctl -u water-monitor -n 50"
-    exit 0
-fi
-case "$MODE" in install | --cutover) ;; *) die "unknown option: $MODE" ;; esac
+# Run a command the way the services run: same user, working directory and
+# EnvironmentFile parsing (systemd's, not a .env library's).
+as_service() {
+    if systemd-run --version >/dev/null 2>&1; then
+        systemd-run --quiet --pipe --wait --collect \
+            -p User="$SERVICE_USER" -p Group="$(id -gn "$SERVICE_USER")" \
+            -p WorkingDirectory="$APP_DIR" -p EnvironmentFile="$ENV_FILE" "$@"
+    else
+        (cd "$APP_DIR" && runuser -u "$SERVICE_USER" -- "$@")
+    fi
+}
+
+show() {
+    sleep "$1"
+    systemctl --no-pager --lines=0 status ping-collector.service || true
+    journalctl -u ping-collector -n 25 --no-pager || true
+}
+
+case "$MODE" in
+    --rollback)
+        log "Stopping ping-collector; the Python app collects, forecasts and notifies again"
+        systemctl disable --now ping-collector.service 2>/dev/null || true
+        set_env FORECAST_IN_COLLECTOR false
+        set_env COLLECTION_ENABLED true
+        systemctl restart water-monitor.service
+        log "Done. Check: journalctl -u water-monitor -n 50"
+        exit 0
+        ;;
+    --rollback-forecast)
+        log "Forecasts and notifications back to the Python app; ping-collector keeps collecting"
+        set_env FORECAST_IN_COLLECTOR false
+        systemctl restart ping-collector.service water-monitor.service
+        log "Done. Check: journalctl -u water-monitor -n 50"
+        exit 0
+        ;;
+    install | --cutover | --cutover-forecast) ;;
+    *) die "unknown option: $MODE" ;;
+esac
 
 BIN="${BIN:-${REPO_DIR}/rust/target/release/ping-collector}"
 [ -x "$BIN" ] || die "no binary at $BIN: build it (see header) or pass BIN=/path/to/ping-collector"
@@ -63,26 +100,47 @@ touch "$STATUS_FILE"
 chown "${SERVICE_USER}:$(id -gn "$SERVICE_USER")" "$STATUS_FILE"
 
 log "Dry run against the configured database (writes nothing)"
-# Run it the way the service will: same user, working directory and
-# EnvironmentFile parsing (systemd's, not a .env library's).
-if systemd-run --version >/dev/null 2>&1; then
-    systemd-run --quiet --pipe --wait --collect \
-        -p User="$SERVICE_USER" -p Group="$(id -gn "$SERVICE_USER")" \
-        -p WorkingDirectory="$APP_DIR" -p EnvironmentFile="$ENV_FILE" \
-        "${APP_DIR}/bin/ping-collector" once --dry-run
+as_service "${APP_DIR}/bin/ping-collector" once --dry-run
+
+forecast_ok=1
+if compgen -G "${APP_DIR}/models/flood_*.joblib" >/dev/null; then
+    log "Exporting the trained models for the Rust forecaster (models/flood_*.json.gz)"
+    as_service "$PYTHON" -m src.ml.export
+    log "Forecast check: Python and ping-collector on the same data, moment and rain"
+    as_service "$PYTHON" scripts/forecast_parity.py || forecast_ok=0
 else
-    (cd "$APP_DIR" && "${APP_DIR}/bin/ping-collector" --env-file "$ENV_FILE" once --dry-run)
+    warn "no trained models in ${APP_DIR}/models: forecast check skipped"
+    forecast_ok=0
 fi
 
-if [ "$MODE" = "--cutover" ]; then
-    log "Cutover: ping-collector collects, the Python app forecasts and notifies"
-    set_env COLLECTION_ENABLED false
-    systemctl enable --now ping-collector.service
-    systemctl restart water-monitor.service
-    sleep 20
-    systemctl --no-pager --lines=0 status ping-collector.service || true
-    journalctl -u ping-collector -n 20 --no-pager || true
-    log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback"
-else
-    log "Installed, not started. Switch over with: sudo bash scripts/install_collector.sh --cutover"
-fi
+case "$MODE" in
+    --cutover)
+        log "Cutover: ping-collector collects; the Python app forecasts and notifies"
+        set_env COLLECTION_ENABLED false
+        systemctl enable ping-collector.service
+        systemctl restart ping-collector.service water-monitor.service
+        show 20
+        log "Next step: --cutover-forecast. Rollback: sudo bash scripts/install_collector.sh --rollback"
+        ;;
+    --cutover-forecast)
+        grep -q '^COLLECTION_ENABLED=false' "$ENV_FILE" ||
+            die "ping-collector must collect first: run --cutover"
+        if [ "$forecast_ok" != 1 ] && [ "${FORCE:-0}" != 1 ]; then
+            die "the forecast check did not pass; not switching (FORCE=1 overrides)"
+        fi
+        log "Cutover: ping-collector also forecasts and notifies; the Python app serves its forecast"
+        set_env FORECAST_IN_COLLECTOR true
+        systemctl enable ping-collector.service
+        systemctl restart ping-collector.service water-monitor.service
+        show 30
+        log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback-forecast"
+        ;;
+    *)
+        if [ "$(systemctl is-active ping-collector.service 2>/dev/null)" = active ]; then
+            systemctl restart ping-collector.service
+            log "Installed; ping-collector restarted on the new binary"
+        else
+            log "Installed, not started. Switch over with: sudo bash scripts/install_collector.sh --cutover"
+        fi
+        ;;
+esac

scripts/ping-collector.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=Ping River Monitor data collector (Rust: RID, HII, dams, Open-Meteo)
+Description=Ping River Monitor collector (Rust: RID, HII, dams, Open-Meteo; forecast + ntfy)
 Documentation=https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/src/branch/master/rust
 After=network-online.target
 Wants=network-online.target
@@ -15,7 +15,9 @@ RestartSec=30
 TimeoutStopSec=20
 
 # Same .env as water-monitor.service (POSTGRES_*, RID_*, GAP_FILL_*,
-# SCRAPING_STATUS_FILE). The Python app must run with COLLECTION_ENABLED=false.
+# SCRAPING_STATUS_FILE, FORECAST_IN_COLLECTOR, NTFY_*). The Python app must run
+# with COLLECTION_ENABLED=false. Models are read from models/ (relative to
+# WorkingDirectory).
 EnvironmentFile=/opt/thailand-water-monitor/.env
 
 # Security settings