feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions

Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.

Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).

scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.

Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).

install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.

scripts/install_collector.sh

@@ -1,18 +1,29 @@
 #!/usr/bin/env bash
-# Install the Rust collector (rust/collector, `ping-collector`) and move work
+# Install the Rust service (rust/collector, `ping-collector`) and move work
 # from the Python app to it, or back.
 #
 #   sudo bash scripts/install_collector.sh                     # install binary + unit; checks only
 #   sudo bash scripts/install_collector.sh --cutover           # Rust collects (COLLECTION_ENABLED=false)
-#   sudo bash scripts/install_collector.sh --cutover-forecast  # Rust also forecasts and sends the
-#                                                              # ntfy notifications (FORECAST_IN_COLLECTOR=true)
+#   sudo bash scripts/install_collector.sh --cutover-forecast  # Rust also forecasts and notifies:
+#                                                              # ntfy + Matrix (FORECAST_IN_COLLECTOR=true)
+#   sudo bash scripts/install_collector.sh --check-api         # compare the Rust website/API with the
+#                                                              # Python app's, change nothing
+#   sudo bash scripts/install_collector.sh --cutover-api       # Rust serves the website and API on
+#                                                              # port 8000 (WEB_LISTEN); the Python app stops
+#   sudo bash scripts/install_collector.sh --rollback-api      # the Python app serves the website again
 #   sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + notifications back to Python
 #   sudo bash scripts/install_collector.sh --rollback          # Python does everything again
 #
 # Every install also exports the trained models for the Rust forecaster and
 # compares its forecast with Python's on the live data
 # (scripts/forecast_parity.py); --cutover-forecast refuses to switch when they
-# differ (FORCE=1 overrides).
+# differ. --cutover-api first runs `ping-collector serve` next to the Python
+# app and compares every route (scripts/api_parity.py); it refuses to switch
+# when they differ. FORCE=1 overrides either check.
+#
+# Once ping-collector sends the Matrix messages (FORECAST_IN_COLLECTOR=true
+# and MATRIX_* set), the old `run.py --alert-check` cron job is commented out
+# (backup next to the app); it repeated an alert on every run.
 #
 # The binary comes from BIN=/path/to/ping-collector (e.g. the "ping-collector-
 # linux-x86_64" CI artifact) or from rust/target/release/ping-collector, built
@@ -28,6 +39,8 @@ ENV_FILE="${APP_DIR}/.env"
 STATUS_FILE="${APP_DIR}/collector-status.json"
 PYTHON="${APP_DIR}/.venv/bin/python"
 REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+WEB_ADDR="${WEB_ADDR:-0.0.0.0:8000}"          # where the Python app listens today
+CHECK_PORT="${CHECK_PORT:-8001}"              # the Rust side during --check-api
 MODE="${1:-install}"
 
 log() { printf '\033[1;32m==>\033[0m %s\n' "$*"; }
@@ -64,24 +77,116 @@ show() {
     journalctl -u ping-collector -n 25 --no-pager || true
 }
 
+env_is() { grep -q "^$1=$2\s*$" "$ENV_FILE"; }
+
+wait_http() { # url seconds
+    local i
+    for i in $(seq "$2"); do
+        curl -fsS -o /dev/null --max-time 5 "$1" && return 0
+        sleep 1
+    done
+    return 1
+}
+
+# The Matrix messages come from ping-collector once it notifies; the old
+# cron job (`run.py --alert-check`, every 15 min, repeating while a level
+# stays high) is commented out, with a backup of the crontab.
+retire_matrix_cron() {
+    env_is FORECAST_IN_COLLECTOR true || return 0
+    grep -q '^MATRIX_ACCESS_TOKEN=..*' "$ENV_FILE" || return 0
+    local user backup
+    for user in root "$SERVICE_USER"; do
+        crontab -l -u "$user" 2>/dev/null | grep -v '^\s*#' | grep -q -- '--alert-check' || continue
+        backup="${APP_DIR}/crontab-${user}.$(date +%Y%m%d%H%M%S).bak"
+        crontab -l -u "$user" >"$backup"
+        sed '/--alert-check/ s|^\([^#]\)|# retired: Matrix is sent by ping-collector # \1|' "$backup" |
+            crontab -u "$user" -
+        log "Matrix: commented out the old --alert-check cron job of $user (backup: $backup)"
+    done
+    if grep -rlqs -- '--alert-check' /etc/cron.d /etc/crontab 2>/dev/null; then
+        warn "an --alert-check job is still in $(grep -rls -- '--alert-check' /etc/cron.d /etc/crontab | tr '\n' ' '): remove it, ping-collector sends Matrix now"
+    fi
+    if systemctl list-timers --all --no-pager 2>/dev/null | grep -qi alert; then
+        warn "a systemd timer mentions 'alert' (systemctl list-timers --all): if it runs --alert-check, disable it"
+    fi
+}
+
+# ping-collector serve next to the Python app; every route compared.
+api_check() {
+    local unit=ping-api-check ok=0
+    systemctl stop "$unit" 2>/dev/null || true
+    systemd-run --quiet --unit="$unit" --collect \
+        -p User="$SERVICE_USER" -p Group="$(id -gn "$SERVICE_USER")" \
+        -p WorkingDirectory="$APP_DIR" -p EnvironmentFile="$ENV_FILE" \
+        "${APP_DIR}/bin/ping-collector" serve --listen "127.0.0.1:${CHECK_PORT}" --no-analytics
+    if wait_http "http://127.0.0.1:${CHECK_PORT}/api/notifications" 30; then
+        as_service "$PYTHON" scripts/api_parity.py \
+            --python "http://127.0.0.1:${WEB_ADDR##*:}" --rust "http://127.0.0.1:${CHECK_PORT}" && ok=1
+    else
+        warn "ping-collector serve did not answer on port ${CHECK_PORT}:"
+        journalctl -u "$unit" -n 20 --no-pager || true
+    fi
+    systemctl stop "$unit" 2>/dev/null || true
+    [ "$ok" = 1 ]
+}
+
+rollback_api() {
+    if env_is WEB_LISTEN '..*'; then
+        log "The Python app serves the website and API again"
+        set_env WEB_LISTEN ""
+        systemctl restart ping-collector.service
+    fi
+    systemctl enable water-monitor.service
+    systemctl restart water-monitor.service
+}
+
+matrix_backups() {
+    ls "${APP_DIR}"/crontab-*.bak 2>/dev/null | sed 's/^/    /' || true
+}
+
 case "$MODE" in
     --rollback)
-        log "Stopping ping-collector; the Python app collects, forecasts and notifies again"
+        log "Stopping ping-collector; the Python app collects, serves, forecasts and notifies again"
         systemctl disable --now ping-collector.service 2>/dev/null || true
+        set_env WEB_LISTEN ""
         set_env FORECAST_IN_COLLECTOR false
         set_env COLLECTION_ENABLED true
+        systemctl enable water-monitor.service
         systemctl restart water-monitor.service
+        if [ -n "$(matrix_backups)" ]; then
+            warn "Matrix alerts came from ping-collector; to restore the old cron job: crontab -u USER BACKUP"
+            matrix_backups
+        fi
         log "Done. Check: journalctl -u water-monitor -n 50"
         exit 0
         ;;
+    --rollback-api)
+        rollback_api
+        log "Done. Check: journalctl -u water-monitor -n 50; curl -s http://127.0.0.1:${WEB_ADDR##*:}/health"
+        exit 0
+        ;;
     --rollback-forecast)
+        # the Python app forecasts in its own process: it must serve again
+        rollback_api
         log "Forecasts and notifications back to the Python app; ping-collector keeps collecting"
         set_env FORECAST_IN_COLLECTOR false
         systemctl restart ping-collector.service water-monitor.service
+        if [ -n "$(matrix_backups)" ]; then
+            warn "Matrix alerts stop with this (the Python app sends ntfy only); old cron job: crontab -u USER BACKUP"
+            matrix_backups
+        fi
         log "Done. Check: journalctl -u water-monitor -n 50"
         exit 0
         ;;
-    install | --cutover | --cutover-forecast) ;;
+    --check-api)
+        [ -x "${APP_DIR}/bin/ping-collector" ] || die "install first"
+        if env_is WEB_LISTEN '..*'; then
+            die "ping-collector already serves the website (WEB_LISTEN is set)"
+        fi
+        api_check && log "The Rust API answers as the Python app" || die "the API check found differences"
+        exit 0
+        ;;
+    install | --cutover | --cutover-forecast | --cutover-api) ;;
     *) die "unknown option: $MODE" ;;
 esac
 
@@ -130,12 +235,47 @@ case "$MODE" in
         fi
         log "Cutover: ping-collector also forecasts and notifies; the Python app serves its forecast"
         set_env FORECAST_IN_COLLECTOR true
+        retire_matrix_cron
         systemctl enable ping-collector.service
         systemctl restart ping-collector.service water-monitor.service
         show 30
-        log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback-forecast"
+        log "Next step: --cutover-api. Rollback: sudo bash scripts/install_collector.sh --rollback-forecast"
+        ;;
+    --cutover-api)
+        env_is FORECAST_IN_COLLECTOR true || die "ping-collector must forecast first: run --cutover-forecast"
+        if env_is WEB_LISTEN '..*'; then
+            log "ping-collector already serves the website; restarting it on the new binary"
+            retire_matrix_cron
+            systemctl restart ping-collector.service
+            show 10
+            exit 0
+        fi
+        api_ok=1
+        if [ "$(systemctl is-active water-monitor.service 2>/dev/null)" = active ]; then
+            log "API check: ping-collector serve on :${CHECK_PORT} next to the Python app, every route compared"
+            api_check || api_ok=0
+        else
+            warn "the Python app is not running: nothing to compare the API with"
+            api_ok=0
+        fi
+        if [ "$api_ok" != 1 ] && [ "${FORCE:-0}" != 1 ]; then
+            die "the API check did not pass; not switching (FORCE=1 overrides)"
+        fi
+        log "Cutover: ping-collector serves the website and API on ${WEB_ADDR}; the Python app stops"
+        set_env WEB_LISTEN "$WEB_ADDR"
+        retire_matrix_cron
+        systemctl disable --now water-monitor.service
+        systemctl restart ping-collector.service
+        if wait_http "http://127.0.0.1:${WEB_ADDR##*:}/health" 60; then
+            log "The website answers from ping-collector: $(curl -s --max-time 5 "http://127.0.0.1:${WEB_ADDR##*:}/scraping/status" | grep -o '"collector":"[^"]*"')"
+        else
+            warn "no answer on port ${WEB_ADDR##*:} yet; check journalctl -u ping-collector"
+        fi
+        show 5
+        log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback-api"
         ;;
     *)
+        retire_matrix_cron
         if [ "$(systemctl is-active ping-collector.service 2>/dev/null)" = active ]; then
             systemctl restart ping-collector.service
             log "Installed; ping-collector restarted on the new binary"