Northern-Thailand-Ping-River-Monitor/rust

Rust services

Rewrite of the Ping River Monitor in Rust, one service at a time. The plan, costs and the parity rules are in docs/RUST_REWRITE.md.

Crate	Replaces	Status
collector (ping-collector)	RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app	phase 1: in production since 2026-10-17
forecast (ping-forecast, library)	the flood forecast (src/ml serving path); run by ping-collector with ntfy	phase 2: built, parity-verified, cutover pending

ping-collector

Collects into the same PostgreSQL tables as the Python app, so the two can be swapped (and swapped back) with no data migration. The Python web app keeps serving the API, forecasting and sending ntfy alerts with COLLECTION_ENABLED=false, and re-forecasts whenever the newest RID reading in the database advances.

ping-collector run                  # the service: startup gap fill, RID at :05 (+10 min retries
                                    # while RID is behind), HII/dams/rain once per clock hour;
                                    # with FORECAST_IN_COLLECTOR=true also forecast + ntfy
ping-collector once [--dry-run]     # one cycle of every source; --dry-run writes nothing
                                    # and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
ping-collector check-db             # connect, create missing tables, row counts
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
                                    # forecast from the database and print it
ping-collector notify --dry-run     # the readings the notifier judges, and what it would send

Configuration is the Python app's .env (read from the working directory, or --env-file) with python-dotenv's rules, verified on 614 generated inputs (collector/testdata/): an unparsable line is skipped with a warning giving only its line number, never its value. Variables already set (e.g. by systemd's EnvironmentFile=) win. Used keys: POSTGRES_CONNECTION_STRING (or POSTGRES_HOST/PORT/DB/USER/PASSWORD), RID_POLL_OFFSET_MINUTES (5), RID_RETRY_MINUTES (10), GAP_FILL_DAYS_ON_START (14), ENABLE_HII_COLLECTION, ENABLE_RESERVOIR_COLLECTION, ENABLE_RAIN_COLLECTION, HII_BASIN_CODE (6), STATION_CONFIG_PATH (falls back to the bundled src/data/stations.json), SCRAPING_STATUS_FILE (status JSON served by the API's /scraping/status), LOG_LEVEL / RUST_LOG; for phase 2 FORECAST_IN_COLLECTOR (false), MODELS_DIR (models), NTFY_SERVER / NTFY_PUBLISH_URL / NTFY_TOPIC_PREFIX / NTFY_TOKEN / PUBLIC_URL (as the Python app), RID_STALE_ALERT_HOURS (18, 0 = off). All timestamps are Asia/Bangkok wall-clock, whatever the server's time zone.

Not ported: the one-off archive backfills (scripts/backfill_*.py), which stay in Python.

Verified

• 20 tests: parsers, gap/schedule logic, and database tests against a real PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC rounding, dam metadata preservation).
• Live parity: the Python collectors and ping-collector run back-to-back against the real upstream APIs into two empty databases produced identical rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows, 35 dams x 2 days, 168 rain hours). Rerun with scripts/collector_parity.py after changing a parser, table or upsert on either side.
• The Python API with COLLECTION_ENABLED=false served /health, /measurements/latest, /api/stats, /scraping/status and model forecasts from a database filled only by ping-collector.

Forecast and notifications (phase 2)

With FORECAST_IN_COLLECTOR=true in the shared .env, ping-collector runs the flood forecast after every cycle that brought newer RID or HII readings (and at least hourly), stores it in forecast_history and forecast_current (the Python API serves the latter as /forecast, falling back to computing itself when it is older than FORECAST_MAX_AGE_SECONDS), and sends the ntfy notifications. The Python app then neither forecasts nor notifies.

• Models stay trained in Python. src/ml/export.py writes models/flood_<station>.json.gz next to each joblib bundle (train_all() does it, retrain.sh promotes it, python -m src.ml.export exports existing bundles); the collector reloads changed files on its next run.
• Shared constants (thresholds, flood stages, upstream leads, notify settings) are compiled in from forecast/flood_config.json, generated by uv run python -m src.ml.export --config. tests/test_model_export.py fails while it differs from Python, and a change needs a collector rebuild.
• Notifications: same topics, texts, hysteresis and notification_state keys as src/notify.py, so switching sends nothing twice. One change: for the stations whose ThaiWater twin is the same sensor, a reading newer than RID's (RID's report lags 2-10 h at night) drives the level alerts, marked "ThaiWater telemetry; the RID report is not in yet". "Gauge feed stale" now means no source has a reading for 3 h; RID alone lagging is reported once after RID_STALE_ALERT_HOURS.
• Memory: 70 MB peak with all 15 models loaded (24-byte tree nodes).

Verified:

• forecast/tests/parity.rs replays fixtures from the Python production path (forecast/testdata/gen_forecast_fixtures.py: the 2024 flood rising and at its peak with RID lagging behind HII, a quiet window with dam data, a stale-feed window, moved thresholds). HII fill, every feature and every level are bit-identical; with all 15 local models 377 of 384 probabilities are bit-identical and the rest within 2 ulp (exp). CI checks the P.1 model.
• scripts/forecast_parity.py does the same on the live database: Python and ping-collector forecast --json at the same moment with the same rain series. The install script runs it before --cutover-forecast.
• collector/testdata/gen_notify_cases.py: 652 hourly steps through src/notify.py (thresholds, capacity guard, outlook, feed gaps, ntfy outages); the Rust notifier sends the same 751 messages word for word.

Develop

cd rust
cargo fmt --all && cargo clippy --all-targets -- -D warnings
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test

Without TEST_DATABASE_URL the database tests skip. CI (.gitea/workflows/rust.yml) runs them against a PostgreSQL service and uploads the Linux binary as the ping-collector-linux-x86_64 artifact.

HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams) only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on Linux needs pkg-config and libssl-dev.

Deploy

Needs Rust >= 1.88. Distro cargo packages are older (resolver 3 is not valid / edition 2024 errors), so install the toolchain once with rustup:

apt-get install -y build-essential pkg-config libssl-dev curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
. "$HOME/.cargo/env"        # puts ~/.cargo/bin before /usr/bin/cargo
cargo --version             # >= 1.88

Then on the server, after git pull (the Python side of the switch ships in the same commit):

cargo build --release --locked --manifest-path rust/Cargo.toml   # or BIN=<CI artifact>
sudo bash scripts/install_collector.sh                     # install + checks, nothing switched
sudo bash scripts/install_collector.sh --cutover           # Rust collects from now on
sudo bash scripts/install_collector.sh --cutover-forecast  # Rust also forecasts and notifies
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
sudo bash scripts/install_collector.sh --rollback          # Python does everything again
journalctl -u ping-collector -f

Every install exports the models and runs scripts/forecast_parity.py; --cutover-forecast refuses to switch if it fails (FORCE=1 overrides).