Northern-Thailand-Ping-River-Monitor/rust

Rust: the service

Since October 2026 the whole Ping River Monitor service is one Rust binary, ping-collector; Python is left with model training and evaluation (src/ml) and the archive backfills. How it got here, and the parity rules the rewrite followed: docs/RUST_REWRITE.md.

Crate	What
collector (ping-collector)	collection (RID, HII/ThaiWater, RID dams, Open-Meteo), the forecast run, ntfy + Matrix notifications, the website and API (collector/src/web)
forecast (ping-forecast, library)	flood-forecast inference from the models trained in Python: features, HII gap fill, gradient-boosted trees, heuristic fallback

ping-collector

ping-collector run                  # the service (systemd): startup gap fill, RID at :05 (+10 min
                                    # retries while RID is behind), HII/dams/rain once per clock
                                    # hour; FORECAST_IN_COLLECTOR=true: forecast + ntfy/Matrix
                                    # after each cycle with newer readings; WEB_LISTEN: website/API
ping-collector serve [--listen A] [--no-analytics]
                                    # the website and API alone (default 127.0.0.1:8001)
ping-collector once [--dry-run]     # one cycle of every source; --dry-run writes nothing
                                    # and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
ping-collector check-db             # connect, create missing tables, row counts
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
                                    # forecast from the database and print it
ping-collector catch-up-forecasts [--hours N]
                                    # forecast the hours whose readings came late (RID's
                                    # overnight batch) as a run then would have; default 336 h
ping-collector notify --dry-run     # the readings the notifier judges, and what it would send

Configuration: the environment, or .env in the working directory (--env-file for another), every key with its default in .env.example. .env is parsed with python-dotenv's rules (collector/src/envfile.rs, verified on 614 generated inputs) because the Python tools read the same file: an unparsable line is skipped with a warning giving only its line number, never its value. Variables already set (e.g. by systemd's EnvironmentFile=) win. All timestamps are Asia/Bangkok wall-clock, whatever the server's time zone.

Collection

The tables, upserts and parsers are the ones the Python collectors had (the DDL is in collector/src/db.rs; the Python backfill stores create the same). RID's evening report often lags 1-2 h: every poll stores every reading the report has, yesterday's report is re-fetched while it has holes, and the startup gap fill repairs the last GAP_FILL_DAYS_ON_START days.

Forecast and notifications

After every cycle that brought newer RID or HII readings (and at least hourly) the forecast runs over the last 336 h, is stored in forecast_history (for skill tracking) and forecast_current, and the notifier compares the readings and forecasts with notification_state.

• Models are trained in Python. src/ml/export.py writes models/flood_<station>.json.gz next to each joblib bundle (train_all() does it; python -m src.ml.export for existing bundles); the service reloads changed files on its next run, so a retrain needs no restart.
• Shared constants (thresholds, flood stages, upstream leads, notifier settings) are compiled in from forecast/flood_config.json, generated by uv run python -m src.ml.export --config. tests/test_model_export.py fails while it differs from Python; a change needs a rebuild.
• ntfy topics, texts and hysteresis: docs/NOTIFICATIONS.md. For the stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts, marked as ThaiWater telemetry. "Gauge feed stale" means no source has a reading for 3 h, except during RID's nightly pause (21:00 to FEED_NIGHT_UNTIL_HOUR, 9) while the river is quiet; RID alone lagging is reported once after RID_STALE_ALERT_HOURS.
• Matrix (MATRIX_*): the room gets the same transitions (per station, outlook, feed; not ntfy's per-topic summaries), each once, tracked under its own matrix: keys in notification_state, so a Matrix outage is retried on Matrix only. docs/MATRIX_QUICK_START.md.
• Memory: under 100 MB resident in production, all 15 models loaded (24-byte tree nodes); the unit allows 512 MB.

Website and API

collector/src/web (axum) serves the dashboard (STATIC_DIR/dashboard.html, default src/static, loaded at start), /static, the crawler files, /docs + /redoc + /openapi.json, and the read API, with gzip, Cache-Control per path, CORS (CORS_ALLOW_ORIGINS), Umami API tracking (UMAMI_*) and stale-while-revalidate response caches with byte budgets. Station editing, /config and POST /scrape/trigger need ADMIN_API_KEY (X-API-Key header).

• Parameters are declared once per route (web/params.rs): the same list validates the request (pydantic-style 422s) and builds the OpenAPI document (web/openapi.rs), so /docs cannot drift from what is accepted.
• The response formats are the retired FastAPI app's, kept so clients did not notice the switch: field order, isoformat() timestamps, Python's round(), numpy's pairwise sums for the catchment means (web/pyfmt.rs), {"detail": ...} errors.
• An old stored forecast is served marked X-Data-Stale: true; /health counts current ThaiWater telemetry of the same gauges while RID lags.

Develop

cd rust
cargo fmt --all && cargo clippy --all-targets -- -D warnings
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
CARGO_TARGET_DIR=target/msrv cargo +1.88 check --all-targets --locked   # the minimum version

Without TEST_DATABASE_URL the database tests skip. CI (.gitea/workflows/rust.yml) runs them against a PostgreSQL service and uploads the Linux binary as the ping-collector-linux-x86_64 artifact.

• The forecast's parity with the Python reference: forecast/tests/parity.rs replays fixtures from the Python production path (forecast/testdata/gen_forecast_fixtures.py; regenerate after a change to src/ml/features.py, predict.py, the HII fill or the dam frame). Features and levels must stay bit-identical, probabilities within a few ulp. scripts/forecast_parity.py compares the two on a live database.
• The notifier's texts are pinned by collector/testdata/notify_python_cases.json.gz (751 messages recorded from the Python notifier before it was retired).

HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams) only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on Linux needs pkg-config and libssl-dev.

Deploy

Needs Rust >= 1.88. Distro cargo packages are older (resolver 3 is not valid / edition 2024 errors), so install the toolchain once with rustup:

apt-get install -y build-essential pkg-config libssl-dev curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
. "$HOME/.cargo/env"        # puts ~/.cargo/bin before /usr/bin/cargo
cargo --version             # >= 1.88

Then on the server, in the checkout:

git pull
cargo build --release --locked --manifest-path rust/Cargo.toml   # or BIN=<CI artifact>
sudo bash scripts/install.sh
journalctl -u ping-collector -f

scripts/install.sh installs the binary and the units, syncs the Python training environment, runs a dry run of every source and the forecast check, restarts the service and waits for the website (details in its header).