Northern-Thailand-Ping-River-Monitor/rust

Rust services

Rewrite of the Ping River Monitor in Rust, one service at a time. The plan, costs and the parity rules are in docs/RUST_REWRITE.md.

Crate	Replaces	Status
collector (ping-collector)	RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app	phase 1: ready to deploy

ping-collector

Collects into the same PostgreSQL tables as the Python app, so the two can be swapped (and swapped back) with no data migration. The Python web app keeps serving the API, forecasting and sending ntfy alerts with COLLECTION_ENABLED=false, and re-forecasts whenever the newest RID reading in the database advances.

ping-collector run                  # the service: startup gap fill, RID at :05 (+10 min retries
                                    # while RID is behind), HII/dams/rain once per clock hour
ping-collector once [--dry-run]     # one cycle of every source; --dry-run writes nothing
                                    # and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
ping-collector check-db             # connect, create missing tables, row counts

Configuration is the Python app's .env (read from the working directory, or --env-file); real environment variables win. Used keys: POSTGRES_CONNECTION_STRING (or POSTGRES_HOST/PORT/DB/USER/PASSWORD), RID_POLL_OFFSET_MINUTES (5), RID_RETRY_MINUTES (10), GAP_FILL_DAYS_ON_START (14), ENABLE_HII_COLLECTION, ENABLE_RESERVOIR_COLLECTION, ENABLE_RAIN_COLLECTION, HII_BASIN_CODE (6), STATION_CONFIG_PATH (falls back to the bundled src/data/stations.json), SCRAPING_STATUS_FILE (status JSON served by the API's /scraping/status), LOG_LEVEL / RUST_LOG. All timestamps are Asia/Bangkok wall-clock, whatever the server's time zone.

Not ported: the one-off archive backfills (scripts/backfill_*.py), which stay in Python.

Verified

• 20 tests: parsers, gap/schedule logic, and database tests against a real PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC rounding, dam metadata preservation).
• Live parity: the Python collectors and ping-collector run back-to-back against the real upstream APIs into two empty databases produced identical rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows, 35 dams x 2 days, 168 rain hours). Rerun with scripts/collector_parity.py after changing a parser, table or upsert on either side.
• The Python API with COLLECTION_ENABLED=false served /health, /measurements/latest, /api/stats, /scraping/status and model forecasts from a database filled only by ping-collector.

Develop

cd rust
cargo fmt --all && cargo clippy --all-targets -- -D warnings
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test

Without TEST_DATABASE_URL the database tests skip. CI (.gitea/workflows/rust.yml) runs them against a PostgreSQL service and uploads the Linux binary as the ping-collector-linux-x86_64 artifact.

HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams) only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on Linux needs pkg-config and libssl-dev.

Deploy

On the server, after git pull (the Python side of the switch ships in the same commit):

cargo build --release --locked --manifest-path rust/Cargo.toml   # as a normal user,
                                                                  # or BIN=<CI artifact>
sudo bash scripts/install_collector.sh             # install + dry run, nothing switched
sudo bash scripts/install_collector.sh --cutover   # Rust collects from now on
sudo bash scripts/install_collector.sh --rollback  # back to Python collection
journalctl -u ping-collector -f