grabowski
5c4f464383
Phase 2a of docs/RUST_REWRITE.md. rust/forecast (ping-forecast) ports the serving half of src/ml: the HII mirror gap fill, the hourly grid, every feature (pandas' Kahan rolling sums included), the Mae Ngat dam frame, HistGradientBoosting tree inference, the heuristic fallback, stale-threshold handling and the P.1 flood-stage probabilities. Hand-over from Python (src/ml/export.py): - models/flood_<station>.json.gz next to each joblib bundle; train_all() writes it, retrain.sh promotes it, `python -m src.ml.export` exports existing ones. - rust/forecast/flood_config.json: thresholds, stages, leads and notify constants, compiled in. tests/test_model_export.py fails while it differs from the Python source. Parity (rust/forecast/tests/parity.rs, fixtures from testdata/gen_forecast_fixtures.py on the production code path): four scenarios incl. the 2024 flood with RID lagging behind HII. Merged readings, all features and all levels identical; with all 15 local models 377/384 probabilities bit-identical, the rest within 2 ulp (exp). Python refactor without behaviour change: predict.forecasts_from_long, and get_latest_forecasts/load_latest/serving_frame take now/rain for reproducible comparisons; the HII query orders its rows.
Rust services
Rewrite of the Ping River Monitor in Rust, one service at a time. The plan, costs and the parity rules are in docs/RUST_REWRITE.md.
| Crate | Replaces | Status |
|---|---|---|
collector (ping-collector) |
RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
ping-collector
Collects into the same PostgreSQL tables as the Python app, so the two can be
swapped (and swapped back) with no data migration. The Python web app keeps
serving the API, forecasting and sending ntfy alerts with
COLLECTION_ENABLED=false, and re-forecasts whenever the newest RID reading in
the database advances.
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
# while RID is behind), HII/dams/rain once per clock hour
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
# and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
ping-collector check-db # connect, create missing tables, row counts
Configuration is the Python app's .env (read from the working directory, or
--env-file) with python-dotenv's rules, verified on 614 generated inputs
(collector/testdata/): an unparsable line is skipped with a warning giving
only its line number, never its value. Variables already set (e.g. by systemd's
EnvironmentFile=) win. Used keys:
POSTGRES_CONNECTION_STRING (or POSTGRES_HOST/PORT/DB/USER/PASSWORD),
RID_POLL_OFFSET_MINUTES (5), RID_RETRY_MINUTES (10),
GAP_FILL_DAYS_ON_START (14), ENABLE_HII_COLLECTION,
ENABLE_RESERVOIR_COLLECTION, ENABLE_RAIN_COLLECTION, HII_BASIN_CODE (6),
STATION_CONFIG_PATH (falls back to the bundled src/data/stations.json),
SCRAPING_STATUS_FILE (status JSON served by the API's /scraping/status),
LOG_LEVEL / RUST_LOG. All timestamps are Asia/Bangkok wall-clock,
whatever the server's time zone.
Not ported: the one-off archive backfills (scripts/backfill_*.py), which
stay in Python.
Verified
- 20 tests: parsers, gap/schedule logic, and database tests against a real PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC rounding, dam metadata preservation).
- Live parity: the Python collectors and
ping-collectorrun back-to-back against the real upstream APIs into two empty databases produced identical rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows, 35 dams x 2 days, 168 rain hours). Rerun withscripts/collector_parity.pyafter changing a parser, table or upsert on either side. - The Python API with
COLLECTION_ENABLED=falseserved/health,/measurements/latest,/api/stats,/scraping/statusand model forecasts from a database filled only byping-collector.
Develop
cd rust
cargo fmt --all && cargo clippy --all-targets -- -D warnings
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
Without TEST_DATABASE_URL the database tests skip. CI
(.gitea/workflows/rust.yml) runs them against a PostgreSQL service and
uploads the Linux binary as the ping-collector-linux-x86_64 artifact.
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
Linux needs pkg-config and libssl-dev.
Deploy
Needs Rust >= 1.88. Distro cargo packages are older (resolver 3 is not valid / edition 2024 errors), so install the toolchain once with rustup:
apt-get install -y build-essential pkg-config libssl-dev curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
cargo --version # >= 1.88
Then on the server, after git pull (the Python side of the switch ships in
the same commit):
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
sudo bash scripts/install_collector.sh --rollback # back to Python collection
journalctl -u ping-collector -f