98 lines
4.4 KiB
Markdown
98 lines
4.4 KiB
Markdown
# Rust services
|
|
|
|
Rewrite of the Ping River Monitor in Rust, one service at a time. The plan,
|
|
costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
|
|
|
| Crate | Replaces | Status |
|
|
| --- | --- | --- |
|
|
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
|
|
|
|
## ping-collector
|
|
|
|
Collects into the same PostgreSQL tables as the Python app, so the two can be
|
|
swapped (and swapped back) with no data migration. The Python web app keeps
|
|
serving the API, forecasting and sending ntfy alerts with
|
|
`COLLECTION_ENABLED=false`, and re-forecasts whenever the newest RID reading in
|
|
the database advances.
|
|
|
|
```text
|
|
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
|
|
# while RID is behind), HII/dams/rain once per clock hour
|
|
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
|
# and compares the RID fetch with the database
|
|
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
|
ping-collector check-db # connect, create missing tables, row counts
|
|
```
|
|
|
|
Configuration is the Python app's `.env` (read from the working directory, or
|
|
`--env-file`) with python-dotenv's rules, verified on 614 generated inputs
|
|
(`collector/testdata/`): an unparsable line is skipped with a warning giving
|
|
only its line number, never its value. Variables already set (e.g. by systemd's
|
|
`EnvironmentFile=`) win. Used keys:
|
|
`POSTGRES_CONNECTION_STRING` (or `POSTGRES_HOST/PORT/DB/USER/PASSWORD`),
|
|
`RID_POLL_OFFSET_MINUTES` (5), `RID_RETRY_MINUTES` (10),
|
|
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
|
|
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
|
|
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
|
|
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
|
|
`LOG_LEVEL` / `RUST_LOG`. All timestamps are Asia/Bangkok wall-clock,
|
|
whatever the server's time zone.
|
|
|
|
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
|
|
stay in Python.
|
|
|
|
### Verified
|
|
|
|
- 20 tests: parsers, gap/schedule logic, and database tests against a real
|
|
PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC
|
|
rounding, dam metadata preservation).
|
|
- Live parity: the Python collectors and `ping-collector` run back-to-back
|
|
against the real upstream APIs into two empty databases produced identical
|
|
rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows,
|
|
35 dams x 2 days, 168 rain hours). Rerun with
|
|
`scripts/collector_parity.py` after changing a parser, table or upsert on
|
|
either side.
|
|
- The Python API with `COLLECTION_ENABLED=false` served `/health`,
|
|
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
|
|
forecasts from a database filled only by `ping-collector`.
|
|
|
|
### Develop
|
|
|
|
```bash
|
|
cd rust
|
|
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
|
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
|
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
|
```
|
|
|
|
Without `TEST_DATABASE_URL` the database tests skip. CI
|
|
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
|
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
|
|
|
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
|
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
|
Linux needs `pkg-config` and `libssl-dev`.
|
|
|
|
### Deploy
|
|
|
|
Needs Rust >= 1.88. Distro `cargo` packages are older (`resolver 3 is not
|
|
valid` / `edition 2024` errors), so install the toolchain once with rustup:
|
|
|
|
```bash
|
|
apt-get install -y build-essential pkg-config libssl-dev curl
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
|
|
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
|
|
cargo --version # >= 1.88
|
|
```
|
|
|
|
Then on the server, after `git pull` (the Python side of the switch ships in
|
|
the same commit):
|
|
|
|
```bash
|
|
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
|
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
|
|
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
|
sudo bash scripts/install_collector.sh --rollback # back to Python collection
|
|
journalctl -u ping-collector -f
|
|
```
|