feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s

rust/collector replaces the collection half of the Python web app: RID hourly
gauges (with the late-report semantics: store everything, retry every 10 min
while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain +
water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and
upsert semantics, one UNNEST statement per table, Bangkok wall-clock time
regardless of the server time zone.

Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and
instead re-forecast + notify when the newest RID reading in the database
advances (polled every FORECAST_POLL_SECONDS) and once per clock hour;
/scraping/status then serves the collector's status file; /scrape/trigger
answers 409.

Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against
PostgreSQL); live parity run of both collectors into two empty databases:
all 9 tables row-for-row identical (scripts/collector_parity.py); the Python
API served health, measurements, stats, status and model forecasts from a
database filled only by ping-collector. 177 Python tests pass.

reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which
rustls refuses.

Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service;
CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service,
release artifact).
This commit is contained in:
grabowski committed 2026-10-17 22:20:02 +07:00
1 parent e3694d7591
commit 4e619aa2ae
26 files changed
+5836 -15

No files matched your search

+85
View File
@@ -0,0 +1,85 @@
# Rust services
Rewrite of the Ping River Monitor in Rust, one service at a time. The plan,
costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
| Crate | Replaces | Status |
| --- | --- | --- |
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: ready to deploy |
## ping-collector
Collects into the same PostgreSQL tables as the Python app, so the two can be
swapped (and swapped back) with no data migration. The Python web app keeps
serving the API, forecasting and sending ntfy alerts with
`COLLECTION_ENABLED=false`, and re-forecasts whenever the newest RID reading in
the database advances.
```text
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
# while RID is behind), HII/dams/rain once per clock hour
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
# and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
ping-collector check-db # connect, create missing tables, row counts
```
Configuration is the Python app's `.env` (read from the working directory, or
`--env-file`); real environment variables win. Used keys:
`POSTGRES_CONNECTION_STRING` (or `POSTGRES_HOST/PORT/DB/USER/PASSWORD`),
`RID_POLL_OFFSET_MINUTES` (5), `RID_RETRY_MINUTES` (10),
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
`LOG_LEVEL` / `RUST_LOG`. All timestamps are Asia/Bangkok wall-clock,
whatever the server's time zone.
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
stay in Python.
### Verified
- 20 tests: parsers, gap/schedule logic, and database tests against a real
PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC
rounding, dam metadata preservation).
- Live parity: the Python collectors and `ping-collector` run back-to-back
against the real upstream APIs into two empty databases produced identical
rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows,
35 dams x 2 days, 168 rain hours). Rerun with
`scripts/collector_parity.py` after changing a parser, table or upsert on
either side.
- The Python API with `COLLECTION_ENABLED=false` served `/health`,
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
forecasts from a database filled only by `ping-collector`.
### Develop
```bash
cd rust
cargo fmt --all && cargo clippy --all-targets -- -D warnings
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
```
Without `TEST_DATABASE_URL` the database tests skip. CI
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
Linux needs `pkg-config` and `libssl-dev`.
### Deploy
On the server, after `git pull` (the Python side of the switch ships in the
same commit):
```bash
cargo build --release --locked --manifest-path rust/Cargo.toml # as a normal user,
# or BIN=<CI artifact>
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
sudo bash scripts/install_collector.sh --rollback # back to Python collection
journalctl -u ping-collector -f
```