feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
rust/collector replaces the collection half of the Python web app: RID hourly gauges (with the late-report semantics: store everything, retry every 10 min while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain + water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and upsert semantics, one UNNEST statement per table, Bangkok wall-clock time regardless of the server time zone. Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and instead re-forecast + notify when the newest RID reading in the database advances (polled every FORECAST_POLL_SECONDS) and once per clock hour; /scraping/status then serves the collector's status file; /scrape/trigger answers 409. Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against PostgreSQL); live parity run of both collectors into two empty databases: all 9 tables row-for-row identical (scripts/collector_parity.py); the Python API served health, measurements, stats, status and model forecasts from a database filled only by ping-collector. 177 Python tests pass. reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which rustls refuses. Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service; CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service, release artifact).
This commit is contained in:
1 parent
e3694d7591
commit
4e619aa2ae
26 files changed
+5836
-15
No files matched your search
@@ -0,0 +1,85 @@
|
||||
# Rust services
|
||||
|
||||
Rewrite of the Ping River Monitor in Rust, one service at a time. The plan,
|
||||
costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
||||
|
||||
| Crate | Replaces | Status |
|
||||
| --- | --- | --- |
|
||||
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: ready to deploy |
|
||||
|
||||
## ping-collector
|
||||
|
||||
Collects into the same PostgreSQL tables as the Python app, so the two can be
|
||||
swapped (and swapped back) with no data migration. The Python web app keeps
|
||||
serving the API, forecasting and sending ntfy alerts with
|
||||
`COLLECTION_ENABLED=false`, and re-forecasts whenever the newest RID reading in
|
||||
the database advances.
|
||||
|
||||
```text
|
||||
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
|
||||
# while RID is behind), HII/dams/rain once per clock hour
|
||||
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
||||
# and compares the RID fetch with the database
|
||||
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
||||
ping-collector check-db # connect, create missing tables, row counts
|
||||
```
|
||||
|
||||
Configuration is the Python app's `.env` (read from the working directory, or
|
||||
`--env-file`); real environment variables win. Used keys:
|
||||
`POSTGRES_CONNECTION_STRING` (or `POSTGRES_HOST/PORT/DB/USER/PASSWORD`),
|
||||
`RID_POLL_OFFSET_MINUTES` (5), `RID_RETRY_MINUTES` (10),
|
||||
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
|
||||
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
|
||||
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
|
||||
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
|
||||
`LOG_LEVEL` / `RUST_LOG`. All timestamps are Asia/Bangkok wall-clock,
|
||||
whatever the server's time zone.
|
||||
|
||||
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
|
||||
stay in Python.
|
||||
|
||||
### Verified
|
||||
|
||||
- 20 tests: parsers, gap/schedule logic, and database tests against a real
|
||||
PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC
|
||||
rounding, dam metadata preservation).
|
||||
- Live parity: the Python collectors and `ping-collector` run back-to-back
|
||||
against the real upstream APIs into two empty databases produced identical
|
||||
rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows,
|
||||
35 dams x 2 days, 168 rain hours). Rerun with
|
||||
`scripts/collector_parity.py` after changing a parser, table or upsert on
|
||||
either side.
|
||||
- The Python API with `COLLECTION_ENABLED=false` served `/health`,
|
||||
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
|
||||
forecasts from a database filled only by `ping-collector`.
|
||||
|
||||
### Develop
|
||||
|
||||
```bash
|
||||
cd rust
|
||||
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
||||
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
||||
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
||||
```
|
||||
|
||||
Without `TEST_DATABASE_URL` the database tests skip. CI
|
||||
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
||||
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
||||
|
||||
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
||||
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
||||
Linux needs `pkg-config` and `libssl-dev`.
|
||||
|
||||
### Deploy
|
||||
|
||||
On the server, after `git pull` (the Python side of the switch ships in the
|
||||
same commit):
|
||||
|
||||
```bash
|
||||
cargo build --release --locked --manifest-path rust/Cargo.toml # as a normal user,
|
||||
# or BIN=<CI artifact>
|
||||
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
|
||||
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
||||
sudo bash scripts/install_collector.sh --rollback # back to Python collection
|
||||
journalctl -u ping-collector -f
|
||||
```
|
||||
Reference in new issue
Block a user