grabowski
c8e89840b5
dotenvy aborted on the production .env (a quote in an unquoted value makes it
read to the end of the file) and echoed the offending line, password
included. envfile.rs ports python-dotenv 1.1.1 (parser + ${VAR} interpolation,
no override): unparsable statements are skipped with a warning that names only
the line number. Verified against python-dotenv on 614 generated inputs
(testdata/). The .env is loaded before the tokio runtime starts.
install_collector.sh now runs the dry run through systemd-run with the unit's
User, WorkingDirectory and EnvironmentFile, i.e. exactly what the service sees.
98 lines
4.4 KiB
Markdown
98 lines
4.4 KiB
Markdown
# Rust services
|
|
|
|
Rewrite of the Ping River Monitor in Rust, one service at a time. The plan,
|
|
costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
|
|
|
| Crate | Replaces | Status |
|
|
| --- | --- | --- |
|
|
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: ready to deploy |
|
|
|
|
## ping-collector
|
|
|
|
Collects into the same PostgreSQL tables as the Python app, so the two can be
|
|
swapped (and swapped back) with no data migration. The Python web app keeps
|
|
serving the API, forecasting and sending ntfy alerts with
|
|
`COLLECTION_ENABLED=false`, and re-forecasts whenever the newest RID reading in
|
|
the database advances.
|
|
|
|
```text
|
|
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
|
|
# while RID is behind), HII/dams/rain once per clock hour
|
|
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
|
# and compares the RID fetch with the database
|
|
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
|
ping-collector check-db # connect, create missing tables, row counts
|
|
```
|
|
|
|
Configuration is the Python app's `.env` (read from the working directory, or
|
|
`--env-file`) with python-dotenv's rules, verified on 614 generated inputs
|
|
(`collector/testdata/`): an unparsable line is skipped with a warning giving
|
|
only its line number, never its value. Variables already set (e.g. by systemd's
|
|
`EnvironmentFile=`) win. Used keys:
|
|
`POSTGRES_CONNECTION_STRING` (or `POSTGRES_HOST/PORT/DB/USER/PASSWORD`),
|
|
`RID_POLL_OFFSET_MINUTES` (5), `RID_RETRY_MINUTES` (10),
|
|
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
|
|
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
|
|
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
|
|
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
|
|
`LOG_LEVEL` / `RUST_LOG`. All timestamps are Asia/Bangkok wall-clock,
|
|
whatever the server's time zone.
|
|
|
|
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
|
|
stay in Python.
|
|
|
|
### Verified
|
|
|
|
- 20 tests: parsers, gap/schedule logic, and database tests against a real
|
|
PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC
|
|
rounding, dam metadata preservation).
|
|
- Live parity: the Python collectors and `ping-collector` run back-to-back
|
|
against the real upstream APIs into two empty databases produced identical
|
|
rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows,
|
|
35 dams x 2 days, 168 rain hours). Rerun with
|
|
`scripts/collector_parity.py` after changing a parser, table or upsert on
|
|
either side.
|
|
- The Python API with `COLLECTION_ENABLED=false` served `/health`,
|
|
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
|
|
forecasts from a database filled only by `ping-collector`.
|
|
|
|
### Develop
|
|
|
|
```bash
|
|
cd rust
|
|
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
|
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
|
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
|
```
|
|
|
|
Without `TEST_DATABASE_URL` the database tests skip. CI
|
|
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
|
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
|
|
|
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
|
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
|
Linux needs `pkg-config` and `libssl-dev`.
|
|
|
|
### Deploy
|
|
|
|
Needs Rust >= 1.88. Distro `cargo` packages are older (`resolver 3 is not
|
|
valid` / `edition 2024` errors), so install the toolchain once with rustup:
|
|
|
|
```bash
|
|
apt-get install -y build-essential pkg-config libssl-dev curl
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
|
|
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
|
|
cargo --version # >= 1.88
|
|
```
|
|
|
|
Then on the server, after `git pull` (the Python side of the switch ships in
|
|
the same commit):
|
|
|
|
```bash
|
|
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
|
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
|
|
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
|
sudo bash scripts/install_collector.sh --rollback # back to Python collection
|
|
journalctl -u ping-collector -f
|
|
```
|