grabowski
a40faf3102
New hours are forecast by the catch-up anyway, and a forecast issued without the discharge RID added later is still what was issued; only a corrected or deleted level makes the issued forecasts wrong.
155 lines
8.3 KiB
Markdown
155 lines
8.3 KiB
Markdown
# Rust: the service
|
|
|
|
Since October 2026 the whole Ping River Monitor service is one Rust binary,
|
|
`ping-collector`; Python is left with model training and evaluation
|
|
(`src/ml`) and the archive backfills. How it got here, and the parity rules
|
|
the rewrite followed: [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
|
|
|
| Crate | What |
|
|
| --- | --- |
|
|
| `collector` (`ping-collector`) | collection (RID, HII/ThaiWater, RID dams, Open-Meteo), the forecast run, ntfy + Matrix notifications, the website and API (`collector/src/web`) |
|
|
| `forecast` (`ping-forecast`, library) | flood-forecast inference from the models trained in Python: features, HII gap fill, gradient-boosted trees, heuristic fallback |
|
|
|
|
## ping-collector
|
|
|
|
```text
|
|
ping-collector run # the service (systemd): startup gap fill, RID at :05 (+10 min
|
|
# retries while RID is behind), HII/dams/rain once per clock
|
|
# hour; FORECAST_IN_COLLECTOR=true: forecast + ntfy/Matrix
|
|
# after each cycle with newer readings; WEB_LISTEN: website/API
|
|
ping-collector serve [--listen A] [--no-analytics]
|
|
# the website and API alone (default 127.0.0.1:8001)
|
|
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
|
# and compares the RID fetch with the database
|
|
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
|
ping-collector refetch FROM [TO] # re-fetch the RID reports of FROM..=TO (YYYY-MM-DD) even
|
|
# without holes, through the spike screening: corrects
|
|
# readings RID revised, deletes stored spikes; prints the
|
|
# catch-up-forecasts command when stored levels changed
|
|
ping-collector check-db # connect, create missing tables, row counts
|
|
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
|
|
# forecast from the database and print it
|
|
ping-collector catch-up-forecasts [--hours N] [--redo-from T]
|
|
# forecast the hours whose readings came late (RID's
|
|
# overnight batch) as a run then would have; default 336 h;
|
|
# --redo-from first deletes the model forecasts from T on
|
|
ping-collector notify --dry-run # the readings the notifier judges, and what it would send
|
|
```
|
|
|
|
Configuration: the environment, or `.env` in the working directory
|
|
(`--env-file` for another), every key with its default in
|
|
[.env.example](../.env.example). `.env` is parsed with python-dotenv's rules
|
|
(`collector/src/envfile.rs`, verified on 614 generated inputs) because the
|
|
Python tools read the same file: an unparsable line is skipped with a warning
|
|
giving only its line number, never its value. Variables already set (e.g. by
|
|
systemd's `EnvironmentFile=`) win. All timestamps are Asia/Bangkok wall-clock,
|
|
whatever the server's time zone.
|
|
|
|
### Collection
|
|
|
|
The tables, upserts and parsers are the ones the Python collectors had (the
|
|
DDL is in `collector/src/db.rs`; the Python backfill stores create the same).
|
|
RID's evening report often lags 1-2 h: every poll stores every reading the
|
|
report has, yesterday's report is re-fetched while it has holes, and the
|
|
startup gap fill repairs the last `GAP_FILL_DAYS_ON_START` days.
|
|
|
|
### Forecast and notifications
|
|
|
|
After every cycle that brought newer RID or HII readings (and at least
|
|
hourly) the forecast runs over the last 336 h, is stored in `forecast_history`
|
|
(for skill tracking) and `forecast_current`, and the notifier compares the
|
|
readings and forecasts with `notification_state`.
|
|
|
|
- **Models** are trained in Python. `src/ml/export.py` writes
|
|
`models/flood_<station>.json.gz` next to each joblib bundle (`train_all()`
|
|
does it; `python -m src.ml.export` for existing bundles); the service
|
|
reloads changed files on its next run, so a retrain needs no restart.
|
|
- **Shared constants** (thresholds, flood stages, upstream leads, notifier
|
|
settings) are compiled in from `forecast/flood_config.json`, generated by
|
|
`uv run python -m src.ml.export --config`. `tests/test_model_export.py`
|
|
fails while it differs from Python; a change needs a rebuild.
|
|
- **ntfy** topics, texts and hysteresis: [docs/NOTIFICATIONS.md](../docs/NOTIFICATIONS.md).
|
|
For the stations whose ThaiWater twin is the same sensor, a reading newer
|
|
than RID's drives the level alerts, marked as ThaiWater telemetry. "Gauge
|
|
feed stale" means no source has a reading for 3 h, except during RID's
|
|
nightly pause (21:00 to `FEED_NIGHT_UNTIL_HOUR`, 9) while the river is
|
|
quiet; RID alone lagging is reported once after `RID_STALE_ALERT_HOURS`.
|
|
- **Matrix** (`MATRIX_*`): the room gets the same transitions (per station,
|
|
outlook, feed; not ntfy's per-topic summaries), each once, tracked under
|
|
its own `matrix:` keys in `notification_state`, so a Matrix outage is
|
|
retried on Matrix only. [docs/MATRIX_QUICK_START.md](../docs/MATRIX_QUICK_START.md).
|
|
- Memory: under 100 MB resident in production, all 15 models loaded
|
|
(24-byte tree nodes); the unit allows 512 MB.
|
|
|
|
### Website and API
|
|
|
|
`collector/src/web` (axum) serves the dashboard (`STATIC_DIR`/dashboard.html,
|
|
default `src/static`, loaded at start), `/static`, the crawler files,
|
|
`/docs` + `/redoc` + `/openapi.json`, and the read API, with gzip,
|
|
Cache-Control per path, CORS (`CORS_ALLOW_ORIGINS`), Umami API tracking
|
|
(`UMAMI_*`) and stale-while-revalidate response caches with byte budgets.
|
|
Station editing, `/config` and `POST /scrape/trigger` need `ADMIN_API_KEY`
|
|
(`X-API-Key` header).
|
|
|
|
- Parameters are declared once per route (`web/params.rs`): the same list
|
|
validates the request (pydantic-style 422s) and builds the OpenAPI
|
|
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
|
|
- The response formats are the retired FastAPI app's, kept so clients did
|
|
not notice the switch: field order, `isoformat()` timestamps, Python's
|
|
`round()`, numpy's pairwise sums for the catchment means (`web/pyfmt.rs`),
|
|
`{"detail": ...}` errors.
|
|
- An old stored forecast is served marked `X-Data-Stale: true`; `/health`
|
|
counts current ThaiWater telemetry of the same gauges while RID lags.
|
|
|
|
## Develop
|
|
|
|
```bash
|
|
cd rust
|
|
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
|
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
|
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
|
CARGO_TARGET_DIR=target/msrv cargo +1.88 check --all-targets --locked # the minimum version
|
|
```
|
|
|
|
Without `TEST_DATABASE_URL` the database tests skip. CI
|
|
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
|
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
|
|
|
- The forecast's parity with the Python reference: `forecast/tests/parity.rs`
|
|
replays fixtures from the Python production path
|
|
(`forecast/testdata/gen_forecast_fixtures.py`; regenerate after a change to
|
|
`src/ml/features.py`, `predict.py`, the HII fill or the dam frame). Features
|
|
and levels must stay bit-identical, probabilities within a few ulp.
|
|
`scripts/forecast_parity.py` compares the two on a live database.
|
|
- The notifier's texts are pinned by `collector/testdata/notify_python_cases.json.gz`
|
|
(751 messages recorded from the Python notifier before it was retired).
|
|
|
|
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
|
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
|
Linux needs `pkg-config` and `libssl-dev`.
|
|
|
|
## Deploy
|
|
|
|
Needs Rust >= 1.88. Distro `cargo` packages are older (`resolver 3 is not
|
|
valid` / `edition 2024` errors), so install the toolchain once with rustup:
|
|
|
|
```bash
|
|
apt-get install -y build-essential pkg-config libssl-dev curl
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
|
|
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
|
|
cargo --version # >= 1.88
|
|
```
|
|
|
|
Then on the server, in the checkout:
|
|
|
|
```bash
|
|
git pull
|
|
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
|
sudo bash scripts/install.sh
|
|
journalctl -u ping-collector -f
|
|
```
|
|
|
|
`scripts/install.sh` installs the binary and the units, syncs the Python
|
|
training environment, runs a dry run of every source and the forecast check,
|
|
restarts the service and waits for the website (details in its header).
|