grabowski
0e30ddbcb6
ping-status sent "gauge feed stale" around midnight and "feed recovered" around 07:00 on each of the last three nights (ntfy and, since phase 3, the Matrix room). RID publishes no hourly report overnight: the 21:00 reading is the last until the 22:00..06:00 hours arrive together (06:35, 06:55, 07:22 on 2026-10-17/18/19). The ThaiWater overlay was meant to bridge that, but ThaiWater's copies of the RID gauges come from the same feed with the same delay (P.1's mirror still has no value for 22:00-00:00 this morning), so it does not. Between 21:00 and FEED_NIGHT_UNTIL_HOUR (default 9, 0 = off) the feed of a quiet river is no longer reported stale; one still without new readings at 09:00 is. While a station is at warning/danger or the P.1 outlook is on, the 3 h rule holds all night. A feed already stale stays stale through the night (no false "recovered"), and a real recovery is announced at any hour. The replay of the Python notifier runs with the window off. Docs no longer claim that ThaiWater keeps the feed current at night.
147 lines
7.7 KiB
Markdown
147 lines
7.7 KiB
Markdown
# Rust: the service
|
|
|
|
Since October 2026 the whole Ping River Monitor service is one Rust binary,
|
|
`ping-collector`; Python is left with model training and evaluation
|
|
(`src/ml`) and the archive backfills. How it got here, and the parity rules
|
|
the rewrite followed: [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
|
|
|
| Crate | What |
|
|
| --- | --- |
|
|
| `collector` (`ping-collector`) | collection (RID, HII/ThaiWater, RID dams, Open-Meteo), the forecast run, ntfy + Matrix notifications, the website and API (`collector/src/web`) |
|
|
| `forecast` (`ping-forecast`, library) | flood-forecast inference from the models trained in Python: features, HII gap fill, gradient-boosted trees, heuristic fallback |
|
|
|
|
## ping-collector
|
|
|
|
```text
|
|
ping-collector run # the service (systemd): startup gap fill, RID at :05 (+10 min
|
|
# retries while RID is behind), HII/dams/rain once per clock
|
|
# hour; FORECAST_IN_COLLECTOR=true: forecast + ntfy/Matrix
|
|
# after each cycle with newer readings; WEB_LISTEN: website/API
|
|
ping-collector serve [--listen A] [--no-analytics]
|
|
# the website and API alone (default 127.0.0.1:8001)
|
|
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
|
# and compares the RID fetch with the database
|
|
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
|
ping-collector check-db # connect, create missing tables, row counts
|
|
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
|
|
# forecast from the database and print it
|
|
ping-collector notify --dry-run # the readings the notifier judges, and what it would send
|
|
```
|
|
|
|
Configuration: the environment, or `.env` in the working directory
|
|
(`--env-file` for another), every key with its default in
|
|
[.env.example](../.env.example). `.env` is parsed with python-dotenv's rules
|
|
(`collector/src/envfile.rs`, verified on 614 generated inputs) because the
|
|
Python tools read the same file: an unparsable line is skipped with a warning
|
|
giving only its line number, never its value. Variables already set (e.g. by
|
|
systemd's `EnvironmentFile=`) win. All timestamps are Asia/Bangkok wall-clock,
|
|
whatever the server's time zone.
|
|
|
|
### Collection
|
|
|
|
The tables, upserts and parsers are the ones the Python collectors had (the
|
|
DDL is in `collector/src/db.rs`; the Python backfill stores create the same).
|
|
RID's evening report often lags 1-2 h: every poll stores every reading the
|
|
report has, yesterday's report is re-fetched while it has holes, and the
|
|
startup gap fill repairs the last `GAP_FILL_DAYS_ON_START` days.
|
|
|
|
### Forecast and notifications
|
|
|
|
After every cycle that brought newer RID or HII readings (and at least
|
|
hourly) the forecast runs over the last 336 h, is stored in `forecast_history`
|
|
(for skill tracking) and `forecast_current`, and the notifier compares the
|
|
readings and forecasts with `notification_state`.
|
|
|
|
- **Models** are trained in Python. `src/ml/export.py` writes
|
|
`models/flood_<station>.json.gz` next to each joblib bundle (`train_all()`
|
|
does it; `python -m src.ml.export` for existing bundles); the service
|
|
reloads changed files on its next run, so a retrain needs no restart.
|
|
- **Shared constants** (thresholds, flood stages, upstream leads, notifier
|
|
settings) are compiled in from `forecast/flood_config.json`, generated by
|
|
`uv run python -m src.ml.export --config`. `tests/test_model_export.py`
|
|
fails while it differs from Python; a change needs a rebuild.
|
|
- **ntfy** topics, texts and hysteresis: [docs/NOTIFICATIONS.md](../docs/NOTIFICATIONS.md).
|
|
For the stations whose ThaiWater twin is the same sensor, a reading newer
|
|
than RID's drives the level alerts, marked as ThaiWater telemetry. "Gauge
|
|
feed stale" means no source has a reading for 3 h, except during RID's
|
|
nightly pause (21:00 to `FEED_NIGHT_UNTIL_HOUR`, 9) while the river is
|
|
quiet; RID alone lagging is reported once after `RID_STALE_ALERT_HOURS`.
|
|
- **Matrix** (`MATRIX_*`): the room gets the same transitions (per station,
|
|
outlook, feed; not ntfy's per-topic summaries), each once, tracked under
|
|
its own `matrix:` keys in `notification_state`, so a Matrix outage is
|
|
retried on Matrix only. [docs/MATRIX_QUICK_START.md](../docs/MATRIX_QUICK_START.md).
|
|
- Memory: under 100 MB resident in production, all 15 models loaded
|
|
(24-byte tree nodes); the unit allows 512 MB.
|
|
|
|
### Website and API
|
|
|
|
`collector/src/web` (axum) serves the dashboard (`STATIC_DIR`/dashboard.html,
|
|
default `src/static`, loaded at start), `/static`, the crawler files,
|
|
`/docs` + `/redoc` + `/openapi.json`, and the read API, with gzip,
|
|
Cache-Control per path, CORS (`CORS_ALLOW_ORIGINS`), Umami API tracking
|
|
(`UMAMI_*`) and stale-while-revalidate response caches with byte budgets.
|
|
Station editing, `/config` and `POST /scrape/trigger` need `ADMIN_API_KEY`
|
|
(`X-API-Key` header).
|
|
|
|
- Parameters are declared once per route (`web/params.rs`): the same list
|
|
validates the request (pydantic-style 422s) and builds the OpenAPI
|
|
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
|
|
- The response formats are the retired FastAPI app's, kept so clients did
|
|
not notice the switch: field order, `isoformat()` timestamps, Python's
|
|
`round()`, numpy's pairwise sums for the catchment means (`web/pyfmt.rs`),
|
|
`{"detail": ...}` errors.
|
|
- An old stored forecast is served marked `X-Data-Stale: true`; `/health`
|
|
counts current ThaiWater telemetry of the same gauges while RID lags.
|
|
|
|
## Develop
|
|
|
|
```bash
|
|
cd rust
|
|
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
|
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
|
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
|
CARGO_TARGET_DIR=target/msrv cargo +1.88 check --all-targets --locked # the minimum version
|
|
```
|
|
|
|
Without `TEST_DATABASE_URL` the database tests skip. CI
|
|
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
|
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
|
|
|
- The forecast's parity with the Python reference: `forecast/tests/parity.rs`
|
|
replays fixtures from the Python production path
|
|
(`forecast/testdata/gen_forecast_fixtures.py`; regenerate after a change to
|
|
`src/ml/features.py`, `predict.py`, the HII fill or the dam frame). Features
|
|
and levels must stay bit-identical, probabilities within a few ulp.
|
|
`scripts/forecast_parity.py` compares the two on a live database.
|
|
- The notifier's texts are pinned by `collector/testdata/notify_python_cases.json.gz`
|
|
(751 messages recorded from the Python notifier before it was retired).
|
|
|
|
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
|
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
|
Linux needs `pkg-config` and `libssl-dev`.
|
|
|
|
## Deploy
|
|
|
|
Needs Rust >= 1.88. Distro `cargo` packages are older (`resolver 3 is not
|
|
valid` / `edition 2024` errors), so install the toolchain once with rustup:
|
|
|
|
```bash
|
|
apt-get install -y build-essential pkg-config libssl-dev curl
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
|
|
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
|
|
cargo --version # >= 1.88
|
|
```
|
|
|
|
Then on the server, in the checkout:
|
|
|
|
```bash
|
|
git pull
|
|
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
|
sudo bash scripts/install.sh
|
|
journalctl -u ping-collector -f
|
|
```
|
|
|
|
`scripts/install.sh` installs the binary and the units, syncs the Python
|
|
training environment, runs a dry run of every source and the forecast check,
|
|
restarts the service and waits for the website (details in its header).
|