grabowski
180d95c6bd
The P.1 chart's "24 h peak, as issued" line had a gap every night and a dip at 21:00. Two causes: - A run forecasts from the newest reading only. RID publishes 22:00..06:00 in one batch in the morning (and now and then one hour late), so those hours were never forecast from. After each run the collector now forecasts the hours of the last 48 h (two weeks on the first run after a start) that have a RID reading but no model forecast, each as a run at that hour would have: gauges, HII fill and dam state up to that hour. Rain is fetched once, so a late hour's rain_fc24 partly holds rain that has since fallen (as in training). Only missing (hour, station) pairs are written. `ping-collector catch-up-forecasts --hours N` goes further back. - Overnight each hourly run re-issues the 21:00 reading, after 6 h as the heuristic, which overwrote the 21:00 model row. A heuristic row no longer replaces a model row in forecast_history. The dashboard draws only model rows on the issued line. Checked on a local copy of 18 days of RID data: the catch-up of an hour matches `ping-collector forecast --now <hour>`; 288 hours took 20 s (debug build). Tests for both database rules.
150 lines
7.9 KiB
Markdown
150 lines
7.9 KiB
Markdown
# Rust: the service
|
|
|
|
Since October 2026 the whole Ping River Monitor service is one Rust binary,
|
|
`ping-collector`; Python is left with model training and evaluation
|
|
(`src/ml`) and the archive backfills. How it got here, and the parity rules
|
|
the rewrite followed: [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
|
|
|
| Crate | What |
|
|
| --- | --- |
|
|
| `collector` (`ping-collector`) | collection (RID, HII/ThaiWater, RID dams, Open-Meteo), the forecast run, ntfy + Matrix notifications, the website and API (`collector/src/web`) |
|
|
| `forecast` (`ping-forecast`, library) | flood-forecast inference from the models trained in Python: features, HII gap fill, gradient-boosted trees, heuristic fallback |
|
|
|
|
## ping-collector
|
|
|
|
```text
|
|
ping-collector run # the service (systemd): startup gap fill, RID at :05 (+10 min
|
|
# retries while RID is behind), HII/dams/rain once per clock
|
|
# hour; FORECAST_IN_COLLECTOR=true: forecast + ntfy/Matrix
|
|
# after each cycle with newer readings; WEB_LISTEN: website/API
|
|
ping-collector serve [--listen A] [--no-analytics]
|
|
# the website and API alone (default 127.0.0.1:8001)
|
|
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
|
# and compares the RID fetch with the database
|
|
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
|
ping-collector check-db # connect, create missing tables, row counts
|
|
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
|
|
# forecast from the database and print it
|
|
ping-collector catch-up-forecasts [--hours N]
|
|
# forecast the hours whose readings came late (RID's
|
|
# overnight batch) as a run then would have; default 336 h
|
|
ping-collector notify --dry-run # the readings the notifier judges, and what it would send
|
|
```
|
|
|
|
Configuration: the environment, or `.env` in the working directory
|
|
(`--env-file` for another), every key with its default in
|
|
[.env.example](../.env.example). `.env` is parsed with python-dotenv's rules
|
|
(`collector/src/envfile.rs`, verified on 614 generated inputs) because the
|
|
Python tools read the same file: an unparsable line is skipped with a warning
|
|
giving only its line number, never its value. Variables already set (e.g. by
|
|
systemd's `EnvironmentFile=`) win. All timestamps are Asia/Bangkok wall-clock,
|
|
whatever the server's time zone.
|
|
|
|
### Collection
|
|
|
|
The tables, upserts and parsers are the ones the Python collectors had (the
|
|
DDL is in `collector/src/db.rs`; the Python backfill stores create the same).
|
|
RID's evening report often lags 1-2 h: every poll stores every reading the
|
|
report has, yesterday's report is re-fetched while it has holes, and the
|
|
startup gap fill repairs the last `GAP_FILL_DAYS_ON_START` days.
|
|
|
|
### Forecast and notifications
|
|
|
|
After every cycle that brought newer RID or HII readings (and at least
|
|
hourly) the forecast runs over the last 336 h, is stored in `forecast_history`
|
|
(for skill tracking) and `forecast_current`, and the notifier compares the
|
|
readings and forecasts with `notification_state`.
|
|
|
|
- **Models** are trained in Python. `src/ml/export.py` writes
|
|
`models/flood_<station>.json.gz` next to each joblib bundle (`train_all()`
|
|
does it; `python -m src.ml.export` for existing bundles); the service
|
|
reloads changed files on its next run, so a retrain needs no restart.
|
|
- **Shared constants** (thresholds, flood stages, upstream leads, notifier
|
|
settings) are compiled in from `forecast/flood_config.json`, generated by
|
|
`uv run python -m src.ml.export --config`. `tests/test_model_export.py`
|
|
fails while it differs from Python; a change needs a rebuild.
|
|
- **ntfy** topics, texts and hysteresis: [docs/NOTIFICATIONS.md](../docs/NOTIFICATIONS.md).
|
|
For the stations whose ThaiWater twin is the same sensor, a reading newer
|
|
than RID's drives the level alerts, marked as ThaiWater telemetry. "Gauge
|
|
feed stale" means no source has a reading for 3 h, except during RID's
|
|
nightly pause (21:00 to `FEED_NIGHT_UNTIL_HOUR`, 9) while the river is
|
|
quiet; RID alone lagging is reported once after `RID_STALE_ALERT_HOURS`.
|
|
- **Matrix** (`MATRIX_*`): the room gets the same transitions (per station,
|
|
outlook, feed; not ntfy's per-topic summaries), each once, tracked under
|
|
its own `matrix:` keys in `notification_state`, so a Matrix outage is
|
|
retried on Matrix only. [docs/MATRIX_QUICK_START.md](../docs/MATRIX_QUICK_START.md).
|
|
- Memory: under 100 MB resident in production, all 15 models loaded
|
|
(24-byte tree nodes); the unit allows 512 MB.
|
|
|
|
### Website and API
|
|
|
|
`collector/src/web` (axum) serves the dashboard (`STATIC_DIR`/dashboard.html,
|
|
default `src/static`, loaded at start), `/static`, the crawler files,
|
|
`/docs` + `/redoc` + `/openapi.json`, and the read API, with gzip,
|
|
Cache-Control per path, CORS (`CORS_ALLOW_ORIGINS`), Umami API tracking
|
|
(`UMAMI_*`) and stale-while-revalidate response caches with byte budgets.
|
|
Station editing, `/config` and `POST /scrape/trigger` need `ADMIN_API_KEY`
|
|
(`X-API-Key` header).
|
|
|
|
- Parameters are declared once per route (`web/params.rs`): the same list
|
|
validates the request (pydantic-style 422s) and builds the OpenAPI
|
|
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
|
|
- The response formats are the retired FastAPI app's, kept so clients did
|
|
not notice the switch: field order, `isoformat()` timestamps, Python's
|
|
`round()`, numpy's pairwise sums for the catchment means (`web/pyfmt.rs`),
|
|
`{"detail": ...}` errors.
|
|
- An old stored forecast is served marked `X-Data-Stale: true`; `/health`
|
|
counts current ThaiWater telemetry of the same gauges while RID lags.
|
|
|
|
## Develop
|
|
|
|
```bash
|
|
cd rust
|
|
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
|
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
|
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
|
CARGO_TARGET_DIR=target/msrv cargo +1.88 check --all-targets --locked # the minimum version
|
|
```
|
|
|
|
Without `TEST_DATABASE_URL` the database tests skip. CI
|
|
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
|
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
|
|
|
- The forecast's parity with the Python reference: `forecast/tests/parity.rs`
|
|
replays fixtures from the Python production path
|
|
(`forecast/testdata/gen_forecast_fixtures.py`; regenerate after a change to
|
|
`src/ml/features.py`, `predict.py`, the HII fill or the dam frame). Features
|
|
and levels must stay bit-identical, probabilities within a few ulp.
|
|
`scripts/forecast_parity.py` compares the two on a live database.
|
|
- The notifier's texts are pinned by `collector/testdata/notify_python_cases.json.gz`
|
|
(751 messages recorded from the Python notifier before it was retired).
|
|
|
|
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
|
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
|
Linux needs `pkg-config` and `libssl-dev`.
|
|
|
|
## Deploy
|
|
|
|
Needs Rust >= 1.88. Distro `cargo` packages are older (`resolver 3 is not
|
|
valid` / `edition 2024` errors), so install the toolchain once with rustup:
|
|
|
|
```bash
|
|
apt-get install -y build-essential pkg-config libssl-dev curl
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
|
|
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
|
|
cargo --version # >= 1.88
|
|
```
|
|
|
|
Then on the server, in the checkout:
|
|
|
|
```bash
|
|
git pull
|
|
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
|
sudo bash scripts/install.sh
|
|
journalctl -u ping-collector -f
|
|
```
|
|
|
|
`scripts/install.sh` installs the binary and the units, syncs the Python
|
|
training environment, runs a dry run of every source and the forecast check,
|
|
restarts the service and waits for the website (details in its header).
|