190 lines
10 KiB
Markdown
190 lines
10 KiB
Markdown
# Rust services
|
|
|
|
Rewrite of the Ping River Monitor in Rust, one service at a time. The plan,
|
|
costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md).
|
|
|
|
| Crate | Replaces | Status |
|
|
| --- | --- | --- |
|
|
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
|
|
| `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: in production since 2026-10-18 |
|
|
| `collector`: `src/web` | the FastAPI app (`src/web_api.py`): dashboard, read API, `/docs` | phase 3: in production since 2026-10-18 |
|
|
|
|
## ping-collector
|
|
|
|
Collects into the same PostgreSQL tables as the Python app, so the two can be
|
|
swapped (and swapped back) with no data migration. The Python web app keeps
|
|
serving the API, forecasting and sending ntfy alerts with
|
|
`COLLECTION_ENABLED=false`, and re-forecasts whenever the newest RID reading in
|
|
the database advances.
|
|
|
|
```text
|
|
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
|
|
# while RID is behind), HII/dams/rain once per clock hour;
|
|
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy/Matrix;
|
|
# with WEB_LISTEN the website and API
|
|
ping-collector serve [--listen A] [--no-analytics]
|
|
# the website and API only (default 127.0.0.1:8001)
|
|
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
|
# and compares the RID fetch with the database
|
|
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
|
ping-collector check-db # connect, create missing tables, row counts
|
|
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
|
|
# forecast from the database and print it
|
|
ping-collector notify --dry-run # the readings the notifier judges, and what it would send
|
|
```
|
|
|
|
Configuration is the Python app's `.env` (read from the working directory, or
|
|
`--env-file`) with python-dotenv's rules, verified on 614 generated inputs
|
|
(`collector/testdata/`): an unparsable line is skipped with a warning giving
|
|
only its line number, never its value. Variables already set (e.g. by systemd's
|
|
`EnvironmentFile=`) win. Used keys:
|
|
`POSTGRES_CONNECTION_STRING` (or `POSTGRES_HOST/PORT/DB/USER/PASSWORD`),
|
|
`RID_POLL_OFFSET_MINUTES` (5), `RID_RETRY_MINUTES` (10),
|
|
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
|
|
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
|
|
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
|
|
`SCRAPING_STATUS_FILE` (status JSON for `/scraping/status` when the website runs elsewhere),
|
|
`LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false),
|
|
`MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX`
|
|
/ `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18,
|
|
0 = off). All timestamps are Asia/Bangkok wall-clock, whatever the server's
|
|
time zone.
|
|
|
|
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
|
|
stay in Python.
|
|
|
|
### Verified
|
|
|
|
- 20 tests: parsers, gap/schedule logic, and database tests against a real
|
|
PostgreSQL (late reports, midnight recovery, upsert revisions, NUMERIC
|
|
rounding, dam metadata preservation).
|
|
- Live parity: the Python collectors and `ping-collector` run back-to-back
|
|
against the real upstream APIs into two empty databases produced identical
|
|
rows in all 9 tables (16 stations, 1,104 RID readings, 103 + 76 HII rows,
|
|
35 dams x 2 days, 168 rain hours). Rerun with
|
|
`scripts/collector_parity.py` after changing a parser, table or upsert on
|
|
either side.
|
|
- The Python API with `COLLECTION_ENABLED=false` served `/health`,
|
|
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
|
|
forecasts from a database filled only by `ping-collector`.
|
|
|
|
## Forecast and notifications (phase 2)
|
|
|
|
With `FORECAST_IN_COLLECTOR=true` in the shared `.env`, `ping-collector` runs
|
|
the flood forecast after every cycle that brought newer RID or HII readings
|
|
(and at least hourly), stores it in `forecast_history` and `forecast_current`
|
|
(the Python API serves the latter as `/forecast`, falling back to computing
|
|
itself when it is older than `FORECAST_MAX_AGE_SECONDS`), and sends the ntfy
|
|
notifications. The Python app then neither forecasts nor notifies.
|
|
|
|
- **Models** stay trained in Python. `src/ml/export.py` writes
|
|
`models/flood_<station>.json.gz` next to each joblib bundle (`train_all()`
|
|
does it, `retrain.sh` promotes it, `python -m src.ml.export` exports
|
|
existing bundles); the collector reloads changed files on its next run.
|
|
- **Shared constants** (thresholds, flood stages, upstream leads, notify
|
|
settings) are compiled in from `forecast/flood_config.json`, generated by
|
|
`uv run python -m src.ml.export --config`. `tests/test_model_export.py`
|
|
fails while it differs from Python, and a change needs a collector rebuild.
|
|
- **Notifications**: same topics, texts, hysteresis and `notification_state`
|
|
keys as `src/notify.py`, so switching sends nothing twice. One change: for
|
|
the stations whose ThaiWater twin is the same sensor, a reading newer than
|
|
RID's (RID's report lags 2-10 h at night) drives the level alerts, marked
|
|
"ThaiWater telemetry; the RID report is not in yet". "Gauge feed stale" now
|
|
means no source has a reading for 3 h; RID alone lagging is reported once
|
|
after `RID_STALE_ALERT_HOURS`.
|
|
- Memory: 70 MB peak with all 15 models loaded (24-byte tree nodes).
|
|
|
|
Verified:
|
|
|
|
- `forecast/tests/parity.rs` replays fixtures from the Python production path
|
|
(`forecast/testdata/gen_forecast_fixtures.py`: the 2024 flood rising and at
|
|
its peak with RID lagging behind HII, a quiet window with dam data, a
|
|
stale-feed window, moved thresholds). HII fill, every feature and every
|
|
level are bit-identical; with all 15 local models 377 of 384 probabilities
|
|
are bit-identical and the rest within 2 ulp (`exp`). CI checks the P.1 model.
|
|
- `scripts/forecast_parity.py` does the same on the live database: Python and
|
|
`ping-collector forecast --json` at the same moment with the same rain
|
|
series. The install script runs it before `--cutover-forecast`.
|
|
- `collector/testdata/gen_notify_cases.py`: 652 hourly steps through
|
|
`src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy
|
|
outages); the Rust notifier sends the same 751 messages word for word.
|
|
|
|
## Website and API (phase 3)
|
|
|
|
`src/web` serves what the FastAPI app served: the dashboard
|
|
(`STATIC_DIR`/dashboard.html, default `src/static`, loaded at start), the
|
|
crawler files, `/static`, `/docs` + `/redoc` + `/openapi.json`, and every API
|
|
route with the same parameters, JSON (field order included), status codes,
|
|
`{"detail": ...}` errors, Cache-Control, gzip, CORS (`CORS_ALLOW_ORIGINS`),
|
|
Umami API tracking (`UMAMI_*`) and response caches (`*_CACHE_TTL_SECONDS`).
|
|
Admin routes need `ADMIN_API_KEY`. With `WEB_LISTEN=0.0.0.0:8000` it runs
|
|
inside `ping-collector run` (one process collects, forecasts, notifies and
|
|
serves; `/scraping/status` and `POST /scrape/trigger` then talk to the
|
|
collection loop directly); `serve` runs it alone.
|
|
|
|
- Parameters are declared once per route (`web/params.rs`): the same list
|
|
validates the request (pydantic-style 422s) and builds the OpenAPI
|
|
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
|
|
The component schemas are FastAPI's (`web/openapi_components.json`).
|
|
- Python's output conventions are kept where they show: `isoformat()`
|
|
timestamps, `round()`, numpy's pairwise sums for the catchment means
|
|
(`web/pyfmt.rs`), the skill metrics' summation order.
|
|
- `scripts/api_parity.py --python URL --rust URL` compares the two live on
|
|
one database; `install_collector.sh --check-api` / `--cutover-api` run it
|
|
on the server against `ping-collector serve` on port 8001.
|
|
|
|
Matrix (`MATRIX_HOMESERVER`, `MATRIX_ACCESS_TOKEN`, `MATRIX_ROOM_ID`): the
|
|
notifier sends the room the same transitions as ntfy (per station, outlook,
|
|
feed; not ntfy's per-topic summaries), each once, with its own `matrix:` keys
|
|
in `notification_state` (seeded from the ntfy ones on first use, so nothing
|
|
already announced is repeated). It replaces the `run.py --alert-check` cron
|
|
job, which the install script comments out.
|
|
|
|
### Develop
|
|
|
|
```bash
|
|
cd rust
|
|
cargo fmt --all && cargo clippy --all-targets -- -D warnings
|
|
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
|
TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping cargo test
|
|
```
|
|
|
|
Without `TEST_DATABASE_URL` the database tests skip. CI
|
|
(`.gitea/workflows/rust.yml`) runs them against a PostgreSQL service and
|
|
uploads the Linux binary as the `ping-collector-linux-x86_64` artifact.
|
|
|
|
HTTP uses native-tls (OpenSSL on Linux), not rustls: app.rid.go.th (dams)
|
|
only offers TLS 1.2 CBC cipher suites, which rustls refuses. Building on
|
|
Linux needs `pkg-config` and `libssl-dev`.
|
|
|
|
### Deploy
|
|
|
|
Needs Rust >= 1.88. Distro `cargo` packages are older (`resolver 3 is not
|
|
valid` / `edition 2024` errors), so install the toolchain once with rustup:
|
|
|
|
```bash
|
|
apt-get install -y build-essential pkg-config libssl-dev curl
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
|
|
. "$HOME/.cargo/env" # puts ~/.cargo/bin before /usr/bin/cargo
|
|
cargo --version # >= 1.88
|
|
```
|
|
|
|
Then on the server, after `git pull` (the Python side of the switch ships in
|
|
the same commit):
|
|
|
|
```bash
|
|
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
|
sudo bash scripts/install_collector.sh # install + checks, nothing switched
|
|
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
|
sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies
|
|
sudo bash scripts/install_collector.sh --check-api # compare the Rust API with FastAPI
|
|
sudo bash scripts/install_collector.sh --cutover-api # Rust serves the website on :8000
|
|
sudo bash scripts/install_collector.sh --rollback-api # FastAPI serves the website again
|
|
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
|
|
sudo bash scripts/install_collector.sh --rollback # Python does everything again
|
|
journalctl -u ping-collector -f
|
|
```
|
|
|
|
Every install exports the models and runs `scripts/forecast_parity.py`;
|
|
`--cutover-forecast` refuses to switch if it fails (`FORCE=1` overrides).
|