feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s

Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared
.env) ping-collector, after each cycle that brought newer RID or HII readings
and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the
rain forecast, runs every exported model (rust/forecast), stores
forecast_history and forecast_current, and sends the ntfy notifications.
The Python API then serves forecast_current as /forecast (computing in-process
only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor
notifies.

ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics,
texts, hysteresis, capacity guard and notification_state keys, checked word
for word on 751 messages generated by Python (testdata/gen_notify_cases.py).
The change: for stations whose ThaiWater twin is the same sensor, a reading
newer than RID's drives the level alerts (marked as ThaiWater telemetry), and
the feed counts as stale only when neither source has a reading for 3 h. RID
alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the
nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports.

Commands: `forecast [--json] [--save] [--now] [--rain-file]`,
`notify --dry-run`. scripts/forecast_parity.py compares Python and the
collector on the live database (same moment, same rain); locally 48/48 rows
identical. install_collector.sh: models export + that check on every install,
--cutover-forecast (refuses on a failed check) and --rollback-forecast.

Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140).
Forecast values are written to NUMERIC as shortest round-trip text, so they
round as psycopg2's do. DB test schemas get a counter (parallel tests collided
on Windows' coarse clock).
This commit is contained in:
grabowski committed 2026-10-18 14:43:39 +07:00
1 parent 5c4f464383
commit 6ae67dec63
27 files changed
+2656 -118

No files matched your search

+60 -6
View File
@@ -6,6 +6,7 @@ costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md
| Crate | Replaces | Status |
| --- | --- | --- |
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
| `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: built, parity-verified, cutover pending |
## ping-collector
@@ -17,11 +18,15 @@ the database advances.
```text
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
# while RID is behind), HII/dams/rain once per clock hour
# while RID is behind), HII/dams/rain once per clock hour;
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
# and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
ping-collector check-db # connect, create missing tables, row counts
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
# forecast from the database and print it
ping-collector notify --dry-run # the readings the notifier judges, and what it would send
```
Configuration is the Python app's `.env` (read from the working directory, or
@@ -35,8 +40,11 @@ only its line number, never its value. Variables already set (e.g. by systemd's
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
`LOG_LEVEL` / `RUST_LOG`. All timestamps are Asia/Bangkok wall-clock,
whatever the server's time zone.
`LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false),
`MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX`
/ `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18,
0 = off). All timestamps are Asia/Bangkok wall-clock, whatever the server's
time zone.
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
stay in Python.
@@ -56,6 +64,47 @@ stay in Python.
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
forecasts from a database filled only by `ping-collector`.
## Forecast and notifications (phase 2)
With `FORECAST_IN_COLLECTOR=true` in the shared `.env`, `ping-collector` runs
the flood forecast after every cycle that brought newer RID or HII readings
(and at least hourly), stores it in `forecast_history` and `forecast_current`
(the Python API serves the latter as `/forecast`, falling back to computing
itself when it is older than `FORECAST_MAX_AGE_SECONDS`), and sends the ntfy
notifications. The Python app then neither forecasts nor notifies.
- **Models** stay trained in Python. `src/ml/export.py` writes
`models/flood_<station>.json.gz` next to each joblib bundle (`train_all()`
does it, `retrain.sh` promotes it, `python -m src.ml.export` exports
existing bundles); the collector reloads changed files on its next run.
- **Shared constants** (thresholds, flood stages, upstream leads, notify
settings) are compiled in from `forecast/flood_config.json`, generated by
`uv run python -m src.ml.export --config`. `tests/test_model_export.py`
fails while it differs from Python, and a change needs a collector rebuild.
- **Notifications**: same topics, texts, hysteresis and `notification_state`
keys as `src/notify.py`, so switching sends nothing twice. One change: for
the stations whose ThaiWater twin is the same sensor, a reading newer than
RID's (RID's report lags 2-10 h at night) drives the level alerts, marked
"ThaiWater telemetry; the RID report is not in yet". "Gauge feed stale" now
means no source has a reading for 3 h; RID alone lagging is reported once
after `RID_STALE_ALERT_HOURS`.
- Memory: 70 MB peak with all 15 models loaded (24-byte tree nodes).
Verified:
- `forecast/tests/parity.rs` replays fixtures from the Python production path
(`forecast/testdata/gen_forecast_fixtures.py`: the 2024 flood rising and at
its peak with RID lagging behind HII, a quiet window with dam data, a
stale-feed window, moved thresholds). HII fill, every feature and every
level are bit-identical; with all 15 local models 377 of 384 probabilities
are bit-identical and the rest within 2 ulp (`exp`). CI checks the P.1 model.
- `scripts/forecast_parity.py` does the same on the live database: Python and
`ping-collector forecast --json` at the same moment with the same rain
series. The install script runs it before `--cutover-forecast`.
- `collector/testdata/gen_notify_cases.py`: 652 hourly steps through
`src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy
outages); the Rust notifier sends the same 751 messages word for word.
### Develop
```bash
@@ -90,8 +139,13 @@ the same commit):
```bash
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
sudo bash scripts/install_collector.sh --rollback # back to Python collection
sudo bash scripts/install_collector.sh # install + checks, nothing switched
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
sudo bash scripts/install_collector.sh --rollback # Python does everything again
journalctl -u ping-collector -f
```
Every install exports the models and runs `scripts/forecast_parity.py`;
`--cutover-forecast` refuses to switch if it fails (`FORCE=1` overrides).