feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock).
This commit is contained in:
1 parent
5c4f464383
commit
6ae67dec63
27 files changed
+2656
-118
No files matched your search
+60
-6
@@ -6,6 +6,7 @@ costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md
|
||||
| Crate | Replaces | Status |
|
||||
| --- | --- | --- |
|
||||
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
|
||||
| `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: built, parity-verified, cutover pending |
|
||||
|
||||
## ping-collector
|
||||
|
||||
@@ -17,11 +18,15 @@ the database advances.
|
||||
|
||||
```text
|
||||
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
|
||||
# while RID is behind), HII/dams/rain once per clock hour
|
||||
# while RID is behind), HII/dams/rain once per clock hour;
|
||||
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy
|
||||
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
||||
# and compares the RID fetch with the database
|
||||
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
||||
ping-collector check-db # connect, create missing tables, row counts
|
||||
ping-collector forecast [--json] [--save] [--now T] [--rain-file F]
|
||||
# forecast from the database and print it
|
||||
ping-collector notify --dry-run # the readings the notifier judges, and what it would send
|
||||
```
|
||||
|
||||
Configuration is the Python app's `.env` (read from the working directory, or
|
||||
@@ -35,8 +40,11 @@ only its line number, never its value. Variables already set (e.g. by systemd's
|
||||
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
|
||||
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
|
||||
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
|
||||
`LOG_LEVEL` / `RUST_LOG`. All timestamps are Asia/Bangkok wall-clock,
|
||||
whatever the server's time zone.
|
||||
`LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false),
|
||||
`MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX`
|
||||
/ `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18,
|
||||
0 = off). All timestamps are Asia/Bangkok wall-clock, whatever the server's
|
||||
time zone.
|
||||
|
||||
Not ported: the one-off archive backfills (`scripts/backfill_*.py`), which
|
||||
stay in Python.
|
||||
@@ -56,6 +64,47 @@ stay in Python.
|
||||
`/measurements/latest`, `/api/stats`, `/scraping/status` and model
|
||||
forecasts from a database filled only by `ping-collector`.
|
||||
|
||||
## Forecast and notifications (phase 2)
|
||||
|
||||
With `FORECAST_IN_COLLECTOR=true` in the shared `.env`, `ping-collector` runs
|
||||
the flood forecast after every cycle that brought newer RID or HII readings
|
||||
(and at least hourly), stores it in `forecast_history` and `forecast_current`
|
||||
(the Python API serves the latter as `/forecast`, falling back to computing
|
||||
itself when it is older than `FORECAST_MAX_AGE_SECONDS`), and sends the ntfy
|
||||
notifications. The Python app then neither forecasts nor notifies.
|
||||
|
||||
- **Models** stay trained in Python. `src/ml/export.py` writes
|
||||
`models/flood_<station>.json.gz` next to each joblib bundle (`train_all()`
|
||||
does it, `retrain.sh` promotes it, `python -m src.ml.export` exports
|
||||
existing bundles); the collector reloads changed files on its next run.
|
||||
- **Shared constants** (thresholds, flood stages, upstream leads, notify
|
||||
settings) are compiled in from `forecast/flood_config.json`, generated by
|
||||
`uv run python -m src.ml.export --config`. `tests/test_model_export.py`
|
||||
fails while it differs from Python, and a change needs a collector rebuild.
|
||||
- **Notifications**: same topics, texts, hysteresis and `notification_state`
|
||||
keys as `src/notify.py`, so switching sends nothing twice. One change: for
|
||||
the stations whose ThaiWater twin is the same sensor, a reading newer than
|
||||
RID's (RID's report lags 2-10 h at night) drives the level alerts, marked
|
||||
"ThaiWater telemetry; the RID report is not in yet". "Gauge feed stale" now
|
||||
means no source has a reading for 3 h; RID alone lagging is reported once
|
||||
after `RID_STALE_ALERT_HOURS`.
|
||||
- Memory: 70 MB peak with all 15 models loaded (24-byte tree nodes).
|
||||
|
||||
Verified:
|
||||
|
||||
- `forecast/tests/parity.rs` replays fixtures from the Python production path
|
||||
(`forecast/testdata/gen_forecast_fixtures.py`: the 2024 flood rising and at
|
||||
its peak with RID lagging behind HII, a quiet window with dam data, a
|
||||
stale-feed window, moved thresholds). HII fill, every feature and every
|
||||
level are bit-identical; with all 15 local models 377 of 384 probabilities
|
||||
are bit-identical and the rest within 2 ulp (`exp`). CI checks the P.1 model.
|
||||
- `scripts/forecast_parity.py` does the same on the live database: Python and
|
||||
`ping-collector forecast --json` at the same moment with the same rain
|
||||
series. The install script runs it before `--cutover-forecast`.
|
||||
- `collector/testdata/gen_notify_cases.py`: 652 hourly steps through
|
||||
`src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy
|
||||
outages); the Rust notifier sends the same 751 messages word for word.
|
||||
|
||||
### Develop
|
||||
|
||||
```bash
|
||||
@@ -90,8 +139,13 @@ the same commit):
|
||||
|
||||
```bash
|
||||
cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI artifact>
|
||||
sudo bash scripts/install_collector.sh # install + dry run, nothing switched
|
||||
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
||||
sudo bash scripts/install_collector.sh --rollback # back to Python collection
|
||||
sudo bash scripts/install_collector.sh # install + checks, nothing switched
|
||||
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
||||
sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies
|
||||
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
|
||||
sudo bash scripts/install_collector.sh --rollback # Python does everything again
|
||||
journalctl -u ping-collector -f
|
||||
```
|
||||
|
||||
Every install exports the models and runs `scripts/forecast_parity.py`;
|
||||
`--cutover-forecast` refuses to switch if it fails (`FORCE=1` overrides).
|
||||
Reference in new issue
Block a user