feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s

Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared
.env) ping-collector, after each cycle that brought newer RID or HII readings
and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the
rain forecast, runs every exported model (rust/forecast), stores
forecast_history and forecast_current, and sends the ntfy notifications.
The Python API then serves forecast_current as /forecast (computing in-process
only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor
notifies.

ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics,
texts, hysteresis, capacity guard and notification_state keys, checked word
for word on 751 messages generated by Python (testdata/gen_notify_cases.py).
The change: for stations whose ThaiWater twin is the same sensor, a reading
newer than RID's drives the level alerts (marked as ThaiWater telemetry), and
the feed counts as stale only when neither source has a reading for 3 h. RID
alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the
nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports.

Commands: `forecast [--json] [--save] [--now] [--rain-file]`,
`notify --dry-run`. scripts/forecast_parity.py compares Python and the
collector on the live database (same moment, same rain); locally 48/48 rows
identical. install_collector.sh: models export + that check on every install,
--cutover-forecast (refuses on a failed check) and --rollback-forecast.

Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140).
Forecast values are written to NUMERIC as shortest round-trip text, so they
round as psycopg2's do. DB test schemas get a counter (parallel tests collided
on Windows' coarse clock).
This commit is contained in:
grabowski committed 2026-10-18 14:43:39 +07:00
1 parent 5c4f464383
commit 6ae67dec63
27 files changed
+2656 -118

No files matched your search

+7
View File
@@ -616,6 +616,13 @@ fix, the served features match what this section evaluates.
### API
With `FORECAST_IN_COLLECTOR=true` the forecast is computed by `ping-collector`
(`rust/forecast`, a parity-tested port of this serving path) from
`models/flood_<station>.json.gz`, which `train_all()` exports next to each
joblib bundle; `/forecast` serves its `forecast_current` row and computes
in-process only when that is older than `FORECAST_MAX_AGE_SECONDS` (2 h). See
`rust/README.md`. The rest of this section describes the in-process path.
`GET /forecast` (`src/web_api.py`) returns one JSON row per station × horizon with
the fields listed in section 1. Results are cached in-process for
`FORECAST_TTL = 900` seconds (15 minutes), which matches the data cadence — the
+16 -2
View File
@@ -22,7 +22,7 @@ seventy-two. In a quiet season a subscriber hears nothing.
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
| `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
| `ping-status` | no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |
Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
@@ -36,7 +36,21 @@ the public dashboard.
## How it runs
`src/notify.py` is called once per collection cycle inside the API process
Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
forecast run. It is a port of `src/notify.py` with the same topics, texts and
state keys (checked word for word against Python on 751 generated messages),
so switching between the two sends nothing twice. One difference: the Rust
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
around midnight and "recovered" in the morning. Now the feed is stale only
when neither source has a reading for 3 h, and RID alone lagging is reported
once after `RID_STALE_ALERT_HOURS` (18, 0 = off).
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
(leader only), right after the forecast precompute, so it sees exactly the
readings and forecasts the dashboard shows. Per-key last-sent state is stored
in the `notification_state` table of the monitor's own database, so a restart
+17 -10
View File
@@ -1,6 +1,7 @@
# A Rust rewrite: assessment
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17; see
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17;
phase 2 (forecast + ntfy) is built and parity-verified, cutover pending; see
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
## Short version
@@ -51,10 +52,10 @@ One binary, one process (tokio):
the 257 KB river GeoJSON), static dashboard, CORS; `utoipa` if `/docs` stays.
- sqlx against the existing schema. Python and Rust can run against the same
database during migration.
- models: `scripts/retrain.sh` also exports each bundle to JSON (spike format,
~4.9 MB per station vs 6.2 MB joblib). Rust loads them at start and reloads on
change. A bundle whose `feature_names` Rust cannot build is refused and the
station falls back to the heuristic, as Python does today.
- models: `train_all()` also exports each bundle (`models/flood_<station>.json.gz`,
1.2-2.3 MB), `scripts/retrain.sh` promotes it. Rust reloads changed files on
its next run. A bundle whose `feature_names` Rust cannot build falls back to
the heuristic, as Python does.
## The permanent cost: the Python/Rust model contract
@@ -83,7 +84,7 @@ This slows model iteration, which is the most active part of the repo.
| --- | --- | --- |
| 0 | Spike: P.1 features + HGB inference | done, bit-identical |
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
| 2 | Forecast crate + ntfy | next |
| 2 | Forecast crate + ntfy | built 2026-10-18, parity-verified; cutover with `install_collector.sh --cutover-forecast` |
| 3 | Read API | |
| 4 | Cleanup | |
@@ -95,10 +96,16 @@ This slows model iteration, which is the most active part of the repo.
RID reading in the database advances. ntfy moved to phase 2: the outlook
alert needs the forecasts, which are still Python. Cutover and rollback
are one command each (`scripts/install_collector.sh`).
2. **Forecast crate + ntfy** (~2k LOC): all 15 stations, dam/HII/rain inputs,
heuristic fallback, probabilities and stages, `forecast_history`,
`notification_state`; parity CI for every station on a flood window and a
dry window. The Python API then only serves.
2. **Forecast crate + ntfy** (`rust/forecast`, run by `ping-collector` with
`FORECAST_IN_COLLECTOR=true`): all stations, HII/rain/dam inputs, heuristic
fallback, probabilities and stages, `forecast_history`, `forecast_current`
(served by the API) and `notification_state`. Parity: fixtures from the
Python production path (CI: P.1 model; locally all 15), a live comparison
script, and 751 notifications replayed word for word. Porting found a
Python serving bug, fixed first: forecast rain (`rain_fc24`) was always
NaN at serving time (docs/FLOOD_FORECASTING.md section 5). The ntfy change:
ThaiWater mirror readings keep level alerts and the feed check current
while RID lags at night. The Python API then only serves.
3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
then a Caddy switch. Python keeps only training/evaluation.
4. **Cleanup:** drop the multi-backend adapters, demo databases and the