feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock).
This commit is contained in:
1 parent
5c4f464383
commit
6ae67dec63
27 files changed
+2656
-118
No files matched your search
@@ -616,6 +616,13 @@ fix, the served features match what this section evaluates.
|
||||
|
||||
### API
|
||||
|
||||
With `FORECAST_IN_COLLECTOR=true` the forecast is computed by `ping-collector`
|
||||
(`rust/forecast`, a parity-tested port of this serving path) from
|
||||
`models/flood_<station>.json.gz`, which `train_all()` exports next to each
|
||||
joblib bundle; `/forecast` serves its `forecast_current` row and computes
|
||||
in-process only when that is older than `FORECAST_MAX_AGE_SECONDS` (2 h). See
|
||||
`rust/README.md`. The rest of this section describes the in-process path.
|
||||
|
||||
`GET /forecast` (`src/web_api.py`) returns one JSON row per station × horizon with
|
||||
the fields listed in section 1. Results are cached in-process for
|
||||
`FORECAST_TTL = 900` seconds (15 minutes), which matches the data cadence — the
|
||||
|
||||
+16
-2
@@ -22,7 +22,7 @@ seventy-two. In a quiet season a subscriber hears nothing.
|
||||
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
|
||||
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
|
||||
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
|
||||
| `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
|
||||
| `ping-status` | no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |
|
||||
|
||||
Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
|
||||
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
|
||||
@@ -36,7 +36,21 @@ the public dashboard.
|
||||
|
||||
## How it runs
|
||||
|
||||
`src/notify.py` is called once per collection cycle inside the API process
|
||||
Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
|
||||
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
|
||||
forecast run. It is a port of `src/notify.py` with the same topics, texts and
|
||||
state keys (checked word for word against Python on 751 generated messages),
|
||||
so switching between the two sends nothing twice. One difference: the Rust
|
||||
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
|
||||
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
|
||||
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
|
||||
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
|
||||
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
|
||||
around midnight and "recovered" in the morning. Now the feed is stale only
|
||||
when neither source has a reading for 3 h, and RID alone lagging is reported
|
||||
once after `RID_STALE_ALERT_HOURS` (18, 0 = off).
|
||||
|
||||
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
|
||||
(leader only), right after the forecast precompute, so it sees exactly the
|
||||
readings and forecasts the dashboard shows. Per-key last-sent state is stored
|
||||
in the `notification_state` table of the monitor's own database, so a restart
|
||||
|
||||
+17
-10
@@ -1,6 +1,7 @@
|
||||
# A Rust rewrite: assessment
|
||||
|
||||
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17; see
|
||||
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17;
|
||||
phase 2 (forecast + ntfy) is built and parity-verified, cutover pending; see
|
||||
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
|
||||
|
||||
## Short version
|
||||
@@ -51,10 +52,10 @@ One binary, one process (tokio):
|
||||
the 257 KB river GeoJSON), static dashboard, CORS; `utoipa` if `/docs` stays.
|
||||
- sqlx against the existing schema. Python and Rust can run against the same
|
||||
database during migration.
|
||||
- models: `scripts/retrain.sh` also exports each bundle to JSON (spike format,
|
||||
~4.9 MB per station vs 6.2 MB joblib). Rust loads them at start and reloads on
|
||||
change. A bundle whose `feature_names` Rust cannot build is refused and the
|
||||
station falls back to the heuristic, as Python does today.
|
||||
- models: `train_all()` also exports each bundle (`models/flood_<station>.json.gz`,
|
||||
1.2-2.3 MB), `scripts/retrain.sh` promotes it. Rust reloads changed files on
|
||||
its next run. A bundle whose `feature_names` Rust cannot build falls back to
|
||||
the heuristic, as Python does.
|
||||
|
||||
## The permanent cost: the Python/Rust model contract
|
||||
|
||||
@@ -83,7 +84,7 @@ This slows model iteration, which is the most active part of the repo.
|
||||
| --- | --- | --- |
|
||||
| 0 | Spike: P.1 features + HGB inference | done, bit-identical |
|
||||
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
|
||||
| 2 | Forecast crate + ntfy | next |
|
||||
| 2 | Forecast crate + ntfy | built 2026-10-18, parity-verified; cutover with `install_collector.sh --cutover-forecast` |
|
||||
| 3 | Read API | |
|
||||
| 4 | Cleanup | |
|
||||
|
||||
@@ -95,10 +96,16 @@ This slows model iteration, which is the most active part of the repo.
|
||||
RID reading in the database advances. ntfy moved to phase 2: the outlook
|
||||
alert needs the forecasts, which are still Python. Cutover and rollback
|
||||
are one command each (`scripts/install_collector.sh`).
|
||||
2. **Forecast crate + ntfy** (~2k LOC): all 15 stations, dam/HII/rain inputs,
|
||||
heuristic fallback, probabilities and stages, `forecast_history`,
|
||||
`notification_state`; parity CI for every station on a flood window and a
|
||||
dry window. The Python API then only serves.
|
||||
2. **Forecast crate + ntfy** (`rust/forecast`, run by `ping-collector` with
|
||||
`FORECAST_IN_COLLECTOR=true`): all stations, HII/rain/dam inputs, heuristic
|
||||
fallback, probabilities and stages, `forecast_history`, `forecast_current`
|
||||
(served by the API) and `notification_state`. Parity: fixtures from the
|
||||
Python production path (CI: P.1 model; locally all 15), a live comparison
|
||||
script, and 751 notifications replayed word for word. Porting found a
|
||||
Python serving bug, fixed first: forecast rain (`rain_fc24`) was always
|
||||
NaN at serving time (docs/FLOOD_FORECASTING.md section 5). The ntfy change:
|
||||
ThaiWater mirror readings keep level alerts and the feed check current
|
||||
while RID lags at night. The Python API then only serves.
|
||||
3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
|
||||
then a Caddy switch. Python keeps only training/evaluation.
|
||||
4. **Cleanup:** drop the multi-backend adapters, demo databases and the
|
||||
|
||||
Reference in new issue
Block a user