chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
+33
-25
@@ -36,12 +36,11 @@ the public dashboard.
|
||||
|
||||
## How it runs
|
||||
|
||||
Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
|
||||
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
|
||||
forecast run. It is a port of `src/notify.py` with the same topics, texts and
|
||||
state keys (checked word for word against Python on 751 generated messages),
|
||||
so switching between the two sends nothing twice. One difference: the Rust
|
||||
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
|
||||
The notifier runs in `ping-collector` (`rust/collector/src/notify.rs`) when
|
||||
`FORECAST_IN_COLLECTOR=true`, after each forecast run. It is a port of the
|
||||
Python notifier it replaced in 2026-10, with the same topics, texts and state
|
||||
keys (checked word for word on 751 generated messages), so the switch sent
|
||||
nothing twice. One difference: it also uses ThaiWater (HII) telemetry. For the stations whose HII twin
|
||||
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
|
||||
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
|
||||
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
|
||||
@@ -52,12 +51,12 @@ once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the
|
||||
same transitions also go to a Matrix room, each once (per station, outlook,
|
||||
feed; not the per-topic summaries), with their own `matrix:` state keys.
|
||||
|
||||
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
|
||||
(leader only), right after the forecast precompute, so it sees exactly the
|
||||
readings and forecasts the dashboard shows. Per-key last-sent state is stored
|
||||
in the `notification_state` table of the monitor's own database, so a restart
|
||||
or redeploy never re-sends and never misses a crossing that happened while
|
||||
the service was down (the next cycle compares against the persisted state).
|
||||
It runs in the same process that serves the dashboard, right after the
|
||||
forecast, so it sees exactly the readings and forecasts the dashboard shows.
|
||||
Per-key last-sent state is stored in the `notification_state` table of the
|
||||
monitor's own database, so a restart or redeploy never re-sends and never
|
||||
misses a crossing that happened while the service was down (the next cycle
|
||||
compares against the persisted state).
|
||||
|
||||
If ntfy is unreachable the transition is **not** recorded, so it is retried
|
||||
on the next cycle rather than silently lost. Any other failure in the notify
|
||||
@@ -86,8 +85,8 @@ token, and appends `NTFY_SERVER` (public URL for subscribers),
|
||||
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
|
||||
|
||||
```bash
|
||||
systemctl restart water-monitor
|
||||
journalctl -u water-monitor -n 20 | grep ntfy # "ntfy notifications: https://... topics ping-*"
|
||||
systemctl restart ping-collector
|
||||
journalctl -u ping-collector -n 20 | grep ntfy # "ntfy notifications: publish to ..., topics ping-*"
|
||||
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
|
||||
```
|
||||
|
||||
@@ -120,21 +119,30 @@ holds back an alert. Test the pipeline before the domain is live with
|
||||
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
|
||||
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
|
||||
|
||||
Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
|
||||
(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
|
||||
| `RID_STALE_ALERT_HOURS` | `18` | report RID lagging this long while ThaiWater is current (0 = never) |
|
||||
| `MATRIX_HOMESERVER` / `MATRIX_ACCESS_TOKEN` / `MATRIX_ROOM_ID` | *(empty = off)* | the team room (docs/MATRIX_QUICK_START.md) |
|
||||
|
||||
Tunables in `src/ml/export.py` (compiled into ping-collector through
|
||||
`rust/forecast/flood_config.json`: regenerate with
|
||||
`uv run python -m src.ml.export --config` and rebuild): `CLEAR_MARGIN_M`
|
||||
(0.10), `OUTLOOK_ON` / `OUTLOOK_OFF` (0.50 / 0.25), `FEED_STALE_AFTER_H` (3),
|
||||
the capacity guard and the station names in the messages.
|
||||
|
||||
## Testing
|
||||
|
||||
`tests/test_notify.py` covers the state machine: quiet river sends nothing;
|
||||
crossing once, then silence while above, then all-clear; hysteresis on the way
|
||||
down; escalation to danger and back; basin digest grouping; outlook on/off;
|
||||
heuristic forecasts ignored; stale feed and recovery; state survives a restart
|
||||
through sqlite; a failed publish is retried next cycle.
|
||||
The tests in `rust/collector/src/notify.rs` cover the state machine: quiet
|
||||
river sends nothing; crossing once, then silence while above, then all-clear;
|
||||
hysteresis on the way down; escalation to danger and back; outlook on/off with
|
||||
heuristic forecasts ignored; HII readings replacing a lagging RID reading;
|
||||
stale feed only when both sources are; RID lagging reported once; a failed
|
||||
publish retried next cycle; the Matrix rendering; and the 652-step replay of
|
||||
the Python notifier's messages. `notification_state` round-trips against
|
||||
PostgreSQL in `db_tests.rs`.
|
||||
|
||||
To exercise the real path against a real ntfy locally: run `ntfy serve` (any
|
||||
platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
|
||||
poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
|
||||
template.
|
||||
`ping-collector notify --dry-run` shows the readings judged and what would be
|
||||
sent right now, changing nothing. To exercise the real path against a real
|
||||
ntfy locally: run `ntfy serve` (any platform, same binary), point
|
||||
`NTFY_SERVER`/`NTFY_TOKEN` at it and poll the topic JSON.
|
||||
|
||||
## Why ntfy and not …
|
||||
|
||||
|
||||
Reference in new issue
Block a user