chore: remove the Python service; Python is the training stack only

Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558

No files matched your search

+33 -25
View File
@@ -36,12 +36,11 @@ the public dashboard.
## How it runs
Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
forecast run. It is a port of `src/notify.py` with the same topics, texts and
state keys (checked word for word against Python on 751 generated messages),
so switching between the two sends nothing twice. One difference: the Rust
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
The notifier runs in `ping-collector` (`rust/collector/src/notify.rs`) when
`FORECAST_IN_COLLECTOR=true`, after each forecast run. It is a port of the
Python notifier it replaced in 2026-10, with the same topics, texts and state
keys (checked word for word on 751 generated messages), so the switch sent
nothing twice. One difference: it also uses ThaiWater (HII) telemetry. For the stations whose HII twin
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
@@ -52,12 +51,12 @@ once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the
same transitions also go to a Matrix room, each once (per station, outlook,
feed; not the per-topic summaries), with their own `matrix:` state keys.
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
(leader only), right after the forecast precompute, so it sees exactly the
readings and forecasts the dashboard shows. Per-key last-sent state is stored
in the `notification_state` table of the monitor's own database, so a restart
or redeploy never re-sends and never misses a crossing that happened while
the service was down (the next cycle compares against the persisted state).
It runs in the same process that serves the dashboard, right after the
forecast, so it sees exactly the readings and forecasts the dashboard shows.
Per-key last-sent state is stored in the `notification_state` table of the
monitor's own database, so a restart or redeploy never re-sends and never
misses a crossing that happened while the service was down (the next cycle
compares against the persisted state).
If ntfy is unreachable the transition is **not** recorded, so it is retried
on the next cycle rather than silently lost. Any other failure in the notify
@@ -86,8 +85,8 @@ token, and appends `NTFY_SERVER` (public URL for subscribers),
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
```bash
systemctl restart water-monitor
journalctl -u water-monitor -n 20 | grep ntfy # "ntfy notifications: https://... topics ping-*"
systemctl restart ping-collector
journalctl -u ping-collector -n 20 | grep ntfy # "ntfy notifications: publish to ..., topics ping-*"
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
```
@@ -120,21 +119,30 @@ holds back an alert. Test the pipeline before the domain is live with
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
| `RID_STALE_ALERT_HOURS` | `18` | report RID lagging this long while ThaiWater is current (0 = never) |
| `MATRIX_HOMESERVER` / `MATRIX_ACCESS_TOKEN` / `MATRIX_ROOM_ID` | *(empty = off)* | the team room (docs/MATRIX_QUICK_START.md) |
Tunables in `src/ml/export.py` (compiled into ping-collector through
`rust/forecast/flood_config.json`: regenerate with
`uv run python -m src.ml.export --config` and rebuild): `CLEAR_MARGIN_M`
(0.10), `OUTLOOK_ON` / `OUTLOOK_OFF` (0.50 / 0.25), `FEED_STALE_AFTER_H` (3),
the capacity guard and the station names in the messages.
## Testing
`tests/test_notify.py` covers the state machine: quiet river sends nothing;
crossing once, then silence while above, then all-clear; hysteresis on the way
down; escalation to danger and back; basin digest grouping; outlook on/off;
heuristic forecasts ignored; stale feed and recovery; state survives a restart
through sqlite; a failed publish is retried next cycle.
The tests in `rust/collector/src/notify.rs` cover the state machine: quiet
river sends nothing; crossing once, then silence while above, then all-clear;
hysteresis on the way down; escalation to danger and back; outlook on/off with
heuristic forecasts ignored; HII readings replacing a lagging RID reading;
stale feed only when both sources are; RID lagging reported once; a failed
publish retried next cycle; the Matrix rendering; and the 652-step replay of
the Python notifier's messages. `notification_state` round-trips against
PostgreSQL in `db_tests.rs`.
To exercise the real path against a real ntfy locally: run `ntfy serve` (any
platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
template.
`ping-collector notify --dry-run` shows the readings judged and what would be
sent right now, changing nothing. To exercise the real path against a real
ntfy locally: run `ntfy serve` (any platform, same binary), point
`NTFY_SERVER`/`NTFY_TOKEN` at it and poll the topic JSON.
## Why ntfy and not …