grabowski
7e642555b4
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
7.9 KiB
Flood notifications (ntfy)
Public push notifications for threshold crossings, without accounts, mailing lists or app-store review: the monitor publishes to a self-hosted ntfy server, and anyone subscribes to the topics they care about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It runs on the same VPS as the monitor.
What subscribers get
Every message is a transition, never a state. Crossing up into a level sends one message; dropping back below it (with 0.10 m hysteresis) sends one all-clear. A river that sits at 3.9 m for three days produces two messages, not seventy-two. In a quiet season a subscriber hears nothing.
| Topic | Trigger | Priority |
|---|---|---|
ping-warning |
any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
ping-danger |
any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
ping-<station>-warning |
that gauge crosses warning; back to normal | 4 / 2 |
ping-<station>-danger |
that gauge crosses danger; back below danger | 5 / 3 |
ping-p1-outlook |
model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
ping-status |
no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |
Station slugs are the code lowercased without the dot: p1, p103, p67.
Thresholds are the ones in src/ml/features.py (THRESHOLDS): P.1 3.70 /
4.20 m, P.103 5.95 / 6.75 m, and so on.
The outlook topic is opt-in for a reason: it is model output, and the message says so. Observed-crossing topics only ever report a gauge reading.
Each message carries a click-through and an "Open dashboard" action button to the public dashboard.
How it runs
The notifier runs in ping-collector (rust/collector/src/notify.rs) when
FORECAST_IN_COLLECTOR=true, after each forecast run. It is a port of the
Python notifier it replaced in 2026-10, with the same topics, texts and state
keys (checked word for word on 751 generated messages), so the switch sent
nothing twice. One difference: it also uses ThaiWater (HII) telemetry. For the stations whose HII twin
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
around midnight and "recovered" in the morning. Now the feed is stale only
when neither source has a reading for 3 h, and RID alone lagging is reported
once after RID_STALE_ALERT_HOURS (18, 0 = off). With MATRIX_* set, the
same transitions also go to a Matrix room, each once (per station, outlook,
feed; not the per-topic summaries), with their own matrix: state keys.
It runs in the same process that serves the dashboard, right after the
forecast, so it sees exactly the readings and forecasts the dashboard shows.
Per-key last-sent state is stored in the notification_state table of the
monitor's own database, so a restart or redeploy never re-sends and never
misses a crossing that happened while the service was down (the next cycle
compares against the persisted state).
If ntfy is unreachable the transition is not recorded, so it is retried on the next cycle rather than silently lost. Any other failure in the notify step is logged and never reaches the collection loop.
The dashboard's "🔔 Get alerts" button appears only when NTFY_SERVER is
set; it reads GET /api/notifications and renders subscribe links
(ntfy:// deep links for the app, https links for the web UI).
Deployment
On the monitor VPS, as root:
cd /opt/thailand-water-monitor
NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
This installs the ntfy .deb, writes /etc/ntfy/server.yml (listen on the
host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
message cache, signup/login/metrics off, tight visitor limits), enables the
systemd unit,
creates the monitor user with write-only access to ping-*, mints a
token, and appends NTFY_SERVER (public URL for subscribers),
NTFY_PUBLISH_URL (loopback, what the monitor POSTs to), NTFY_TOPIC_PREFIX
and NTFY_TOKEN to .env if they are not there yet. Then:
systemctl restart ping-collector
journalctl -u ping-collector -n 20 | grep ntfy # "ntfy notifications: publish to ..., topics ping-*"
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on the monitor host's Tailscale address and nothing is exposed on a public interface. On the Caddy machine:
ntfy.buildfor.life {
reverse_proxy <monitor tailscale ip>:2586
}
Caddy proxies websockets and keeps long-poll connections open by default;
subscribers hold one open. behind-proxy: true makes ntfy rate-limit on
X-Forwarded-For rather than treating every subscriber as the proxy.
Publishing does not depend on the domain: NTFY_PUBLISH_URL points the
monitor at the Tailscale address directly, so a DNS or proxy problem never
holds back an alert. Test the pipeline before the domain is live with
curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'.
Configuration
| Variable | Default | Meaning |
|---|---|---|
NTFY_SERVER |
(empty = off) | public base URL subscribers use; shown on the dashboard |
NTFY_PUBLISH_URL |
= NTFY_SERVER |
where the monitor POSTs; the local ntfy address (http://<tailscale ip>:2586), so publishing never waits on DNS/proxy |
NTFY_TOPIC_PREFIX |
ping |
first segment of every topic |
NTFY_TOKEN |
(empty) | bearer token if the server requires auth to publish (it does, see above) |
PUBLIC_URL |
https://water.buildfor.life/ |
click-through target in messages |
| RID_STALE_ALERT_HOURS | 18 | report RID lagging this long while ThaiWater is current (0 = never) |
| MATRIX_HOMESERVER / MATRIX_ACCESS_TOKEN / MATRIX_ROOM_ID | (empty = off) | the team room (docs/MATRIX_QUICK_START.md) |
Tunables in src/ml/export.py (compiled into ping-collector through
rust/forecast/flood_config.json: regenerate with
uv run python -m src.ml.export --config and rebuild): CLEAR_MARGIN_M
(0.10), OUTLOOK_ON / OUTLOOK_OFF (0.50 / 0.25), FEED_STALE_AFTER_H (3),
the capacity guard and the station names in the messages.
Testing
The tests in rust/collector/src/notify.rs cover the state machine: quiet
river sends nothing; crossing once, then silence while above, then all-clear;
hysteresis on the way down; escalation to danger and back; outlook on/off with
heuristic forecasts ignored; HII readings replacing a lagging RID reading;
stale feed only when both sources are; RID lagging reported once; a failed
publish retried next cycle; the Matrix rendering; and the 652-step replay of
the Python notifier's messages. notification_state round-trips against
PostgreSQL in db_tests.rs.
ping-collector notify --dry-run shows the readings judged and what would be
sent right now, changing nothing. To exercise the real path against a real
ntfy locally: run ntfy serve (any platform, same binary), point
NTFY_SERVER/NTFY_TOKEN at it and poll the topic JSON.
Why ntfy and not …
- Matrix: needs a homeserver account per subscriber and a room invite; fine for a team, wrong for the public. The team room gets the same transitions from ping-collector (docs/MATRIX_QUICK_START.md).
- Gotify: also self-hosted and light, but Android-only client and one account per subscriber.
- Email / SMS: deliverability work, cost per message, no priority semantics; ntfy can forward to email per subscription if someone wants it.
- Telegram / LINE bots: platform lock-in and a bot token in the loop; can be added later as ntfy→webhook fan-out without touching the monitor.