grabowski
7e642555b4
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
158 lines
7.9 KiB
Markdown
158 lines
7.9 KiB
Markdown
# Flood notifications (ntfy)
|
|
|
|
Public push notifications for threshold crossings, without accounts, mailing
|
|
lists or app-store review: the monitor publishes to a self-hosted
|
|
[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
|
|
about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
|
|
|
|
ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
|
|
runs on the same VPS as the monitor.
|
|
|
|
## What subscribers get
|
|
|
|
Every message is a **transition**, never a state. Crossing up into a level sends
|
|
one message; dropping back below it (with 0.10 m hysteresis) sends one
|
|
all-clear. A river that sits at 3.9 m for three days produces two messages, not
|
|
seventy-two. In a quiet season a subscriber hears nothing.
|
|
|
|
| Topic | Trigger | Priority |
|
|
|---|---|---|
|
|
| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
|
|
| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
|
|
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
|
|
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
|
|
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
|
|
| `ping-status` | no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |
|
|
|
|
Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
|
|
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
|
|
4.20 m, P.103 5.95 / 6.75 m, and so on.
|
|
|
|
The outlook topic is opt-in for a reason: it is model output, and the message
|
|
says so. Observed-crossing topics only ever report a gauge reading.
|
|
|
|
Each message carries a click-through and an "Open dashboard" action button to
|
|
the public dashboard.
|
|
|
|
## How it runs
|
|
|
|
The notifier runs in `ping-collector` (`rust/collector/src/notify.rs`) when
|
|
`FORECAST_IN_COLLECTOR=true`, after each forecast run. It is a port of the
|
|
Python notifier it replaced in 2026-10, with the same topics, texts and state
|
|
keys (checked word for word on 751 generated messages), so the switch sent
|
|
nothing twice. One difference: it also uses ThaiWater (HII) telemetry. For the stations whose HII twin
|
|
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
|
|
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
|
|
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
|
|
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
|
|
around midnight and "recovered" in the morning. Now the feed is stale only
|
|
when neither source has a reading for 3 h, and RID alone lagging is reported
|
|
once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the
|
|
same transitions also go to a Matrix room, each once (per station, outlook,
|
|
feed; not the per-topic summaries), with their own `matrix:` state keys.
|
|
|
|
It runs in the same process that serves the dashboard, right after the
|
|
forecast, so it sees exactly the readings and forecasts the dashboard shows.
|
|
Per-key last-sent state is stored in the `notification_state` table of the
|
|
monitor's own database, so a restart or redeploy never re-sends and never
|
|
misses a crossing that happened while the service was down (the next cycle
|
|
compares against the persisted state).
|
|
|
|
If ntfy is unreachable the transition is **not** recorded, so it is retried
|
|
on the next cycle rather than silently lost. Any other failure in the notify
|
|
step is logged and never reaches the collection loop.
|
|
|
|
The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
|
|
set; it reads `GET /api/notifications` and renders subscribe links
|
|
(`ntfy://` deep links for the app, https links for the web UI).
|
|
|
|
## Deployment
|
|
|
|
On the monitor VPS, as root:
|
|
|
|
```bash
|
|
cd /opt/thailand-water-monitor
|
|
NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
|
|
```
|
|
|
|
This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
|
|
host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
|
|
message cache, signup/login/metrics off, tight visitor limits), enables the
|
|
systemd unit,
|
|
creates the `monitor` user with **write-only access to `ping-*`**, mints a
|
|
token, and appends `NTFY_SERVER` (public URL for subscribers),
|
|
`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
|
|
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
|
|
|
|
```bash
|
|
systemctl restart ping-collector
|
|
journalctl -u ping-collector -n 20 | grep ntfy # "ntfy notifications: publish to ..., topics ping-*"
|
|
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
|
|
```
|
|
|
|
The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
|
|
the monitor host's Tailscale address and nothing is exposed on a public
|
|
interface. On the Caddy machine:
|
|
|
|
```caddyfile
|
|
ntfy.buildfor.life {
|
|
reverse_proxy <monitor tailscale ip>:2586
|
|
}
|
|
```
|
|
|
|
Caddy proxies websockets and keeps long-poll connections open by default;
|
|
subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
|
|
`X-Forwarded-For` rather than treating every subscriber as the proxy.
|
|
|
|
Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
|
|
monitor at the Tailscale address directly, so a DNS or proxy problem never
|
|
holds back an alert. Test the pipeline before the domain is live with
|
|
`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
|
|
| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
|
|
| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
|
|
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
|
|
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
|
|
|
|
| `RID_STALE_ALERT_HOURS` | `18` | report RID lagging this long while ThaiWater is current (0 = never) |
|
|
| `MATRIX_HOMESERVER` / `MATRIX_ACCESS_TOKEN` / `MATRIX_ROOM_ID` | *(empty = off)* | the team room (docs/MATRIX_QUICK_START.md) |
|
|
|
|
Tunables in `src/ml/export.py` (compiled into ping-collector through
|
|
`rust/forecast/flood_config.json`: regenerate with
|
|
`uv run python -m src.ml.export --config` and rebuild): `CLEAR_MARGIN_M`
|
|
(0.10), `OUTLOOK_ON` / `OUTLOOK_OFF` (0.50 / 0.25), `FEED_STALE_AFTER_H` (3),
|
|
the capacity guard and the station names in the messages.
|
|
|
|
## Testing
|
|
|
|
The tests in `rust/collector/src/notify.rs` cover the state machine: quiet
|
|
river sends nothing; crossing once, then silence while above, then all-clear;
|
|
hysteresis on the way down; escalation to danger and back; outlook on/off with
|
|
heuristic forecasts ignored; HII readings replacing a lagging RID reading;
|
|
stale feed only when both sources are; RID lagging reported once; a failed
|
|
publish retried next cycle; the Matrix rendering; and the 652-step replay of
|
|
the Python notifier's messages. `notification_state` round-trips against
|
|
PostgreSQL in `db_tests.rs`.
|
|
|
|
`ping-collector notify --dry-run` shows the readings judged and what would be
|
|
sent right now, changing nothing. To exercise the real path against a real
|
|
ntfy locally: run `ntfy serve` (any platform, same binary), point
|
|
`NTFY_SERVER`/`NTFY_TOKEN` at it and poll the topic JSON.
|
|
|
|
## Why ntfy and not …
|
|
|
|
- **Matrix**: needs a homeserver account per subscriber and a room invite;
|
|
fine for a team, wrong for the public. The team room gets the same
|
|
transitions from ping-collector (docs/MATRIX_QUICK_START.md).
|
|
- **Gotify**: also self-hosted and light, but Android-only client and one
|
|
account per subscriber.
|
|
- **Email / SMS**: deliverability work, cost per message, no priority
|
|
semantics; ntfy can forward to email per subscription if someone wants it.
|
|
- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
|
|
added later as ntfy→webhook fan-out without touching the monitor.
|