grabowski
779728e38b
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
150 lines
7.4 KiB
Markdown
150 lines
7.4 KiB
Markdown
# Flood notifications (ntfy)
|
|
|
|
Public push notifications for threshold crossings, without accounts, mailing
|
|
lists or app-store review: the monitor publishes to a self-hosted
|
|
[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
|
|
about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
|
|
|
|
ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
|
|
runs on the same VPS as the monitor.
|
|
|
|
## What subscribers get
|
|
|
|
Every message is a **transition**, never a state. Crossing up into a level sends
|
|
one message; dropping back below it (with 0.10 m hysteresis) sends one
|
|
all-clear. A river that sits at 3.9 m for three days produces two messages, not
|
|
seventy-two. In a quiet season a subscriber hears nothing.
|
|
|
|
| Topic | Trigger | Priority |
|
|
|---|---|---|
|
|
| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
|
|
| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
|
|
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
|
|
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
|
|
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
|
|
| `ping-status` | no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |
|
|
|
|
Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
|
|
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
|
|
4.20 m, P.103 5.95 / 6.75 m, and so on.
|
|
|
|
The outlook topic is opt-in for a reason: it is model output, and the message
|
|
says so. Observed-crossing topics only ever report a gauge reading.
|
|
|
|
Each message carries a click-through and an "Open dashboard" action button to
|
|
the public dashboard.
|
|
|
|
## How it runs
|
|
|
|
Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
|
|
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
|
|
forecast run. It is a port of `src/notify.py` with the same topics, texts and
|
|
state keys (checked word for word against Python on 751 generated messages),
|
|
so switching between the two sends nothing twice. One difference: the Rust
|
|
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
|
|
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
|
|
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
|
|
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
|
|
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
|
|
around midnight and "recovered" in the morning. Now the feed is stale only
|
|
when neither source has a reading for 3 h, and RID alone lagging is reported
|
|
once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the
|
|
same transitions also go to a Matrix room, each once (per station, outlook,
|
|
feed; not the per-topic summaries), with their own `matrix:` state keys.
|
|
|
|
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
|
|
(leader only), right after the forecast precompute, so it sees exactly the
|
|
readings and forecasts the dashboard shows. Per-key last-sent state is stored
|
|
in the `notification_state` table of the monitor's own database, so a restart
|
|
or redeploy never re-sends and never misses a crossing that happened while
|
|
the service was down (the next cycle compares against the persisted state).
|
|
|
|
If ntfy is unreachable the transition is **not** recorded, so it is retried
|
|
on the next cycle rather than silently lost. Any other failure in the notify
|
|
step is logged and never reaches the collection loop.
|
|
|
|
The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
|
|
set; it reads `GET /api/notifications` and renders subscribe links
|
|
(`ntfy://` deep links for the app, https links for the web UI).
|
|
|
|
## Deployment
|
|
|
|
On the monitor VPS, as root:
|
|
|
|
```bash
|
|
cd /opt/thailand-water-monitor
|
|
NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
|
|
```
|
|
|
|
This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
|
|
host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
|
|
message cache, signup/login/metrics off, tight visitor limits), enables the
|
|
systemd unit,
|
|
creates the `monitor` user with **write-only access to `ping-*`**, mints a
|
|
token, and appends `NTFY_SERVER` (public URL for subscribers),
|
|
`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
|
|
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
|
|
|
|
```bash
|
|
systemctl restart water-monitor
|
|
journalctl -u water-monitor -n 20 | grep ntfy # "ntfy notifications: https://... topics ping-*"
|
|
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
|
|
```
|
|
|
|
The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
|
|
the monitor host's Tailscale address and nothing is exposed on a public
|
|
interface. On the Caddy machine:
|
|
|
|
```caddyfile
|
|
ntfy.buildfor.life {
|
|
reverse_proxy <monitor tailscale ip>:2586
|
|
}
|
|
```
|
|
|
|
Caddy proxies websockets and keeps long-poll connections open by default;
|
|
subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
|
|
`X-Forwarded-For` rather than treating every subscriber as the proxy.
|
|
|
|
Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
|
|
monitor at the Tailscale address directly, so a DNS or proxy problem never
|
|
holds back an alert. Test the pipeline before the domain is live with
|
|
`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
|
|
| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
|
|
| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
|
|
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
|
|
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
|
|
|
|
Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
|
|
(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
|
|
|
|
## Testing
|
|
|
|
`tests/test_notify.py` covers the state machine: quiet river sends nothing;
|
|
crossing once, then silence while above, then all-clear; hysteresis on the way
|
|
down; escalation to danger and back; basin digest grouping; outlook on/off;
|
|
heuristic forecasts ignored; stale feed and recovery; state survives a restart
|
|
through sqlite; a failed publish is retried next cycle.
|
|
|
|
To exercise the real path against a real ntfy locally: run `ntfy serve` (any
|
|
platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
|
|
poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
|
|
template.
|
|
|
|
## Why ntfy and not …
|
|
|
|
- **Matrix**: needs a homeserver account per subscriber and a room invite;
|
|
fine for a team, wrong for the public. The team room gets the same
|
|
transitions from ping-collector (docs/MATRIX_QUICK_START.md).
|
|
- **Gotify**: also self-hosted and light, but Android-only client and one
|
|
account per subscriber.
|
|
- **Email / SMS**: deliverability work, cost per message, no priority
|
|
semantics; ntfy can forward to email per subscription if someone wants it.
|
|
- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
|
|
added later as ntfy→webhook fan-out without touching the monitor.
|