grabowski
f4d42c90f4
Security / Dependency vulnerabilities (push) Successful in 44s
Security / Static analysis (push) Successful in 9s
CI / Format & lint (push) Successful in 10s
CI / Test suite (push) Successful in 26s
Security / License report (push) Successful in 50s
Docs / Validate documentation (push) Successful in 16s
The reverse proxy is a separate VPS on the tailnet, so a loopback-only ntfy was unreachable from it. install_ntfy.sh now binds the host's Tailscale IP (NTFY_LISTEN overrides). New NTFY_PUBLISH_URL: where the monitor POSTs, separate from the public NTFY_SERVER subscribers see, so an alert never waits on DNS or the proxy (first cycle logged 502s from Cloudflare while the domain was not yet proxied).
133 lines
6.1 KiB
Markdown
133 lines
6.1 KiB
Markdown
# Flood notifications (ntfy)
|
|
|
|
Public push notifications for threshold crossings, without accounts, mailing
|
|
lists or app-store review: the monitor publishes to a self-hosted
|
|
[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
|
|
about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
|
|
|
|
ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
|
|
runs on the same VPS as the monitor.
|
|
|
|
## What subscribers get
|
|
|
|
Every message is a **transition**, never a state. Crossing up into a level sends
|
|
one message; dropping back below it (with 0.10 m hysteresis) sends one
|
|
all-clear. A river that sits at 3.9 m for three days produces two messages, not
|
|
seventy-two. In a quiet season a subscriber hears nothing.
|
|
|
|
| Topic | Trigger | Priority |
|
|
|---|---|---|
|
|
| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
|
|
| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
|
|
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
|
|
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
|
|
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
|
|
| `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
|
|
|
|
Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
|
|
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
|
|
4.20 m, P.103 5.95 / 6.75 m, and so on.
|
|
|
|
The outlook topic is opt-in for a reason: it is model output, and the message
|
|
says so. Observed-crossing topics only ever report a gauge reading.
|
|
|
|
Each message carries a click-through and an "Open dashboard" action button to
|
|
the public dashboard.
|
|
|
|
## How it runs
|
|
|
|
`src/notify.py` is called once per collection cycle inside the API process
|
|
(leader only), right after the forecast precompute, so it sees exactly the
|
|
readings and forecasts the dashboard shows. Per-key last-sent state is stored
|
|
in the `notification_state` table of the monitor's own database, so a restart
|
|
or redeploy never re-sends and never misses a crossing that happened while
|
|
the service was down (the next cycle compares against the persisted state).
|
|
|
|
If ntfy is unreachable the transition is **not** recorded, so it is retried
|
|
on the next cycle rather than silently lost. Any other failure in the notify
|
|
step is logged and never reaches the collection loop.
|
|
|
|
The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
|
|
set; it reads `GET /api/notifications` and renders subscribe links
|
|
(`ntfy://` deep links for the app, https links for the web UI).
|
|
|
|
## Deployment
|
|
|
|
On the monitor VPS, as root:
|
|
|
|
```bash
|
|
cd /opt/thailand-water-monitor
|
|
NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
|
|
```
|
|
|
|
This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
|
|
host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
|
|
message cache, signup/login/metrics off, tight visitor limits), enables the
|
|
systemd unit,
|
|
creates the `monitor` user with **write-only access to `ping-*`**, mints a
|
|
token, and appends `NTFY_SERVER` (public URL for subscribers),
|
|
`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
|
|
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
|
|
|
|
```bash
|
|
systemctl restart water-monitor
|
|
journalctl -u water-monitor -n 20 | grep ntfy # "ntfy notifications: https://... topics ping-*"
|
|
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
|
|
```
|
|
|
|
The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
|
|
the monitor host's Tailscale address and nothing is exposed on a public
|
|
interface. On the Caddy machine:
|
|
|
|
```caddyfile
|
|
ntfy.buildfor.life {
|
|
reverse_proxy <monitor tailscale ip>:2586
|
|
}
|
|
```
|
|
|
|
Caddy proxies websockets and keeps long-poll connections open by default;
|
|
subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
|
|
`X-Forwarded-For` rather than treating every subscriber as the proxy.
|
|
|
|
Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
|
|
monitor at the Tailscale address directly, so a DNS or proxy problem never
|
|
holds back an alert. Test the pipeline before the domain is live with
|
|
`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
|
|
| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
|
|
| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
|
|
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
|
|
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
|
|
|
|
Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
|
|
(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
|
|
|
|
## Testing
|
|
|
|
`tests/test_notify.py` covers the state machine: quiet river sends nothing;
|
|
crossing once, then silence while above, then all-clear; hysteresis on the way
|
|
down; escalation to danger and back; basin digest grouping; outlook on/off;
|
|
heuristic forecasts ignored; stale feed and recovery; state survives a restart
|
|
through sqlite; a failed publish is retried next cycle.
|
|
|
|
To exercise the real path against a real ntfy locally: run `ntfy serve` (any
|
|
platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
|
|
poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
|
|
template.
|
|
|
|
## Why ntfy and not …
|
|
|
|
- **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
|
|
subscriber and a room invite; fine for a team, wrong for the public.
|
|
- **Gotify**: also self-hosted and light, but Android-only client and one
|
|
account per subscriber.
|
|
- **Email / SMS**: deliverability work, cost per message, no priority
|
|
semantics; ntfy can forward to email per subscription if someone wants it.
|
|
- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
|
|
added later as ntfy→webhook fan-out without touching the monitor.
|