Northern-Thailand-Ping-River-Monitor/docs/NOTIFICATIONS.md

# Flood notifications (ntfy)

Public push notifications for threshold crossings, without accounts, mailing
lists or app-store review: the monitor publishes to a self-hosted
[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.

ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
runs on the same VPS as the monitor.

## What subscribers get

Every message is a **transition**, never a state. Crossing up into a level sends
one message; dropping back below it (with 0.10 m hysteresis) sends one
all-clear. A river that sits at 3.9 m for three days produces two messages, not
seventy-two. In a quiet season a subscriber hears nothing.

| Topic | Trigger | Priority |
|---|---|---|
| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
| `ping-status` | no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |

Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
4.20 m, P.103 5.95 / 6.75 m, and so on.

The outlook topic is opt-in for a reason: it is model output, and the message
says so. Observed-crossing topics only ever report a gauge reading.

Each message carries a click-through and an "Open dashboard" action button to
the public dashboard.

## How it runs

Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
forecast run. It is a port of `src/notify.py` with the same topics, texts and
state keys (checked word for word against Python on 751 generated messages),
so switching between the two sends nothing twice. One difference: the Rust
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
around midnight and "recovered" in the morning. Now the feed is stale only
when neither source has a reading for 3 h, and RID alone lagging is reported
once after `RID_STALE_ALERT_HOURS` (18, 0 = off).

With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
(leader only), right after the forecast precompute, so it sees exactly the
readings and forecasts the dashboard shows. Per-key last-sent state is stored
in the `notification_state` table of the monitor's own database, so a restart
or redeploy never re-sends and never misses a crossing that happened while
the service was down (the next cycle compares against the persisted state).

If ntfy is unreachable the transition is **not** recorded, so it is retried
on the next cycle rather than silently lost. Any other failure in the notify
step is logged and never reaches the collection loop.

The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
set; it reads `GET /api/notifications` and renders subscribe links
(`ntfy://` deep links for the app, https links for the web UI).

## Deployment

On the monitor VPS, as root:

```bash
cd /opt/thailand-water-monitor
NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
```

This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
message cache, signup/login/metrics off, tight visitor limits), enables the
systemd unit,
creates the `monitor` user with **write-only access to `ping-*`**, mints a
token, and appends `NTFY_SERVER` (public URL for subscribers),
`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:

```bash
systemctl restart water-monitor
journalctl -u water-monitor -n 20 | grep ntfy      # "ntfy notifications: https://... topics ping-*"
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1'   # anonymous read works
```

The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
the monitor host's Tailscale address and nothing is exposed on a public
interface. On the Caddy machine:

```caddyfile
ntfy.buildfor.life {
    reverse_proxy <monitor tailscale ip>:2586
}
```

Caddy proxies websockets and keeps long-poll connections open by default;
subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
`X-Forwarded-For` rather than treating every subscriber as the proxy.

Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
monitor at the Tailscale address directly, so a DNS or proxy problem never
holds back an alert. Test the pipeline before the domain is live with
`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.

## Configuration

| Variable | Default | Meaning |
|---|---|---|
| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |

Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).

## Testing

`tests/test_notify.py` covers the state machine: quiet river sends nothing;
crossing once, then silence while above, then all-clear; hysteresis on the way
down; escalation to danger and back; basin digest grouping; outlook on/off;
heuristic forecasts ignored; stale feed and recovery; state survives a restart
through sqlite; a failed publish is retried next cycle.

To exercise the real path against a real ntfy locally: run `ntfy serve` (any
platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
template.

## Why ntfy and not …

- **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
  subscriber and a room invite; fine for a team, wrong for the public.
- **Gotify**: also self-hosted and light, but Android-only client and one
  account per subscriber.
- **Email / SMS**: deliverability work, cost per message, no priority
  semantics; ntfy can forward to email per subscription if someone wants it.
- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
  added later as ntfy→webhook fan-out without touching the monitor.