grabowski
0e30ddbcb6
ping-status sent "gauge feed stale" around midnight and "feed recovered" around 07:00 on each of the last three nights (ntfy and, since phase 3, the Matrix room). RID publishes no hourly report overnight: the 21:00 reading is the last until the 22:00..06:00 hours arrive together (06:35, 06:55, 07:22 on 2026-10-17/18/19). The ThaiWater overlay was meant to bridge that, but ThaiWater's copies of the RID gauges come from the same feed with the same delay (P.1's mirror still has no value for 22:00-00:00 this morning), so it does not. Between 21:00 and FEED_NIGHT_UNTIL_HOUR (default 9, 0 = off) the feed of a quiet river is no longer reported stale; one still without new readings at 09:00 is. While a station is at warning/danger or the P.1 outlook is on, the 3 h rule holds all night. A feed already stale stays stale through the night (no false "recovered"), and a real recovery is announced at any hour. The replay of the Python notifier runs with the window off. Docs no longer claim that ThaiWater keeps the feed current at night.
170 lines
8.8 KiB
Markdown
170 lines
8.8 KiB
Markdown
# Flood notifications (ntfy)
|
|
|
|
Public push notifications for threshold crossings, without accounts, mailing
|
|
lists or app-store review: the monitor publishes to a self-hosted
|
|
[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
|
|
about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
|
|
|
|
ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
|
|
runs on the same VPS as the monitor.
|
|
|
|
## What subscribers get
|
|
|
|
Every message is a **transition**, never a state. Crossing up into a level sends
|
|
one message; dropping back below it (with 0.10 m hysteresis) sends one
|
|
all-clear. A river that sits at 3.9 m for three days produces two messages, not
|
|
seventy-two. In a quiet season a subscriber hears nothing.
|
|
|
|
| Topic | Trigger | Priority |
|
|
|---|---|---|
|
|
| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
|
|
| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
|
|
| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
|
|
| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
|
|
| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
|
|
| `ping-status` | no reading for ≥ 3 h; feed recovered; with the Rust notifier also: RID reports ≥ 18 h behind while ThaiWater is current, and back | 3 / 2 |
|
|
|
|
Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
|
|
Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
|
|
4.20 m, P.103 5.95 / 6.75 m, and so on.
|
|
|
|
The outlook topic is opt-in for a reason: it is model output, and the message
|
|
says so. Observed-crossing topics only ever report a gauge reading.
|
|
|
|
Each message carries a click-through and an "Open dashboard" action button to
|
|
the public dashboard.
|
|
|
|
## How it runs
|
|
|
|
The notifier runs in `ping-collector` (`rust/collector/src/notify.rs`) when
|
|
`FORECAST_IN_COLLECTOR=true`, after each forecast run. It is a port of the
|
|
Python notifier it replaced in 2026-10, with the same topics, texts and state
|
|
keys (checked word for word on 751 generated messages), so the switch sent
|
|
nothing twice. One difference: it also uses ThaiWater (HII) telemetry. For the stations whose HII twin
|
|
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
|
|
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
|
|
message then says "ThaiWater telemetry; the RID report is not in yet". The feed
|
|
counts as stale when no source has a reading for 3 h, and RID alone lagging is
|
|
reported once after `RID_STALE_ALERT_HOURS` (18, 0 = off).
|
|
|
|
**RID's nightly pause.** RID publishes no hourly report overnight: on
|
|
2026-10-17, -18 and -19 the 21:00 reading was the last one until the hours
|
|
22:00-06:00 arrived together between 06:35 and 07:22. ThaiWater's copies of the
|
|
RID gauges come from the same feed and arrive with the same delay (P.1's
|
|
mirror had no value for 22:00-00:00 even the next morning), so they do not
|
|
bridge the night. Without a rule for it, every night sent "gauge feed stale"
|
|
around midnight and "feed recovered" around 07:00. So between 21:00 and
|
|
`FEED_NIGHT_UNTIL_HOUR` (9, 0 = off) a quiet river's feed is not reported
|
|
stale; a feed still without new readings at 09:00 is. While any station is at
|
|
warning or danger, or the P.1 outlook is on, the 3 h rule holds all night:
|
|
then a stalled dashboard matters. A feed that was already stale stays stale
|
|
through the night, and a recovery is announced whenever it happens. With `MATRIX_*` set, the
|
|
same transitions also go to a Matrix room, each once (per station, outlook,
|
|
feed; not the per-topic summaries), with their own `matrix:` state keys.
|
|
|
|
It runs in the same process that serves the dashboard, right after the
|
|
forecast, so it sees exactly the readings and forecasts the dashboard shows.
|
|
Per-key last-sent state is stored in the `notification_state` table of the
|
|
monitor's own database, so a restart or redeploy never re-sends and never
|
|
misses a crossing that happened while the service was down (the next cycle
|
|
compares against the persisted state).
|
|
|
|
If ntfy is unreachable the transition is **not** recorded, so it is retried
|
|
on the next cycle rather than silently lost. Any other failure in the notify
|
|
step is logged and never reaches the collection loop.
|
|
|
|
The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
|
|
set; it reads `GET /api/notifications` and renders subscribe links
|
|
(`ntfy://` deep links for the app, https links for the web UI).
|
|
|
|
## Deployment
|
|
|
|
On the monitor VPS, as root:
|
|
|
|
```bash
|
|
cd /opt/thailand-water-monitor
|
|
NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
|
|
```
|
|
|
|
This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
|
|
host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
|
|
message cache, signup/login/metrics off, tight visitor limits), enables the
|
|
systemd unit,
|
|
creates the `monitor` user with **write-only access to `ping-*`**, mints a
|
|
token, and appends `NTFY_SERVER` (public URL for subscribers),
|
|
`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
|
|
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
|
|
|
|
```bash
|
|
systemctl restart ping-collector
|
|
journalctl -u ping-collector -n 20 | grep ntfy # "ntfy notifications: publish to ..., topics ping-*"
|
|
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
|
|
```
|
|
|
|
The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
|
|
the monitor host's Tailscale address and nothing is exposed on a public
|
|
interface. On the Caddy machine:
|
|
|
|
```caddyfile
|
|
ntfy.buildfor.life {
|
|
reverse_proxy <monitor tailscale ip>:2586
|
|
}
|
|
```
|
|
|
|
Caddy proxies websockets and keeps long-poll connections open by default;
|
|
subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
|
|
`X-Forwarded-For` rather than treating every subscriber as the proxy.
|
|
|
|
Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
|
|
monitor at the Tailscale address directly, so a DNS or proxy problem never
|
|
holds back an alert. Test the pipeline before the domain is live with
|
|
`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
|
|
| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
|
|
| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
|
|
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
|
|
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
|
|
|
|
| `RID_STALE_ALERT_HOURS` | `18` | report RID lagging this long while ThaiWater is current (0 = never) |
|
|
| `FEED_NIGHT_UNTIL_HOUR` | `9` | end of the night window from 21:00 in which a quiet river's feed is not reported stale (0 = off) |
|
|
| `MATRIX_HOMESERVER` / `MATRIX_ACCESS_TOKEN` / `MATRIX_ROOM_ID` | *(empty = off)* | the team room (docs/MATRIX_QUICK_START.md) |
|
|
|
|
Tunables in `src/ml/export.py` (compiled into ping-collector through
|
|
`rust/forecast/flood_config.json`: regenerate with
|
|
`uv run python -m src.ml.export --config` and rebuild): `CLEAR_MARGIN_M`
|
|
(0.10), `OUTLOOK_ON` / `OUTLOOK_OFF` (0.50 / 0.25), `FEED_STALE_AFTER_H` (3),
|
|
the capacity guard and the station names in the messages.
|
|
|
|
## Testing
|
|
|
|
The tests in `rust/collector/src/notify.rs` cover the state machine: quiet
|
|
river sends nothing; crossing once, then silence while above, then all-clear;
|
|
hysteresis on the way down; escalation to danger and back; outlook on/off with
|
|
heuristic forecasts ignored; HII readings replacing a lagging RID reading;
|
|
stale feed only when both sources are; RID lagging reported once; a failed
|
|
publish retried next cycle; the Matrix rendering; and the 652-step replay of
|
|
the Python notifier's messages. `notification_state` round-trips against
|
|
PostgreSQL in `db_tests.rs`.
|
|
|
|
`ping-collector notify --dry-run` shows the readings judged and what would be
|
|
sent right now, changing nothing. To exercise the real path against a real
|
|
ntfy locally: run `ntfy serve` (any platform, same binary), point
|
|
`NTFY_SERVER`/`NTFY_TOKEN` at it and poll the topic JSON.
|
|
|
|
## Why ntfy and not …
|
|
|
|
- **Matrix**: needs a homeserver account per subscriber and a room invite;
|
|
fine for a team, wrong for the public. The team room gets the same
|
|
transitions from ping-collector (docs/MATRIX_QUICK_START.md).
|
|
- **Gotify**: also self-hosted and light, but Android-only client and one
|
|
account per subscriber.
|
|
- **Email / SMS**: deliverability work, cost per message, no priority
|
|
semantics; ntfy can forward to email per subscription if someone wants it.
|
|
- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
|
|
added later as ntfy→webhook fan-out without touching the monitor.
|