grabowski
777b230baf
Anyone can now get push alerts on their phone without an account: the monitor publishes to an ntfy server (one Go binary, ~30 MB RSS) and subscribers pick topics in the free iOS/Android/web app. Semantics are transitions, never state. One message when a gauge crosses its warning or danger threshold, one all-clear when it drops back (0.10 m hysteresis), nothing while it sits above. A three-day flood is two messages; a quiet season is zero. Topics: ping-warning / ping-danger (basin digest), ping-<station>-warning / -danger, ping-p1-outlook (opt-in: model P(warning within 24 h) at P.1 rises through 50 %, clears below 25 %, message says it is experimental), ping-status (feed stale >= 3 h / recovered). Priority 5 on danger so it rings through Do Not Disturb. src/notify.py runs once per collection cycle in the API process (leader only, after the forecast precompute, same data the dashboard shows). Last-sent state lives in a notification_state table so a restart never re-sends; a failed publish leaves state untouched so the crossing is retried next cycle instead of lost. Off unless NTFY_SERVER is set. Dashboard: a "Get alerts" button (only when configured) opens a panel with the server, per-topic cards, ntfy:// deep links and web links, app store links and a disclaimer. EN + TH. GET /api/notifications feeds it. scripts/install_ntfy.sh: .deb install, server.yml (loopback listen, anonymous read, token-only write scoped to ping-*, 72 h cache, signup/ login/metrics off, tight visitor limits), systemd, user + token, .env. Verified against ntfy 2.28.0: anon publish 403, token publish 200, token on foreign topic 403, anon read 200, and a seeded crossing through the real _notify_transitions path arrived in the topic with priority, tags, click and action button. docs/NOTIFICATIONS.md has the deployment and reverse-proxy notes. Tests: 10 for the state machine (159 total).
9.0 KiB
Northern Thailand Ping River Monitor
Live water levels, discharge, rainfall and machine-learning flood forecasts for the Ping River basin around Chiang Mai. Collects hourly gauge data from public sources, keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST API, and 6/12/24-hour flood-risk forecasts per gauge.
Live: water.buildfor.life · API reference at /docs · built by buildfor.life after the October 2024 flood — background in Teaching a Model to See the Ping River Rise 13 Hours Early.
What it does
- Collects hourly water level and discharge from 16 Royal Irrigation Department (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and its quirks: docs/DATA_SOURCES.md.
- Fills gaps. The raw RID grid had readings for ~56 % of hours; a full-history
re-fetch plus HII cross-fill brought it to ~93 %.
GET /api/statsreports the current figure. - Forecasts. Per gauge and horizon, a gradient-boosted model predicts the rise within 6/12/24 h and the probability of crossing the station's warning and danger levels. Trained on the monitor's own history plus catchment rain; evaluated rolling-origin, event by event. On the October 2024 record flood, trained only on data through August 2024, the first alert came 13 hours before P.1 crossed 3.70 m. Everything about the model, including what did not work: docs/FLOOD_FORECASTING.md.
- Shows it. A Leaflet map with the river drawn as OSM geometry and styled by live discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the forecast card, a replay of the 2024 flood, English/Thai, light/dark.
- Notifies. Public push alerts over a self-hosted ntfy: one message when a gauge crosses its warning or danger level, one all-clear on the way down, an opt-in early-warning topic from the model, nothing in between. Subscribe from the free app, no account. Matrix room alerts for a team are also supported.
Quick start
Python 3.11 (3.13 breaks the pinned psycopg2-binary), PostgreSQL for anything
beyond a quick look, uv.
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
cd Northern-Thailand-Ping-River-Monitor
uv sync --python 3.11
cp .env.example .env # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
uv run python run.py --web-api # dashboard + API on http://localhost:8000
DB_TYPE=sqlite works for the dashboard and API; the forecasting path expects the
PostgreSQL history.
uv run python run.py --status # collector status
uv run python run.py --test # one collection cycle
uv run python run.py --fill-gaps 7 # re-fetch the last 7 days from RID
uv run python run.py --collect-hii # one ThaiWater/HII collection cycle
uv run python run.py --alert-check # evaluate thresholds, notify Matrix
uv run python scripts/train_flood_model.py --stations all # retrain (~12 min)
make test # pytest, synthetic data, no network
make format # black + isort (the CI contract)
API
Read-only, no key, JSON. Base URL https://water.buildfor.life; timestamps are
Asia/Bangkok wall-clock without an offset suffix.
| Endpoint | Returns |
|---|---|
GET /stations |
The 16 RID gauges: code, Thai/English names, coordinates |
GET /measurements/latest?limit=N |
Newest reading per station |
GET /measurements/history/{code}?hours=N |
Hourly history; or ?start=YYYY-MM-DD&end=YYYY-MM-DD; limit ≤ 100000 |
GET /forecast |
Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
GET /api/forecast/history/{code}?hours=N&horizon=24 |
Forecasts as issued, for auditing lead time after the fact |
GET /api/hii/rainfall/latest, /api/hii/waterlevel/latest |
Latest ThaiWater/HII gauge readings |
GET /api/hii/rainfall/catchment?days=N |
HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
GET /api/forecast/skill?station_code=P.1 |
Issued forecasts vs what happened, per deployed model version |
GET /api/notifications |
ntfy server and topic names for the subscribe panel |
GET /api/stats |
Row counts per source, date range, coverage |
GET /health |
DB / upstream / memory checks |
Interactive reference with schemas: water.buildfor.life/docs. Responses are cached briefly server-side; poll no faster than once a minute — the data changes hourly.
Deployment
Production is a systemd unit on a small VPS behind Cloudflare, updated by git pull.
scripts/install.sh (run as root from a checkout) creates the water-monitor user,
deploys to /opt/thailand-water-monitor, runs uv sync into .venv, installs
water-monitor.service and the monthly water-monitor-retrain.timer.
sudo bash scripts/install.sh
sudo systemctl start water-monitor.service
systemctl list-timers water-monitor-retrain.timer
The retrain timer runs scripts/retrain.sh, which trains into models/.staging,
refuses to promote anything that is not a rain-enabled (hgb-v3+) set covering the
expected stations, and renames the bundles into place. Details and the operations
runbook: docs/FLOOD_FORECASTING.md sections 6–8.
Repository layout
src/ collector, API (web_api.py), dashboard (static/dashboard.html)
src/ml/ features, training, evaluation harness, prediction, rain/dam/HII loaders
scripts/ train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
tests/ pytest suite (synthetic data; no DB or network)
docs/ FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
models/ trained bundles + metrics.json (gitignored) and evaluation results (tracked)
.gitea/workflows/ ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
Documentation
- docs/FLOOD_FORECASTING.md — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
- docs/DATA_SOURCES.md — every ingested and candidate source, endpoints, quirks
- docs/STATION_MANAGEMENT_GUIDE.md — adding/editing gauges
- docs/DATABASE_DEPLOYMENT_GUIDE.md, POSTGRESQL_SETUP.md — database setup
- docs/NOTIFICATIONS.md — public push alerts: topics, semantics, ntfy deployment
- docs/MATRIX_QUICK_START.md — Matrix room alerts for a team
- docs/GAP_FILLING_GUIDE.md — data integrity tooling
- docs/references/NOTABLE_DOCUMENTS.md — official Thai government resources
- Public overview: buildfor.life/docs/tooling/ping-river-monitor
Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
dashboards under grafana/ are supported by the adapters but not what production
runs; see docs/VICTORIAMETRICS_SETUP.md if you want them.
Contributing
make format before committing (black 88 columns, isort black profile — the CI gate),
make test must stay green, tests use synthetic data only. See
CONTRIBUTING.md. Issues and merge requests on
git.b4l.co.th.
Data sources and thanks
Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) / ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry; Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All instruments are theirs; we aggregate, store, fill gaps and forecast.
License
MIT — see LICENSE.