grabowski
779728e38b
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
Northern Thailand Ping River Monitor
Live water levels, discharge, rainfall and machine-learning flood forecasts for the Ping River basin around Chiang Mai. Collects hourly gauge data from public sources, keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST API, and 6/12/24-hour flood-risk forecasts per gauge.
Live: water.buildfor.life · API reference at /docs · built by buildfor.life after the October 2024 flood — background in Teaching a Model to See the Ping River Rise 13 Hours Early.
What it does
- Collects hourly water level and discharge from 16 Royal Irrigation Department (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and its quirks: docs/DATA_SOURCES.md.
- Fills gaps. The raw RID grid had readings for ~56 % of hours; a full-history
re-fetch plus HII cross-fill brought it to ~93 %.
GET /api/statsreports the current figure. - Forecasts. Per gauge and horizon, a gradient-boosted model predicts the rise within 6/12/24 h and the probability of crossing the station's warning and danger levels. Trained on the monitor's own history plus catchment rain; evaluated rolling-origin, event by event. On the October 2024 record flood, trained only on data through August 2024, the first alert came 13 hours before P.1 crossed 3.70 m. Everything about the model, including what did not work: docs/FLOOD_FORECASTING.md.
- Shows it. A Leaflet map with the river drawn as OSM geometry and styled by live discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the forecast card, a replay of the 2024 flood, English/Thai, light/dark.
- Notifies. Public push alerts over a self-hosted ntfy: one message when a gauge crosses its warning or danger level, one all-clear on the way down, an opt-in early-warning topic from the model, nothing in between. Subscribe from the free app, no account. Matrix room alerts for a team are also supported.
Quick start
Python 3.11 (3.13 breaks the pinned psycopg2-binary), PostgreSQL for anything
beyond a quick look, uv.
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
cd Northern-Thailand-Ping-River-Monitor
uv sync --python 3.11
cp .env.example .env # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
uv run python run.py --web-api # dashboard + API on http://localhost:8000
DB_TYPE=sqlite works for the dashboard and API; the forecasting path expects the
PostgreSQL history.
uv run python run.py --status # collector status
uv run python run.py --test # one collection cycle
uv run python run.py --fill-gaps 7 # re-fetch the last 7 days from RID
uv run python run.py --collect-hii # one ThaiWater/HII collection cycle
uv run python run.py --alert-check # evaluate thresholds, notify Matrix
uv run python scripts/train_flood_model.py --stations all # retrain (~12 min)
make test # pytest, synthetic data, no network
make format # black + isort (the CI contract)
RID collection in --web-api mode: polled at :05 past each hour
(RID_POLL_OFFSET_MINUTES); while RID has not published the current hour yet,
retried every RID_RETRY_MINUTES (10). Every poll stores all readings the report
has, and yesterday's report is re-fetched while it still has holes (RID's evening
hours often arrive 1-2 h late). On startup the leader re-fetches holes from the
last GAP_FILL_DAYS_ON_START (14) days; for older history run --fill-gaps all.
API
Read-only, no key, JSON. Base URL https://water.buildfor.life; timestamps are
Asia/Bangkok wall-clock without an offset suffix.
| Endpoint | Returns |
|---|---|
GET /stations |
The 16 RID gauges: code, Thai/English names, coordinates |
GET /measurements/latest?limit=N |
Newest reading per station |
GET /measurements/history/{code}?hours=N |
Hourly history; or ?start=YYYY-MM-DD&end=YYYY-MM-DD; limit ≤ 100000 |
GET /forecast |
Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
GET /api/forecast/history/{code}?hours=N&horizon=24 |
Forecasts as issued, for auditing lead time after the fact |
GET /api/hii/rainfall/latest, /api/hii/waterlevel/latest |
Latest ThaiWater/HII gauge readings |
GET /api/hii/rainfall/catchment?days=N |
HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
GET /api/forecast/skill?station_code=P.1 |
Issued forecasts vs what happened, per deployed model version |
GET /api/notifications |
ntfy server and topic names for the subscribe panel |
GET /api/stats |
Row counts per source, date range, coverage |
GET /health |
DB / upstream / memory checks |
Interactive reference with schemas: water.buildfor.life/docs. Responses are cached briefly server-side; poll no faster than once a minute — the data changes hourly.
Deployment
Production is a systemd unit on a small VPS behind Cloudflare, updated by git pull.
scripts/install.sh (run as root from a checkout) creates the water-monitor user,
deploys to /opt/thailand-water-monitor, runs uv sync into .venv, installs
water-monitor.service and the monthly water-monitor-retrain.timer.
sudo bash scripts/install.sh
sudo systemctl start water-monitor.service
systemctl list-timers water-monitor-retrain.timer
The retrain timer runs scripts/retrain.sh, which trains into models/.staging,
refuses to promote anything that is not a rain-enabled (hgb-v3+) set covering the
expected stations, and renames the bundles into place. Details and the operations
runbook: docs/FLOOD_FORECASTING.md sections 6–8.
Repository layout
src/ collector, API (web_api.py), dashboard (static/dashboard.html)
src/ml/ features, training, evaluation harness, prediction, rain/dam/HII loaders
scripts/ train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
tests/ pytest suite (synthetic data; no DB or network)
docs/ FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
models/ trained bundles + metrics.json (gitignored) and evaluation results (tracked)
.gitea/workflows/ ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
Documentation
- docs/FLOOD_FORECASTING.md — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
- docs/DATA_SOURCES.md — every ingested and candidate source, endpoints, quirks
- docs/STATION_MANAGEMENT_GUIDE.md — adding/editing gauges
- docs/DATABASE_DEPLOYMENT_GUIDE.md, POSTGRESQL_SETUP.md — database setup
- docs/NOTIFICATIONS.md — public push alerts: topics, semantics, ntfy deployment
- docs/MATRIX_QUICK_START.md — Matrix room alerts for a team
- docs/GAP_FILLING_GUIDE.md — data integrity tooling
- docs/references/NOTABLE_DOCUMENTS.md — official Thai government resources
- Public overview: buildfor.life/docs/tooling/ping-river-monitor
Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
dashboards under grafana/ are supported by the adapters but not what production
runs; see docs/VICTORIAMETRICS_SETUP.md if you want them.
Contributing
make format before committing (black 88 columns, isort black profile — the CI gate),
make test must stay green, tests use synthetic data only. See
CONTRIBUTING.md. Issues and merge requests on
git.b4l.co.th.
Data sources and thanks
Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) / ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry; Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All instruments are theirs; we aggregate, store, fill gaps and forecast.
License
MIT — see LICENSE.