grabowski
701bb87d34
CI / Format & lint (push) Successful in 49s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 27s
Security / Dependency vulnerabilities (push) Successful in 1m5s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 45s
Rust / fmt, clippy, test, build (push) Successful in 3m42s
CMU Water Center's Chiang Mai flood hazard map (CMFlood) replaces the older municipal map. It is built on the October 2024 flood and has the Ping leave its banks above P.1 4.20 m, with five stages: 4.30, 4.50, 4.70, 5.00 and 5.30 m (the 2024 record). The old map's seven stages (3.70-4.60 m) said the city floods from 3.70 m; CMU calls that map out of date. - P1_FLOOD_STAGES: the five CMU stages, discharge on RID's 2024 rating. /forecast's P.1 `stages` now has five entries (same format). Thresholds are unchanged: 4.20 m danger is the bank top, 3.70 m warning half a metre below (as on CMU's gauge page). flood_config.json and the Rust parity fixtures regenerated (only the stages changed). - flood-zones-p1.geojson: CMU's KML (L1..L5) as five disjoint zones, zone k floods first at stage k; scripts/build_flood_zones.py rebuilds it. - Dashboard: CMU's colours and credit (map attribution); the stage grid follows the number of stages; zone lookup for markers honours holes. The city verdict uses statusOf's classes: flooding (over the banks) from 4.20 m, high water from 3.70 m. Zones show by themselves when the river is over its banks or stage 1 is >= 50 % likely. Alert, replay and verdict texts no longer call 3.70 m city flooding (EN + TH). - Docs: DATA_SOURCES 5.1 (CMFlood: the map, flood marks, poles, city gauges), FLOOD_FORECASTING, DASHBOARD_REDESIGN, NOTIFICATIONS, README credits.
205 lines
12 KiB
Markdown
205 lines
12 KiB
Markdown
# Northern Thailand Ping River Monitor
|
||
|
||
Live water levels, discharge, rainfall and machine-learning flood forecasts for the
|
||
Ping River basin around Chiang Mai. Collects hourly gauge data from public sources,
|
||
keeps the full history in PostgreSQL, serves a bilingual dashboard and an open REST
|
||
API, forecasts the flood risk per gauge 6/12/24 hours ahead, and pushes alerts when
|
||
a gauge crosses its warning or danger level.
|
||
|
||
**Live: [water.buildfor.life](https://water.buildfor.life/)** · API reference at
|
||
[/docs](https://water.buildfor.life/docs) · built by [buildfor.life](https://buildfor.life)
|
||
after the [October 2024 flood](https://buildfor.life/blog/chiang-mai-flood-2024/) —
|
||
background in [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
|
||
|
||
|
||
|
||
[](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
|
||
[](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
|
||
[](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
|
||
[](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
|
||
[](LICENSE)
|
||
|
||
## What it does
|
||
|
||
- **Collects** hourly water level and discharge from 16 Royal Irrigation Department
|
||
(RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly
|
||
rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment
|
||
rainfall (archive + 48 h forecast); daily state of the large RID dams, Mae Ngat
|
||
above all. Every source and its quirks: [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md).
|
||
- **Fills gaps.** The raw RID grid had readings for ~56 % of hours; a full-history
|
||
re-fetch plus HII cross-fill brought it to ~93 %. RID's report often arrives
|
||
hours late in the evening, so late readings are always stored and holes are
|
||
re-fetched. `GET /api/stats` reports the current coverage.
|
||
- **Forecasts.** Per gauge and horizon, a gradient-boosted model predicts the rise
|
||
within 6/12/24 h and the probability of crossing the station's warning and danger
|
||
levels. Trained on the monitor's own history plus catchment rain; evaluated
|
||
rolling-origin, event by event. On the October 2024 record flood, trained only on
|
||
data through August 2024, the first alert came **13 hours before** P.1 crossed
|
||
3.70 m. Everything about the model, including what did not work:
|
||
[docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md).
|
||
- **Shows it.** One screen: Chiang Mai's flood verdict, the gauges with their status
|
||
and trend, a self-hosted map of the basin with rain and the city's inundation
|
||
zones, and a panel per station with its history since 2018, the forecast and how
|
||
accurate it has been. ThaiWater's basin stations, a replay of the 2024 flood,
|
||
Thai/English, dark/light.
|
||
- **Notifies.** Public push alerts over a self-hosted [ntfy](https://ntfy.sh): one
|
||
message when a gauge crosses its warning or danger level, one all-clear on the
|
||
way down, an opt-in early-warning topic from the model, nothing in between.
|
||
Subscribe from the free app, no account. A Matrix room can get the same
|
||
messages for a team. [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md).
|
||
|
||
## How it runs
|
||
|
||
```text
|
||
RID gauges · ThaiWater/HII · RID dams · Open-Meteo
|
||
│ hourly
|
||
▼
|
||
┌──────────────────────────────────────┐ ┌───────────────────────────┐
|
||
│ ping-collector (Rust, one process) │ │ monthly retrain (Python) │
|
||
│ collect → forecast → notify │◄───────│ scikit-learn, exports │
|
||
│ website + API on :8000 │ models │ models/flood_*.json.gz │
|
||
└──────┬──────────────┬────────────────┘ └─────────────┬─────────────┘
|
||
│ │ │
|
||
▼ ▼ │
|
||
ntfy topics, PostgreSQL ◄──────────── history ────────────┘
|
||
Matrix room (gauges, rain, dams, forecasts, alert state)
|
||
```
|
||
|
||
- **The service** is one Rust binary, `ping-collector` ([rust/README.md](rust/README.md)),
|
||
run by systemd. It polls RID at :05 past each hour and retries every 10 minutes
|
||
while the report is behind, fetches HII, the dams and the rain once an hour,
|
||
forecasts after every cycle that brought new readings, sends the notifications,
|
||
and serves the dashboard and API, in under 100 MB of memory with every model
|
||
loaded.
|
||
- **The models** are trained in Python (`src/ml`, scikit-learn
|
||
HistGradientBoosting) by a monthly timer and exported for the Rust forecaster,
|
||
which reproduces Python's features bit for bit and reloads new models without
|
||
a restart.
|
||
- Until October 2026 all of this was a Python/FastAPI app; how and why it moved:
|
||
[docs/RUST_REWRITE.md](docs/RUST_REWRITE.md).
|
||
|
||
## API
|
||
|
||
Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
|
||
Asia/Bangkok wall-clock without an offset suffix.
|
||
|
||
| Endpoint | Returns |
|
||
| --- | --- |
|
||
| `GET /stations` | The 16 RID gauges: code, Thai/English names, coordinates |
|
||
| `GET /measurements/latest?limit=N` | Newest reading per station |
|
||
| `GET /measurements/history/{code}?hours=N` | Hourly history; or `?start=YYYY-MM-DD&end=YYYY-MM-DD`; `limit` ≤ 100000 |
|
||
| `GET /forecast` | Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
|
||
| `GET /api/forecast/history/{code}?hours=N&horizon=24` | Forecasts as issued, for auditing lead time after the fact |
|
||
| `GET /api/forecast/skill?station_code=P.1` | Issued forecasts vs what happened, per deployed model version |
|
||
| `GET /api/hii/rainfall/latest`, `/api/hii/waterlevel/latest` | Latest ThaiWater/HII gauge readings |
|
||
| `GET /api/hii/rainfall/catchment?days=N` | HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
|
||
| `GET /api/notifications` | ntfy server and topic names for the subscribe panel |
|
||
| `GET /api/stats` | Row counts per source, date range, coverage |
|
||
| `GET /health` | Database and data freshness (RID, ThaiWater as backup), the RID API, memory |
|
||
| `GET /scraping/status` | Collection runs, the last forecast run, service version |
|
||
|
||
Interactive reference with schemas: [water.buildfor.life/docs](https://water.buildfor.life/docs)
|
||
(also `/redoc`, `/openapi.json`). Responses are cached briefly server-side; poll no
|
||
faster than once a minute — the data changes hourly. Station editing
|
||
(`POST/PUT/DELETE /stations`), `/config` and `POST /scrape/trigger` need the
|
||
server's `ADMIN_API_KEY` in an `X-API-Key` header.
|
||
|
||
## Run it locally
|
||
|
||
Needs Rust >= 1.88 ([rustup](https://rustup.rs); on Linux also `pkg-config` and
|
||
`libssl-dev`) and PostgreSQL. Python 3.11 with [uv](https://docs.astral.sh/uv/)
|
||
only for training models.
|
||
|
||
```bash
|
||
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
|
||
cd Northern-Thailand-Ping-River-Monitor
|
||
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
||
cp .env.example .env
|
||
# POSTGRES_CONNECTION_STRING=postgresql://postgres:ping@localhost:55432/ping
|
||
# WEB_LISTEN=127.0.0.1:8000
|
||
cargo build --release --manifest-path rust/Cargo.toml
|
||
rust/target/release/ping-collector check-db # create the tables
|
||
rust/target/release/ping-collector fill-gaps 30 # the last 30 days from RID
|
||
rust/target/release/ping-collector run # collect, forecast, serve http://127.0.0.1:8000
|
||
```
|
||
|
||
Notifications stay off until `NTFY_SERVER` or `MATRIX_*` is set. Without trained
|
||
models the forecast is the persistence heuristic, marked `source: "heuristic"`.
|
||
Training needs years of history (the server's database, or `FLOOD_ML_DB_URL`) and
|
||
the Open-Meteo archive:
|
||
|
||
```bash
|
||
uv sync --python 3.11
|
||
uv run python scripts/train_flood_model.py --stations all # writes models/flood_*
|
||
uv run python scripts/evaluate_variants.py --stations P.1 # the rolling-origin harness
|
||
```
|
||
|
||
`make help` lists the rest (`make check` runs every CI gate locally).
|
||
|
||
## Deployment
|
||
|
||
Production is one systemd unit on a small VPS behind Cloudflare; the app directory
|
||
`/opt/thailand-water-monitor` is a git checkout. To update:
|
||
|
||
```bash
|
||
cd /opt/thailand-water-monitor && git pull
|
||
cargo build --release --locked --manifest-path rust/Cargo.toml
|
||
sudo bash scripts/install.sh
|
||
```
|
||
|
||
`scripts/install.sh` (idempotent, also for a fresh host) installs the binary and
|
||
`ping-collector.service`, syncs the Python environment for the monthly
|
||
`water-monitor-retrain.timer`, runs a dry run of every source and a forecast check
|
||
against the Python reference, restarts the service and waits for the website. The
|
||
retrain trains into `models/.staging`, refuses to promote anything that is not a
|
||
rain-enabled (`hgb-v3+`) set covering the expected stations, and renames the
|
||
bundles into place; the service picks them up on its next forecast. Details and
|
||
the operations runbook: [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md)
|
||
sections 6–8. Configuration: [.env.example](.env.example).
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
rust/collector/ ping-collector: collection, forecast run, ntfy/Matrix, website + API (src/web)
|
||
rust/forecast/ forecast inference from the exported models; parity tests against Python
|
||
src/static/ the dashboard (single file, EN/TH) and its map data
|
||
src/data/ the bundled station list
|
||
src/ml/ features, training, evaluation harness, export; predict.py is the reference
|
||
src/*.py HII and RID-dam clients/stores for the archive backfills
|
||
scripts/ install.sh, retrain.sh, train/evaluate/backfill tools, systemd units
|
||
tests/ pytest (synthetic data; database tests need TEST_DATABASE_URL)
|
||
docs/ model write-up, data sources, notifications, Matrix, rewrite history
|
||
models/ trained bundles (gitignored) and evaluation results
|
||
```
|
||
|
||
## Documentation
|
||
|
||
- [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
|
||
- [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) — every ingested and candidate source, endpoints, quirks
|
||
- [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md) — public push alerts: topics, semantics, ntfy deployment
|
||
- [docs/MATRIX_QUICK_START.md](docs/MATRIX_QUICK_START.md) — the same alerts in a Matrix room
|
||
- [rust/README.md](rust/README.md) — the service: commands, configuration, develop, deploy
|
||
- [docs/RUST_REWRITE.md](docs/RUST_REWRITE.md) — why and how it moved from Python to Rust
|
||
- [docs/DASHBOARD_REDESIGN.md](docs/DASHBOARD_REDESIGN.md) — the dashboard: layout, rules, how it replaced the old page
|
||
- [docs/references/NOTABLE_DOCUMENTS.md](docs/references/NOTABLE_DOCUMENTS.md) — official Thai government resources
|
||
- Public overview: [buildfor.life/docs/tooling/ping-river-monitor](https://buildfor.life/docs/tooling/ping-river-monitor/)
|
||
|
||
## Contributing
|
||
|
||
`make format` before committing, `make check` must stay green; see
|
||
[CONTRIBUTING.md](CONTRIBUTING.md). Issues and merge requests on
|
||
[git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
|
||
|
||
## Data sources and thanks
|
||
|
||
Royal Irrigation Department (RID) gauge telemetry and reservoir reports;
|
||
Hydro-Informatics Institute (HII) / ThaiWater open API; Open-Meteo; OpenStreetMap
|
||
contributors for the river geometry; the CMU Water Center (Chiang Mai University)
|
||
for the flood hazard map of the 2024 flood that the P.1 stages and the city's flood
|
||
zones come from. All instruments are theirs; we aggregate, store, fill
|
||
gaps and forecast.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE).
|