grabowski
7e642555b4
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
265 lines
17 KiB
Markdown
265 lines
17 KiB
Markdown
# Data Sources & External API Catalog
|
||
|
||
Catalog of every data source available to the Ping River Monitor — what we ingest
|
||
today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for
|
||
future model inputs (rainfall, dam releases, forecasts).
|
||
|
||
All "verified" claims below were empirically probed on **2026-08-11**. Endpoints
|
||
marked *(catalog)* were recovered from the thaiwater.net frontend JS bundle but
|
||
not exercised (auth required).
|
||
|
||
---
|
||
|
||
## 1. Currently ingested — RID hydrology telemetry
|
||
|
||
The only source persisted to the database and used by the ML pipeline.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Endpoint | `POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx` |
|
||
| Agency | Royal Irrigation Department (RID) |
|
||
| Auth | None |
|
||
| Cadence | Hourly (`hourlytime` 1.00–24.00; hour 24 = midnight next day) |
|
||
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
|
||
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
|
||
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
|
||
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports) |
|
||
|
||
Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
|
||
|
||
---
|
||
|
||
## 2. ThaiWater / HII ecosystem
|
||
|
||
ThaiWater (<https://twa.thaiwater.net>) is the National Hydroinformatics
|
||
Institute (HII) portal. It sits on **two distinct API layers** with very
|
||
different access rules.
|
||
|
||
### 2.1 `api-v3.thaiwater.net` — open, no authentication ✅
|
||
|
||
Base: `https://api-v3.thaiwater.net/api/v1/thaiwater30/public/`
|
||
|
||
Undocumented backend of the portal. No key, no session. No published rate
|
||
limits or terms of use — be a good citizen (hourly polls, cache, filter to
|
||
Ping basin `basin_code == 6`).
|
||
|
||
#### `waterlevel_load` — national water-level snapshot (verified)
|
||
|
||
```
|
||
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load
|
||
```
|
||
|
||
- ~2.4 MB, 1,426 stations nationwide; **125 in Ping Basin, 61 in Chiang Mai
|
||
province, 34 with discharge**.
|
||
- Per station: `waterlevel_m`, `waterlevel_msl`, `waterlevel_msl_previous`,
|
||
`flow_rate`, `discharge`, `storage_percent`, `situation_level` (1–4 flood
|
||
severity), `diff_wl_bank`, bank/critical levels (`min_bank`,
|
||
`critical_level_msl`, `warning_level_m`, `critical_level_m`, `qmax`),
|
||
`river_name`, basin, geocode, agency, lat/long, `is_key_station`.
|
||
- Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A,
|
||
P.7A, P.77, P.2A — plus RID mirrors (`ridhydro_P.1`, …), Tak-reach `TP.`/
|
||
`TUP.` codes, and HII sensor clusters (`PIN001-011`, `CHM001-005`).
|
||
- **P.1 = internal station id `3226`** (lat 18.786961, long 99.005089).
|
||
|
||
#### `waterlevel_graph` — hourly historical time series (verified) ⭐
|
||
|
||
```
|
||
GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08
|
||
```
|
||
|
||
- Returns `{data: {graph_data: [{datetime, value, value_out, discharge}]}}`,
|
||
hourly. `value` is **water level in m MSL** (not gauge datum).
|
||
- **Respects arbitrary date ranges. Archive verified back to at least
|
||
2019-08** (P.1 returned data for 2019-08-01). 14-day windows tested OK;
|
||
maximum window size not probed.
|
||
- Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00,
|
||
discharge 656 m³/s.
|
||
- Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒
|
||
**gauge ≈ MSL − 300.5 m** (verify per station before use; each station has
|
||
its own datum offset).
|
||
- Value: independent second historical source for cross-validating / gap-filling
|
||
the RID feed (RID grid is only ~56% filled).
|
||
|
||
#### `rain_24h` — national rainfall snapshot (verified)
|
||
|
||
```
|
||
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h
|
||
```
|
||
|
||
- ~4.5 MB, 4,445 stations; **321 in Chiang Mai province**; agencies include
|
||
HII, DWR, RID.
|
||
- Per station: `rain_1h`, `rain_24h` (mm), `rainfall_datetime`,
|
||
`station.id` (small int — the graph key), `station.tele_station_oldcode`
|
||
(e.g. `CHM005`, `STN0410`, `ridtele_TUP.14`), `sub_basin_id`, lat/long,
|
||
basin, geocode, agency.
|
||
- **This is the missing rainfall input** for the flood model — near-real-time
|
||
hourly gauge rain across the upper Ping catchment.
|
||
|
||
#### `rain_24h_graph` — trailing-window rainfall series (verified, limited) ⚠️
|
||
|
||
```
|
||
GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...
|
||
```
|
||
|
||
- `station_id` is the **small `station.id`** from `rain_24h` (e.g. 418 =
|
||
CHM005 "Chiang Mai 5", Mae Taeng), *not* the top-level record id.
|
||
- Returns hourly `{rainfall_datetime, rainfall_value}` — **but the date range
|
||
is IGNORED**: every request returns the same trailing ~36-hour window
|
||
(39 rows). Requests for 2020/2024 return identical data to today.
|
||
- Consequence: **no rainfall history via this API**. To build training data,
|
||
persist `rain_24h` from now on and backfill history from satellite QPE or an
|
||
HII data request (§4).
|
||
|
||
#### Probed and NOT available on api-v3 (all HTTP 404)
|
||
|
||
`dam_daily`, `dam`, `dam_json`, `big_dam`, `mainstream_dam`, `weather`,
|
||
`rain_graph`, `rainfall_graph`, `rain24hr_graph`. Dam data is v2-only (§2.2).
|
||
|
||
### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒
|
||
|
||
The layer the website's `GET /sensors/thaiwater` uses (`GET /v2/waterlevel`
|
||
with an `x-api-key` header from `THAIWATER_API_KEY`;
|
||
`rust/collector/src/web/hii.rs`, display-only, never persisted).
|
||
Without a key: HTTP 401/500. No public key-registration page was found —
|
||
obtain a sanctioned key from HII (<https://hii.or.th>).
|
||
|
||
Full endpoint catalog *(catalog — recovered from frontend JS, not exercised)*:
|
||
|
||
- **Water level / discharge**: `/v2/waterlevel`, `/v2/waterlevel/list`,
|
||
`/v2/waterlevel/{id}/detail`, `/v2/waterlevel/canal`,
|
||
`/v2/waterlevel/sea-waterlevel`, `/v2/waterlevel-discharge`,
|
||
`/v2/waterlevel-discharge/list`, `/v2/waterlevel-discharge/{id}/detail`,
|
||
`/v2/waterlevel-discharge/{id}/forecast-table`,
|
||
`/v2/waterlevel-discharge/forecast`, `/v2/waterlevel-discharge/forecast/list`,
|
||
`/v2/watergate`, `/v2/waterload-tide`
|
||
- **Dams** (incl. Bhumibol): `/v2/large-dam/daily-geo-json`,
|
||
`/v2/large-dam/daily/list`, `/v2/large-dam/hourly/list`,
|
||
`/v2/large-dam/daily/{id}/detail`, `/v2/large-dam/hourly/{id}/detail`,
|
||
`/v2/medium-dam/daily-geo-json`, `/v2/medium-dam/daily/list`,
|
||
`/v2/medium-dam/{id}/detail`, `/v2/summary/summary4dam`,
|
||
`/v2/summary/dam-summary`, `/v2/summary/dam-crisis`
|
||
- **Rainfall**: `/v2/rainfall/{type}`, `/v2/rainfall/{type}/list`,
|
||
`/v2/district-rain/actual-measure`, `/v2/district-rain/forecast`,
|
||
`/v2/district-rain/accumulate`, `/v2/summary/rainfall24h-ranking-province`,
|
||
`/v2/summary/rainfall-24hr-forecast`, `/v2/summary/rainfall-forecast`,
|
||
`/v2/summary/warning-rainfall-24h`, `/v2/summary/warning-rainfall-48h`
|
||
- **Weather / hazards**: `/v2/weather`, `/v2/storm`, `/v2/wave`, `/v2/pm25`,
|
||
`/v2/pm10`, `/v2/flood/flash-flood`, `/v2/flood/flash-flood-alert`,
|
||
`/v2/drought/alert`, `/v2/drought/risk-area/list`,
|
||
`/v2/summary/weather-summary`, `/v2/summary/temperature-forecast`,
|
||
`/v2/summary-area/rainfall`
|
||
- **Time-series / graph** (base `/data/platform/v1/public/`):
|
||
`tele_waterlevel/graph`, `flow/graph`, `latest_waterlevel/forecast/graph`,
|
||
`latest_watertide/forecast/graph`, `dam_pdaily_sum_by_date`,
|
||
`dam_pdaily_sum_by_region_graph`, `dam_rulecurve/graph`,
|
||
`medium_dam/graph_year`, `monthly_rainfall/stations`,
|
||
`monthly_rainfall/anomaly-stations`, `tele_watergate/graph`,
|
||
`salinity_forecast_cpy/graph`, `sea_waterlevel_forecast/graph`,
|
||
`latest_weather_area`, `latest_weather_area_daily`
|
||
|
||
### 2.3 Other HII hosts
|
||
|
||
| Host | What | Access |
|
||
|---|---|---|
|
||
| `https://standard.thaiwater.net` | **Official water-data standard** — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai) | Open, docs site |
|
||
| `https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}` | Daily isohyet rainfall by province | Token in path |
|
||
| `https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif` | Rainfall anomaly & 1–6-month forecast GeoTIFF rasters | Open |
|
||
| `https://data.hii.or.th` | HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate) | Open browsing |
|
||
| `https://tiwrm.hii.or.th` | Legacy reports | Open |
|
||
|
||
Historical bulk telemetry: HII documents a request channel at
|
||
**nhcsoc@hii.or.th**.
|
||
|
||
---
|
||
|
||
## 3. Dams & reservoirs
|
||
|
||
> **Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1** and
|
||
> cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.
|
||
|
||
The predictive upstream reservoir is **Mae Ngat Somboon Chon** (Mae Ngat
|
||
tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the
|
||
Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
|
||
|
||
| Source | What | Access |
|
||
|---|---|---|
|
||
| `https://app.rid.go.th/reservoir/api/dams` | **INGESTED** — daily snapshot of all ~35 large dams (storage/inflow/outflow MCM, % of usable). `POST` with form field `date=YYYY-MM-DD` (empty = today); GET returns 404 "Unknown method." Archive ≥ 2009; `level_msl` (`DMD_Q`) populated in older years only. Mae Ngat = `DAM_ID 200103` — hit 113% usable capacity, ~19 MCM/day inflow, in Oct 2024. Collected daily by `src/rid_reservoir.py` into `rid_dams` + `rid_reservoir_daily`; backfill via `scripts/backfill_rid_reservoir.py` | Open, no auth |
|
||
| `https://lsim.rid.go.th/ForeCast?reservoirid=22` | Mae Ngat daily status/forecast (RID) | **UNREACHABLE — do not plan around it.** Probed 2026-08-13 from a Thai consumer ISP (AIS Fibre, TH) *and* from abroad: DNS resolves (122.154.18.207) but ICMP is 100% loss and ports 80/443/8080 are filtered, while `app.rid.go.th` answers in 0.27 s over the same connection. Down or RID-internal-only — not a geo-block |
|
||
| `https://app.rid.go.th/reservoir/api/dam` | **Per-dam daily series in ONE request** — `GET` with `dam_id=200103&date_start=YYYY-MM-DD&date_end=YYYY-MM-DD&percent=`. Archive to 2009 (scattered single-day gaps). Far cheaper than the per-day `api/dams` loop the backfill used (one request vs ~2,900); prefer it for gap repair and for adding other dams. Sibling `api/damgraph` takes the same params | Open, no auth |
|
||
| `https://bigdata-api.rid.go.th` (SWOC) | **Intraday reservoir state** — RID SWOC telemetry, hourly with an explicit `hourly_time_utc` stamp; includes Mae Ngat (`TUP.16`) reservoir level m MSL and % capacity. **Snapshot-only — no archive**, so it can only be accumulated forward | Open, no auth |
|
||
| ThaiWater `public/waterlevel_load` stations `ridhydro_TUP.16` (at the dam) / `ridhydro_TUP.11` (dam outlet) | Hourly Mae Ngat reservoir level and outlet stage/flow. **Snapshot-only — `waterlevel_graph` returns empty grids for these ids at every era** (verified 2026-08-13). Collected hourly by our HII collector since 2026-08-11; accumulating forward | Open, no auth |
|
||
| ThaiWater `public/waterlevel_graph` station `P.75` (id 3253) | **The practical dam-release signal**: hourly stage+discharge 3.8 km below the Mae Ngat dam, history to 2019. Already ingested as a core RID station and a model feature since v1 | Open, no auth |
|
||
| ThaiWater `public/waterlevel_graph` station `MOU301` "สะพานน้ำแม่งัด" (id 1475118) | 10-minute stage on the Mae Ngat *above* the reservoir (inflow arm). History only from ~mid-2025; level only, no discharge | Open, no auth |
|
||
| ThaiWater `api-v3 .../analyst/dam` (dam.id 53) | EGAT-sourced copy of Mae Ngat carrying reservoir **level in m MSL historically** — the field RID's own API stopped populating (`DMD_Q`) after ~2013. Daily, no observation time | Open, no auth |
|
||
| `https://tiwrm.hii.or.th/DATA/REPORT/php/rid_bigcm_raw.php?sdate=YYYY-MM-DD` | HII HTML mirror of the RID large-dam daily table. Daily and *intermittent* (2026 YTD publishes ~108 of 225 days) — a cross-check, not a primary source | Open, no auth |
|
||
| `https://water.egat.co.th` | EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level | Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol |
|
||
| ThaiWater `/v2/large-dam/*`, `dam_rulecurve/graph` | All large/medium dams incl. hourly | Requires HII API key (§2.2) |
|
||
|
||
---
|
||
|
||
## 4. Rainfall & weather (external)
|
||
|
||
### Near-real-time (usable in the live inference path)
|
||
|
||
| Source | Cadence / latency | Access | Notes |
|
||
|---|---|---|---|
|
||
| HII `rain_24h` (§2.1) | Hourly, near-real-time | Open JSON | Primary rain-gauge feed; persist from now on |
|
||
| GSMaP NRT (JAXA) | Hourly, ~4 h latency, 0.1° | Free JAXA registration (FTP); or Google Earth Engine `JAXA/GPM_L3/GSMaP/v8/operational` (no registration) | Gauge-corrected `hourlyPrecipRateGC`; catchment-average rain where gauges are sparse |
|
||
| NASA IMERG **Early Run** | Half-hourly, ~4 h latency, 0.1° | Free Earthdata login | NASA-stack alternative to GSMaP |
|
||
|
||
### Forecasts (the only way past the ~17 h physical lead-time cap)
|
||
|
||
| Source | What | Access |
|
||
|---|---|---|
|
||
| **Open-Meteo** (<https://open-meteo.com>) — ✅ **INGESTED** since 2026-08-12 (`src/ml/rain.py`): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the `openmeteo_rain` table | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
|
||
| TMD NWP API (`https://data.tmd.go.th/nwpapi/v1/forecast/location/...`) | WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (`/nwpapi/doc/main/`) |
|
||
| GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |
|
||
|
||
### Training-only (too slow for live)
|
||
|
||
| Source | Cadence | Latency |
|
||
|---|---|---|
|
||
| CHIRPS (`data.chc.ucsb.edu/products/CHIRPS-2.0/`) | Daily, 0.05° | ~2 days prelim / 3+ weeks final |
|
||
| IMERG Late / Final | Half-hourly | ~14 h / ~3.5 months |
|
||
| TMD observation API (`data.tmd.go.th/api/index1.php`) | 3-hourly / daily station obs, XML | Free uid+key registration |
|
||
|
||
---
|
||
|
||
## 5. Historical / open-data portals
|
||
|
||
| Portal | Content |
|
||
|---|---|
|
||
| `https://data.hii.or.th` | 36 HII datasets (rainfall telemetry the most viewed) |
|
||
| `https://data.go.th/dataset?organization=rid` | 4 RID datasets (API + ZIP) |
|
||
| `https://gdcatalog.go.th` | Nationwide daily rainfall-station catalogs |
|
||
| `https://hydro-1.net` | RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download |
|
||
| `https://water.rid.go.th/flood/flood/daily.pdf` | RID daily flood bulletin (PDF only) |
|
||
|
||
---
|
||
|
||
## 6. Integration status & recommended order
|
||
|
||
| Source | Status | Action |
|
||
|---|---|---|
|
||
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
|
||
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`/sensors/thaiwater`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
|
||
| HII `rain_24h` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
|
||
| HII `waterlevel_load` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
|
||
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
|
||
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
|
||
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
|
||
| Open-Meteo rain | ✅ Hourly by ping-collector → `openmeteo_rain` (5 catchment points, past 5 days + 48 h forecast); the 2021+ archive trains the hgb-v3 rain features | — |
|
||
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
|
||
|
||
**Collector configuration** (ping-collector): HII, the dams and Open-Meteo are
|
||
fetched once per clock hour, also while RID is being retried; one-shot via
|
||
`ping-collector once`. Env vars: `ENABLE_HII_COLLECTION`,
|
||
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION` (default `true`),
|
||
`HII_BASIN_CODE` (default `6` = Ping). Tables are created automatically
|
||
(PostgreSQL). The archive backfills stay in Python (`scripts/backfill_*.py`,
|
||
using `src/hii_collector.py` and `src/rid_reservoir.py`).
|
||
|
||
**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
|
||
without notice. Poll hourly at most, cache aggressively, and pursue official
|
||
HII access for anything production-critical.
|