grabowski
7e642555b4
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
17 KiB
17 KiB
Data Sources & External API Catalog
Catalog of every data source available to the Ping River Monitor — what we ingest today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for future model inputs (rainfall, dam releases, forecasts).
All "verified" claims below were empirically probed on 2026-08-11. Endpoints marked (catalog) were recovered from the thaiwater.net frontend JS bundle but not exercised (auth required).
1. Currently ingested — RID hydrology telemetry
The only source persisted to the database and used by the ML pipeline.
| Endpoint | POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx |
| Agency | Royal Irrigation Department (RID) |
| Auth | None |
| Cadence | Hourly (hourlytime 1.00–24.00; hour 24 = midnight next day) |
| Params | DW[UtokID]=1, DW[BasinID]=6 (Ping), DW[TimeCurrent]=<Buddhist-calendar date>, rows=100 |
| Variables | Water level (m, gauge datum), discharge (m³/s, '***' = malformed), discharge % of channel capacity |
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see src/data/stations.json) |
| Client | rust/collector/src/rid.rs (ping-collector; fill-gaps re-fetches old reports) |
Human-facing page: https://hyd-app-db.rid.go.th/hydro1h.html
2. ThaiWater / HII ecosystem
ThaiWater (https://twa.thaiwater.net) is the National Hydroinformatics Institute (HII) portal. It sits on two distinct API layers with very different access rules.
2.1 api-v3.thaiwater.net — open, no authentication ✅
Base: https://api-v3.thaiwater.net/api/v1/thaiwater30/public/
Undocumented backend of the portal. No key, no session. No published rate
limits or terms of use — be a good citizen (hourly polls, cache, filter to
Ping basin basin_code == 6).
waterlevel_load — national water-level snapshot (verified)
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load
- ~2.4 MB, 1,426 stations nationwide; 125 in Ping Basin, 61 in Chiang Mai province, 34 with discharge.
- Per station:
waterlevel_m,waterlevel_msl,waterlevel_msl_previous,flow_rate,discharge,storage_percent,situation_level(1–4 flood severity),diff_wl_bank, bank/critical levels (min_bank,critical_level_msl,warning_level_m,critical_level_m,qmax),river_name, basin, geocode, agency, lat/long,is_key_station. - Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A,
P.7A, P.77, P.2A — plus RID mirrors (
ridhydro_P.1, …), Tak-reachTP./TUP.codes, and HII sensor clusters (PIN001-011,CHM001-005). - P.1 = internal station id
3226(lat 18.786961, long 99.005089).
waterlevel_graph — hourly historical time series (verified) ⭐
GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08
- Returns
{data: {graph_data: [{datetime, value, value_out, discharge}]}}, hourly.valueis water level in m MSL (not gauge datum). - Respects arbitrary date ranges. Archive verified back to at least 2019-08 (P.1 returned data for 2019-08-01). 14-day windows tested OK; maximum window size not probed.
- Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00, discharge 656 m³/s.
- Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒ gauge ≈ MSL − 300.5 m (verify per station before use; each station has its own datum offset).
- Value: independent second historical source for cross-validating / gap-filling the RID feed (RID grid is only ~56% filled).
rain_24h — national rainfall snapshot (verified)
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h
- ~4.5 MB, 4,445 stations; 321 in Chiang Mai province; agencies include HII, DWR, RID.
- Per station:
rain_1h,rain_24h(mm),rainfall_datetime,station.id(small int — the graph key),station.tele_station_oldcode(e.g.CHM005,STN0410,ridtele_TUP.14),sub_basin_id, lat/long, basin, geocode, agency. - This is the missing rainfall input for the flood model — near-real-time hourly gauge rain across the upper Ping catchment.
rain_24h_graph — trailing-window rainfall series (verified, limited) ⚠️
GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...
station_idis the smallstation.idfromrain_24h(e.g. 418 = CHM005 "Chiang Mai 5", Mae Taeng), not the top-level record id.- Returns hourly
{rainfall_datetime, rainfall_value}— but the date range is IGNORED: every request returns the same trailing ~36-hour window (39 rows). Requests for 2020/2024 return identical data to today. - Consequence: no rainfall history via this API. To build training data,
persist
rain_24hfrom now on and backfill history from satellite QPE or an HII data request (§4).
Probed and NOT available on api-v3 (all HTTP 404)
dam_daily, dam, dam_json, big_dam, mainstream_dam, weather,
rain_graph, rainfall_graph, rain24hr_graph. Dam data is v2-only (§2.2).
2.2 twa-api-public.thaiwater.net — auth-gated (x-api-key / session) 🔒
The layer the website's GET /sensors/thaiwater uses (GET /v2/waterlevel
with an x-api-key header from THAIWATER_API_KEY;
rust/collector/src/web/hii.rs, display-only, never persisted).
Without a key: HTTP 401/500. No public key-registration page was found —
obtain a sanctioned key from HII (https://hii.or.th).
Full endpoint catalog (catalog — recovered from frontend JS, not exercised):
- Water level / discharge:
/v2/waterlevel,/v2/waterlevel/list,/v2/waterlevel/{id}/detail,/v2/waterlevel/canal,/v2/waterlevel/sea-waterlevel,/v2/waterlevel-discharge,/v2/waterlevel-discharge/list,/v2/waterlevel-discharge/{id}/detail,/v2/waterlevel-discharge/{id}/forecast-table,/v2/waterlevel-discharge/forecast,/v2/waterlevel-discharge/forecast/list,/v2/watergate,/v2/waterload-tide - Dams (incl. Bhumibol):
/v2/large-dam/daily-geo-json,/v2/large-dam/daily/list,/v2/large-dam/hourly/list,/v2/large-dam/daily/{id}/detail,/v2/large-dam/hourly/{id}/detail,/v2/medium-dam/daily-geo-json,/v2/medium-dam/daily/list,/v2/medium-dam/{id}/detail,/v2/summary/summary4dam,/v2/summary/dam-summary,/v2/summary/dam-crisis - Rainfall:
/v2/rainfall/{type},/v2/rainfall/{type}/list,/v2/district-rain/actual-measure,/v2/district-rain/forecast,/v2/district-rain/accumulate,/v2/summary/rainfall24h-ranking-province,/v2/summary/rainfall-24hr-forecast,/v2/summary/rainfall-forecast,/v2/summary/warning-rainfall-24h,/v2/summary/warning-rainfall-48h - Weather / hazards:
/v2/weather,/v2/storm,/v2/wave,/v2/pm25,/v2/pm10,/v2/flood/flash-flood,/v2/flood/flash-flood-alert,/v2/drought/alert,/v2/drought/risk-area/list,/v2/summary/weather-summary,/v2/summary/temperature-forecast,/v2/summary-area/rainfall - Time-series / graph (base
/data/platform/v1/public/):tele_waterlevel/graph,flow/graph,latest_waterlevel/forecast/graph,latest_watertide/forecast/graph,dam_pdaily_sum_by_date,dam_pdaily_sum_by_region_graph,dam_rulecurve/graph,medium_dam/graph_year,monthly_rainfall/stations,monthly_rainfall/anomaly-stations,tele_watergate/graph,salinity_forecast_cpy/graph,sea_waterlevel_forecast/graph,latest_weather_area,latest_weather_area_daily
2.3 Other HII hosts
| Host | What | Access |
|---|---|---|
https://standard.thaiwater.net |
Official water-data standard — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai) | Open, docs site |
https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code} |
Daily isohyet rainfall by province | Token in path |
https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif |
Rainfall anomaly & 1–6-month forecast GeoTIFF rasters | Open |
https://data.hii.or.th |
HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate) | Open browsing |
https://tiwrm.hii.or.th |
Legacy reports | Open |
Historical bulk telemetry: HII documents a request channel at nhcsoc@hii.or.th.
3. Dams & reservoirs
Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1 and cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.
The predictive upstream reservoir is Mae Ngat Somboon Chon (Mae Ngat tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
| Source | What | Access |
|---|---|---|
https://app.rid.go.th/reservoir/api/dams |
INGESTED — daily snapshot of all ~35 large dams (storage/inflow/outflow MCM, % of usable). POST with form field date=YYYY-MM-DD (empty = today); GET returns 404 "Unknown method." Archive ≥ 2009; level_msl (DMD_Q) populated in older years only. Mae Ngat = DAM_ID 200103 — hit 113% usable capacity, ~19 MCM/day inflow, in Oct 2024. Collected daily by src/rid_reservoir.py into rid_dams + rid_reservoir_daily; backfill via scripts/backfill_rid_reservoir.py |
Open, no auth |
https://lsim.rid.go.th/ForeCast?reservoirid=22 |
Mae Ngat daily status/forecast (RID) | UNREACHABLE — do not plan around it. Probed 2026-08-13 from a Thai consumer ISP (AIS Fibre, TH) and from abroad: DNS resolves (122.154.18.207) but ICMP is 100% loss and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Down or RID-internal-only — not a geo-block |
https://app.rid.go.th/reservoir/api/dam |
Per-dam daily series in ONE request — GET with dam_id=200103&date_start=YYYY-MM-DD&date_end=YYYY-MM-DD&percent=. Archive to 2009 (scattered single-day gaps). Far cheaper than the per-day api/dams loop the backfill used (one request vs ~2,900); prefer it for gap repair and for adding other dams. Sibling api/damgraph takes the same params |
Open, no auth |
https://bigdata-api.rid.go.th (SWOC) |
Intraday reservoir state — RID SWOC telemetry, hourly with an explicit hourly_time_utc stamp; includes Mae Ngat (TUP.16) reservoir level m MSL and % capacity. Snapshot-only — no archive, so it can only be accumulated forward |
Open, no auth |
ThaiWater public/waterlevel_load stations ridhydro_TUP.16 (at the dam) / ridhydro_TUP.11 (dam outlet) |
Hourly Mae Ngat reservoir level and outlet stage/flow. Snapshot-only — waterlevel_graph returns empty grids for these ids at every era (verified 2026-08-13). Collected hourly by our HII collector since 2026-08-11; accumulating forward |
Open, no auth |
ThaiWater public/waterlevel_graph station P.75 (id 3253) |
The practical dam-release signal: hourly stage+discharge 3.8 km below the Mae Ngat dam, history to 2019. Already ingested as a core RID station and a model feature since v1 | Open, no auth |
ThaiWater public/waterlevel_graph station MOU301 "สะพานน้ำแม่งัด" (id 1475118) |
10-minute stage on the Mae Ngat above the reservoir (inflow arm). History only from ~mid-2025; level only, no discharge | Open, no auth |
ThaiWater api-v3 .../analyst/dam (dam.id 53) |
EGAT-sourced copy of Mae Ngat carrying reservoir level in m MSL historically — the field RID's own API stopped populating (DMD_Q) after ~2013. Daily, no observation time |
Open, no auth |
https://tiwrm.hii.or.th/DATA/REPORT/php/rid_bigcm_raw.php?sdate=YYYY-MM-DD |
HII HTML mirror of the RID large-dam daily table. Daily and intermittent (2026 YTD publishes ~108 of 225 days) — a cross-check, not a primary source | Open, no auth |
https://water.egat.co.th |
EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level | Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol |
ThaiWater /v2/large-dam/*, dam_rulecurve/graph |
All large/medium dams incl. hourly | Requires HII API key (§2.2) |
4. Rainfall & weather (external)
Near-real-time (usable in the live inference path)
| Source | Cadence / latency | Access | Notes |
|---|---|---|---|
HII rain_24h (§2.1) |
Hourly, near-real-time | Open JSON | Primary rain-gauge feed; persist from now on |
| GSMaP NRT (JAXA) | Hourly, ~4 h latency, 0.1° | Free JAXA registration (FTP); or Google Earth Engine JAXA/GPM_L3/GSMaP/v8/operational (no registration) |
Gauge-corrected hourlyPrecipRateGC; catchment-average rain where gauges are sparse |
| NASA IMERG Early Run | Half-hourly, ~4 h latency, 0.1° | Free Earthdata login | NASA-stack alternative to GSMaP |
Forecasts (the only way past the ~17 h physical lead-time cap)
| Source | What | Access |
|---|---|---|
Open-Meteo (https://open-meteo.com) — ✅ INGESTED since 2026-08-12 (src/ml/rain.py): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the openmeteo_rain table |
Hourly precip forecast ≤16 days, any lat/lon; Historical Forecast API archive from 2021 (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
TMD NWP API (https://data.tmd.go.th/nwpapi/v1/forecast/location/...) |
WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (/nwpapi/doc/main/) |
| GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |
Training-only (too slow for live)
| Source | Cadence | Latency |
|---|---|---|
CHIRPS (data.chc.ucsb.edu/products/CHIRPS-2.0/) |
Daily, 0.05° | ~2 days prelim / 3+ weeks final |
| IMERG Late / Final | Half-hourly | ~14 h / ~3.5 months |
TMD observation API (data.tmd.go.th/api/index1.php) |
3-hourly / daily station obs, XML | Free uid+key registration |
5. Historical / open-data portals
| Portal | Content |
|---|---|
https://data.hii.or.th |
36 HII datasets (rainfall telemetry the most viewed) |
https://data.go.th/dataset?organization=rid |
4 RID datasets (API + ZIP) |
https://gdcatalog.go.th |
Nationwide daily rainfall-station catalogs |
https://hydro-1.net |
RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download |
https://water.rid.go.th/flood/flood/daily.pdf |
RID daily flood bulletin (PDF only) |
6. Integration status & recommended order
| Source | Status | Action |
|---|---|---|
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
ThaiWater /v2/waterlevel |
🟡 Display-only (/sensors/thaiwater, needs THAIWATER_API_KEY, never persisted) |
Optionally persist |
HII rain_24h |
✅ Ingested hourly by ping-collector (rust/collector/src/hii.rs) → hii_rain_stations + hii_rainfall (Ping-filtered; ~300 stations) |
— |
HII waterlevel_load |
✅ Ingested hourly by ping-collector (rust/collector/src/hii.rs) → hii_wl_stations + hii_waterlevel (125 Ping stations, m MSL; rid_code column maps mirrors like ridhydro_P.1 → P.1, offset_msl converts MSL → gauge datum) |
— |
HII waterlevel_graph |
✅ Backfill via scripts/backfill_hii_waterlevel.py → hii_waterlevel (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) |
Run once on the box: python scripts/backfill_hii_waterlevel.py (defaults: 2019-01-01 → today, RID-mirror + key stations; --stations P.1,P.67, --all for every Ping station) |
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → rid_reservoir_daily; candidate model features for next retrain |
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
| Open-Meteo rain | ✅ Hourly by ping-collector → openmeteo_rain (5 catchment points, past 5 days + 48 h forecast); the 2021+ archive trains the hgb-v3 rain features |
— |
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
Collector configuration (ping-collector): HII, the dams and Open-Meteo are
fetched once per clock hour, also while RID is being retried; one-shot via
ping-collector once. Env vars: ENABLE_HII_COLLECTION,
ENABLE_RESERVOIR_COLLECTION, ENABLE_RAIN_COLLECTION (default true),
HII_BASIN_CODE (default 6 = Ping). Tables are created automatically
(PostgreSQL). The archive backfills stay in Python (scripts/backfill_*.py,
using src/hii_collector.py and src/rid_reservoir.py).
Caveats: api-v3 is an undocumented backend — no SLA, no ToS, may change
without notice. Poll hourly at most, cache aggressively, and pursue official
HII access for anything production-critical.