Northern-Thailand-Ping-River-Monitor/docs/DATA_SOURCES.md

Data Sources & External API Catalog

Catalog of every data source available to the Ping River Monitor — what we ingest today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for future model inputs (rainfall, dam releases, forecasts).

All "verified" claims below were empirically probed on 2026-08-11. Endpoints marked (catalog) were recovered from the thaiwater.net frontend JS bundle but not exercised (auth required).

---

1. Currently ingested — RID hydrology telemetry

The only source persisted to the database and used by the ML pipeline.

Endpoint	POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx
Agency	Royal Irrigation Department (RID)
Auth	None
Cadence	Hourly (hourlytime 1.00–24.00; hour 24 = midnight next day)
Params	DW[UtokID]=1, DW[BasinID]=6 (Ping), DW[TimeCurrent]=<Buddhist-calendar date>, rows=100
Variables	Water level (m, gauge datum), discharge (m³/s, '***' = malformed), discharge % of channel capacity
Stations	16 P-series gauges (P.1 anchor at Nawarat Bridge; see src/data/stations.json)
Client	src/water_scraper_v3.py

Human-facing page: https://hyd-app-db.rid.go.th/hydro1h.html

---

2. ThaiWater / HII ecosystem

ThaiWater (https://twa.thaiwater.net) is the National Hydroinformatics Institute (HII) portal. It sits on two distinct API layers with very different access rules.

2.1 api-v3.thaiwater.net — open, no authentication ✅

Base: https://api-v3.thaiwater.net/api/v1/thaiwater30/public/

Undocumented backend of the portal. No key, no session. No published rate limits or terms of use — be a good citizen (hourly polls, cache, filter to Ping basin basin_code == 6).

waterlevel_load — national water-level snapshot (verified)

GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load

• ~2.4 MB, 1,426 stations nationwide; 125 in Ping Basin, 61 in Chiang Mai province, 34 with discharge.
• Per station: waterlevel_m, waterlevel_msl, waterlevel_msl_previous, flow_rate, discharge, storage_percent, situation_level (1–4 flood severity), diff_wl_bank, bank/critical levels (min_bank, critical_level_msl, warning_level_m, critical_level_m, qmax), river_name, basin, geocode, agency, lat/long, is_key_station.
• Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A, P.7A, P.77, P.2A — plus RID mirrors (ridhydro_P.1, …), Tak-reach TP./ TUP. codes, and HII sensor clusters (PIN001-011, CHM001-005).
• P.1 = internal station id 3226 (lat 18.786961, long 99.005089).

waterlevel_graph — hourly historical time series (verified) ⭐

GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08

• Returns {data: {graph_data: [{datetime, value, value_out, discharge}]}}, hourly. value is water level in m MSL (not gauge datum).
• Respects arbitrary date ranges. Archive verified back to at least 2019-08 (P.1 returned data for 2019-08-01). 14-day windows tested OK; maximum window size not probed.
• Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00, discharge 656 m³/s.
• Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒ gauge ≈ MSL − 300.5 m (verify per station before use; each station has its own datum offset).
• Value: independent second historical source for cross-validating / gap-filling the RID feed (RID grid is only ~56% filled).

rain_24h — national rainfall snapshot (verified)

GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h

• ~4.5 MB, 4,445 stations; 321 in Chiang Mai province; agencies include HII, DWR, RID.
• Per station: rain_1h, rain_24h (mm), rainfall_datetime, station.id (small int — the graph key), station.tele_station_oldcode (e.g. CHM005, STN0410, ridtele_TUP.14), sub_basin_id, lat/long, basin, geocode, agency.
• This is the missing rainfall input for the flood model — near-real-time hourly gauge rain across the upper Ping catchment.

rain_24h_graph — trailing-window rainfall series (verified, limited) ⚠️

GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...

• station_id is the small station.id from rain_24h (e.g. 418 = CHM005 "Chiang Mai 5", Mae Taeng), not the top-level record id.
• Returns hourly {rainfall_datetime, rainfall_value} — but the date range is IGNORED: every request returns the same trailing ~36-hour window (39 rows). Requests for 2020/2024 return identical data to today.
• Consequence: no rainfall history via this API. To build training data, persist rain_24h from now on and backfill history from satellite QPE or an HII data request (§4).

Probed and NOT available on api-v3 (all HTTP 404)

dam_daily, dam, dam_json, big_dam, mainstream_dam, weather, rain_graph, rainfall_graph, rain24hr_graph. Dam data is v2-only (§2.2).

2.2 twa-api-public.thaiwater.net — auth-gated (x-api-key / session) 🔒

The layer our existing src/thaiwater.py client uses (GET /v2/waterlevel with x-api-key: $THAIWATER_API_KEY; wired to GET /sensors/thaiwater in the web API, display-only, never persisted). Without a key: HTTP 401/500. No public key-registration page was found — obtain a sanctioned key from HII (https://hii.or.th).

Full endpoint catalog (catalog — recovered from frontend JS, not exercised):

• Water level / discharge: /v2/waterlevel, /v2/waterlevel/list, /v2/waterlevel/{id}/detail, /v2/waterlevel/canal, /v2/waterlevel/sea-waterlevel, /v2/waterlevel-discharge, /v2/waterlevel-discharge/list, /v2/waterlevel-discharge/{id}/detail, /v2/waterlevel-discharge/{id}/forecast-table, /v2/waterlevel-discharge/forecast, /v2/waterlevel-discharge/forecast/list, /v2/watergate, /v2/waterload-tide
• Dams (incl. Bhumibol): /v2/large-dam/daily-geo-json, /v2/large-dam/daily/list, /v2/large-dam/hourly/list, /v2/large-dam/daily/{id}/detail, /v2/large-dam/hourly/{id}/detail, /v2/medium-dam/daily-geo-json, /v2/medium-dam/daily/list, /v2/medium-dam/{id}/detail, /v2/summary/summary4dam, /v2/summary/dam-summary, /v2/summary/dam-crisis
• Rainfall: /v2/rainfall/{type}, /v2/rainfall/{type}/list, /v2/district-rain/actual-measure, /v2/district-rain/forecast, /v2/district-rain/accumulate, /v2/summary/rainfall24h-ranking-province, /v2/summary/rainfall-24hr-forecast, /v2/summary/rainfall-forecast, /v2/summary/warning-rainfall-24h, /v2/summary/warning-rainfall-48h
• Weather / hazards: /v2/weather, /v2/storm, /v2/wave, /v2/pm25, /v2/pm10, /v2/flood/flash-flood, /v2/flood/flash-flood-alert, /v2/drought/alert, /v2/drought/risk-area/list, /v2/summary/weather-summary, /v2/summary/temperature-forecast, /v2/summary-area/rainfall
• Time-series / graph (base /data/platform/v1/public/): tele_waterlevel/graph, flow/graph, latest_waterlevel/forecast/graph, latest_watertide/forecast/graph, dam_pdaily_sum_by_date, dam_pdaily_sum_by_region_graph, dam_rulecurve/graph, medium_dam/graph_year, monthly_rainfall/stations, monthly_rainfall/anomaly-stations, tele_watergate/graph, salinity_forecast_cpy/graph, sea_waterlevel_forecast/graph, latest_weather_area, latest_weather_area_daily

2.3 Other HII hosts

Host	What	Access
https://standard.thaiwater.net	Official water-data standard — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai)	Open, docs site
https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}	Daily isohyet rainfall by province	Token in path
https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif	Rainfall anomaly & 1–6-month forecast GeoTIFF rasters	Open
https://data.hii.or.th	HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate)	Open browsing
https://tiwrm.hii.or.th	Legacy reports	Open

Historical bulk telemetry: HII documents a request channel at nhcsoc@hii.or.th.

---

3. Dams & reservoirs

Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1 and cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.

The predictive upstream reservoir is Mae Ngat Somboon Chon (Mae Ngat tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.

Source	What	Access
https://lsim.rid.go.th/ForeCast?reservoirid=22	Mae Ngat daily status/forecast (RID)	Open, scrape
https://app.rid.go.th/reservoir/	RID reservoir DB, per-reservoir daily detail with date-range URLs	Open, scrape (confirm URL pattern before hard-coding)
https://water.egat.co.th	EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level	Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol
ThaiWater /v2/large-dam/*, dam_rulecurve/graph	All large/medium dams incl. hourly	Requires HII API key (§2.2)

---

4. Rainfall & weather (external)

Near-real-time (usable in the live inference path)

Source	Cadence / latency	Access	Notes
HII rain_24h (§2.1)	Hourly, near-real-time	Open JSON	Primary rain-gauge feed; persist from now on
GSMaP NRT (JAXA)	Hourly, ~4 h latency, 0.1°	Free JAXA registration (FTP); or Google Earth Engine JAXA/GPM_L3/GSMaP/v8/operational (no registration)	Gauge-corrected hourlyPrecipRateGC; catchment-average rain where gauges are sparse
NASA IMERG Early Run	Half-hourly, ~4 h latency, 0.1°	Free Earthdata login	NASA-stack alternative to GSMaP

Forecasts (the only way past the ~17 h physical lead-time cap)

Source	What	Access
Open-Meteo (https://open-meteo.com)	Hourly precip forecast ≤16 days, any lat/lon; Historical Forecast API archive from 2021 (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940	Free, no key, 10k calls/day, non-commercial w/ attribution
TMD NWP API (https://data.tmd.go.th/nwpapi/v1/forecast/location/...)	WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily	Free Bearer-token registration (/nwpapi/doc/main/)
GFS / ECMWF IFS open data	0.25° global, 4×/day	Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both

Training-only (too slow for live)

Source	Cadence	Latency
CHIRPS (data.chc.ucsb.edu/products/CHIRPS-2.0/)	Daily, 0.05°	~2 days prelim / 3+ weeks final
IMERG Late / Final	Half-hourly	~14 h / ~3.5 months
TMD observation API (data.tmd.go.th/api/index1.php)	3-hourly / daily station obs, XML	Free uid+key registration

---

5. Historical / open-data portals

Portal	Content
https://data.hii.or.th	36 HII datasets (rainfall telemetry the most viewed)
https://data.go.th/dataset?organization=rid	4 RID datasets (API + ZIP)
https://gdcatalog.go.th	Nationwide daily rainfall-station catalogs
https://hydro-1.net	RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download
https://water.rid.go.th/flood/flood/daily.pdf	RID daily flood bulletin (PDF only)

---

6. Integration status & recommended order

Source	Status	Action
RID hourly gauges	✅ Ingested (hourly → PostgreSQL)	—
ThaiWater /v2/waterlevel	🟡 Display-only (src/thaiwater.py, needs THAIWATER_API_KEY, never persisted)	Optionally persist
HII rain_24h	✅ Ingested hourly via src/hii_collector.py → hii_rain_stations + hii_rainfall (Ping-filtered; ~300 stations)	—
HII waterlevel_load	✅ Ingested hourly via src/hii_collector.py → hii_wl_stations + hii_waterlevel (125 Ping stations, m MSL; rid_code column maps mirrors like ridhydro_P.1 → P.1, offset_msl converts MSL → gauge datum)	—
HII waterlevel_graph	❌ Not ingested	Backfill script to cross-validate / gap-fill RID history (≥2019)
Mae Ngat reservoir (lsim.rid.go.th)	❌ Not ingested	Add daily storage/release scrape
Satellite QPE (GSMaP/IMERG)	❌	Backfill training rainfall (GEE)
Open-Meteo forecasts	❌	Add forecast features (live + 2021 archive for training)
HII API key (dams, forecasts)	❌	Contact HII for sanctioned access

Collector configuration (src/hii_collector.py): runs automatically every scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry mode) from both --web-api and continuous-monitoring modes; one-shot via python -m src.main --collect-hii. Env vars: ENABLE_HII_COLLECTION (default true), HII_BASIN_CODE (default 6 = Ping). Requires a SQL DB_TYPE (sqlite/postgresql/mysql); tables are created automatically.

Caveats: api-v3 is an undocumented backend — no SLA, no ToS, may change without notice. Poll hourly at most, cache aggressively, and pursue official HII access for anything production-critical.