Northern-Thailand-Ping-River-Monitor/docs/DATA_SOURCES.md

Data Sources & External API Catalog

Catalog of every data source available to the Ping River Monitor — what we ingest today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for future model inputs (rainfall, dam releases, forecasts).

All "verified" claims below were empirically probed on 2026-08-11. Endpoints marked (catalog) were recovered from the thaiwater.net frontend JS bundle but not exercised (auth required).

---

1. Currently ingested — RID hydrology telemetry

The only source persisted to the database and used by the ML pipeline.

Endpoint	POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx
Agency	Royal Irrigation Department (RID)
Auth	None
Cadence	Hourly (hourlytime 1.00–24.00; hour 24 = midnight next day)
Params	DW[UtokID]=1, DW[BasinID]=6 (Ping), DW[TimeCurrent]=<Buddhist-calendar date>, rows=100
Variables	Water level (m, gauge datum), discharge (m³/s, '***' = malformed), discharge % of channel capacity
Stations	16 P-series gauges (P.1 anchor at Nawarat Bridge; see src/data/stations.json)
Client	src/water_scraper_v3.py

Human-facing page: https://hyd-app-db.rid.go.th/hydro1h.html

---

2. ThaiWater / HII ecosystem

ThaiWater (https://twa.thaiwater.net) is the National Hydroinformatics Institute (HII) portal. It sits on two distinct API layers with very different access rules.

2.1 api-v3.thaiwater.net — open, no authentication ✅

Base: https://api-v3.thaiwater.net/api/v1/thaiwater30/public/

Undocumented backend of the portal. No key, no session. No published rate limits or terms of use — be a good citizen (hourly polls, cache, filter to Ping basin basin_code == 6).

waterlevel_load — national water-level snapshot (verified)

GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load

• ~2.4 MB, 1,426 stations nationwide; 125 in Ping Basin, 61 in Chiang Mai province, 34 with discharge.
• Per station: waterlevel_m, waterlevel_msl, waterlevel_msl_previous, flow_rate, discharge, storage_percent, situation_level (1–4 flood severity), diff_wl_bank, bank/critical levels (min_bank, critical_level_msl, warning_level_m, critical_level_m, qmax), river_name, basin, geocode, agency, lat/long, is_key_station.
• Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A, P.7A, P.77, P.2A — plus RID mirrors (ridhydro_P.1, …), Tak-reach TP./ TUP. codes, and HII sensor clusters (PIN001-011, CHM001-005).
• P.1 = internal station id 3226 (lat 18.786961, long 99.005089).

waterlevel_graph — hourly historical time series (verified) ⭐

GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08

• Returns {data: {graph_data: [{datetime, value, value_out, discharge}]}}, hourly. value is water level in m MSL (not gauge datum).
• Respects arbitrary date ranges. Archive verified back to at least 2019-08 (P.1 returned data for 2019-08-01). 14-day windows tested OK; maximum window size not probed.
• Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00, discharge 656 m³/s.
• Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒ gauge ≈ MSL − 300.5 m (verify per station before use; each station has its own datum offset).
• Value: independent second historical source for cross-validating / gap-filling the RID feed (RID grid is only ~56% filled).

rain_24h — national rainfall snapshot (verified)

GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h

• ~4.5 MB, 4,445 stations; 321 in Chiang Mai province; agencies include HII, DWR, RID.
• Per station: rain_1h, rain_24h (mm), rainfall_datetime, station.id (small int — the graph key), station.tele_station_oldcode (e.g. CHM005, STN0410, ridtele_TUP.14), sub_basin_id, lat/long, basin, geocode, agency.
• This is the missing rainfall input for the flood model — near-real-time hourly gauge rain across the upper Ping catchment.

rain_24h_graph — trailing-window rainfall series (verified, limited) ⚠️

GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...

• station_id is the small station.id from rain_24h (e.g. 418 = CHM005 "Chiang Mai 5", Mae Taeng), not the top-level record id.
• Returns hourly {rainfall_datetime, rainfall_value} — but the date range is IGNORED: every request returns the same trailing ~36-hour window (39 rows). Requests for 2020/2024 return identical data to today.
• Consequence: no rainfall history via this API. To build training data, persist rain_24h from now on and backfill history from satellite QPE or an HII data request (§4).

Probed and NOT available on api-v3 (all HTTP 404)

dam_daily, dam, dam_json, big_dam, mainstream_dam, weather, rain_graph, rainfall_graph, rain24hr_graph. Dam data is v2-only (§2.2).

2.2 twa-api-public.thaiwater.net — auth-gated (x-api-key / session) 🔒

The layer our existing src/thaiwater.py client uses (GET /v2/waterlevel with x-api-key: $THAIWATER_API_KEY; wired to GET /sensors/thaiwater in the web API, display-only, never persisted). Without a key: HTTP 401/500. No public key-registration page was found — obtain a sanctioned key from HII (https://hii.or.th).

Full endpoint catalog (catalog — recovered from frontend JS, not exercised):

• Water level / discharge: /v2/waterlevel, /v2/waterlevel/list, /v2/waterlevel/{id}/detail, /v2/waterlevel/canal, /v2/waterlevel/sea-waterlevel, /v2/waterlevel-discharge, /v2/waterlevel-discharge/list, /v2/waterlevel-discharge/{id}/detail, /v2/waterlevel-discharge/{id}/forecast-table, /v2/waterlevel-discharge/forecast, /v2/waterlevel-discharge/forecast/list, /v2/watergate, /v2/waterload-tide
• Dams (incl. Bhumibol): /v2/large-dam/daily-geo-json, /v2/large-dam/daily/list, /v2/large-dam/hourly/list, /v2/large-dam/daily/{id}/detail, /v2/large-dam/hourly/{id}/detail, /v2/medium-dam/daily-geo-json, /v2/medium-dam/daily/list, /v2/medium-dam/{id}/detail, /v2/summary/summary4dam, /v2/summary/dam-summary, /v2/summary/dam-crisis
• Rainfall: /v2/rainfall/{type}, /v2/rainfall/{type}/list, /v2/district-rain/actual-measure, /v2/district-rain/forecast, /v2/district-rain/accumulate, /v2/summary/rainfall24h-ranking-province, /v2/summary/rainfall-24hr-forecast, /v2/summary/rainfall-forecast, /v2/summary/warning-rainfall-24h, /v2/summary/warning-rainfall-48h
• Weather / hazards: /v2/weather, /v2/storm, /v2/wave, /v2/pm25, /v2/pm10, /v2/flood/flash-flood, /v2/flood/flash-flood-alert, /v2/drought/alert, /v2/drought/risk-area/list, /v2/summary/weather-summary, /v2/summary/temperature-forecast, /v2/summary-area/rainfall
• Time-series / graph (base /data/platform/v1/public/): tele_waterlevel/graph, flow/graph, latest_waterlevel/forecast/graph, latest_watertide/forecast/graph, dam_pdaily_sum_by_date, dam_pdaily_sum_by_region_graph, dam_rulecurve/graph, medium_dam/graph_year, monthly_rainfall/stations, monthly_rainfall/anomaly-stations, tele_watergate/graph, salinity_forecast_cpy/graph, sea_waterlevel_forecast/graph, latest_weather_area, latest_weather_area_daily

2.3 Other HII hosts

Host	What	Access
https://standard.thaiwater.net	Official water-data standard — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai)	Open, docs site
https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}	Daily isohyet rainfall by province	Token in path
https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif	Rainfall anomaly & 1–6-month forecast GeoTIFF rasters	Open
https://data.hii.or.th	HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate)	Open browsing
https://tiwrm.hii.or.th	Legacy reports	Open

Historical bulk telemetry: HII documents a request channel at nhcsoc@hii.or.th.

---

3. Dams & reservoirs

Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1 and cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.

The predictive upstream reservoir is Mae Ngat Somboon Chon (Mae Ngat tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.

Source	What	Access
https://app.rid.go.th/reservoir/api/dams	INGESTED — daily snapshot of all ~35 large dams (storage/inflow/outflow MCM, % of usable). POST with form field date=YYYY-MM-DD (empty = today); GET returns 404 "Unknown method." Archive ≥ 2009; level_msl (DMD_Q) populated in older years only. Mae Ngat = DAM_ID 200103 — hit 113% usable capacity, ~19 MCM/day inflow, in Oct 2024. Collected daily by src/rid_reservoir.py into rid_dams + rid_reservoir_daily; backfill via scripts/backfill_rid_reservoir.py	Open, no auth
https://lsim.rid.go.th/ForeCast?reservoirid=22	Mae Ngat daily status/forecast (RID)	UNREACHABLE — do not plan around it. Probed 2026-08-13 from a Thai consumer ISP (AIS Fibre, TH) and from abroad: DNS resolves (122.154.18.207) but ICMP is 100% loss and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Down or RID-internal-only — not a geo-block
https://app.rid.go.th/reservoir/api/dam	Per-dam daily series in ONE request — GET with dam_id=200103&date_start=YYYY-MM-DD&date_end=YYYY-MM-DD&percent=. Archive to 2009 (scattered single-day gaps). Far cheaper than the per-day api/dams loop the backfill used (one request vs ~2,900); prefer it for gap repair and for adding other dams. Sibling api/damgraph takes the same params	Open, no auth
https://bigdata-api.rid.go.th (SWOC)	Intraday reservoir state — RID SWOC telemetry, hourly with an explicit hourly_time_utc stamp; includes Mae Ngat (TUP.16) reservoir level m MSL and % capacity. Snapshot-only — no archive, so it can only be accumulated forward	Open, no auth
ThaiWater public/waterlevel_load stations ridhydro_TUP.16 (at the dam) / ridhydro_TUP.11 (dam outlet)	Hourly Mae Ngat reservoir level and outlet stage/flow. Snapshot-only — waterlevel_graph returns empty grids for these ids at every era (verified 2026-08-13). Collected hourly by our HII collector since 2026-08-11; accumulating forward	Open, no auth
ThaiWater public/waterlevel_graph station P.75 (id 3253)	The practical dam-release signal: hourly stage+discharge 3.8 km below the Mae Ngat dam, history to 2019. Already ingested as a core RID station and a model feature since v1	Open, no auth
ThaiWater public/waterlevel_graph station MOU301 "สะพานน้ำแม่งัด" (id 1475118)	10-minute stage on the Mae Ngat above the reservoir (inflow arm). History only from ~mid-2025; level only, no discharge	Open, no auth
ThaiWater api-v3 .../analyst/dam (dam.id 53)	EGAT-sourced copy of Mae Ngat carrying reservoir level in m MSL historically — the field RID's own API stopped populating (DMD_Q) after ~2013. Daily, no observation time	Open, no auth
https://tiwrm.hii.or.th/DATA/REPORT/php/rid_bigcm_raw.php?sdate=YYYY-MM-DD	HII HTML mirror of the RID large-dam daily table. Daily and intermittent (2026 YTD publishes ~108 of 225 days) — a cross-check, not a primary source	Open, no auth
https://water.egat.co.th	EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level	Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol
ThaiWater /v2/large-dam/*, dam_rulecurve/graph	All large/medium dams incl. hourly	Requires HII API key (§2.2)

---

4. Rainfall & weather (external)

Near-real-time (usable in the live inference path)

Source	Cadence / latency	Access	Notes
HII rain_24h (§2.1)	Hourly, near-real-time	Open JSON	Primary rain-gauge feed; persist from now on
GSMaP NRT (JAXA)	Hourly, ~4 h latency, 0.1°	Free JAXA registration (FTP); or Google Earth Engine JAXA/GPM_L3/GSMaP/v8/operational (no registration)	Gauge-corrected hourlyPrecipRateGC; catchment-average rain where gauges are sparse
NASA IMERG Early Run	Half-hourly, ~4 h latency, 0.1°	Free Earthdata login	NASA-stack alternative to GSMaP

Forecasts (the only way past the ~17 h physical lead-time cap)

Source	What	Access
Open-Meteo (https://open-meteo.com) — ✅ INGESTED since 2026-08-12 (src/ml/rain.py): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the openmeteo_rain table	Hourly precip forecast ≤16 days, any lat/lon; Historical Forecast API archive from 2021 (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940	Free, no key, 10k calls/day, non-commercial w/ attribution
TMD NWP API (https://data.tmd.go.th/nwpapi/v1/forecast/location/...)	WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily	Free Bearer-token registration (/nwpapi/doc/main/)
GFS / ECMWF IFS open data	0.25° global, 4×/day	Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both

Training-only (too slow for live)

Source	Cadence	Latency
CHIRPS (data.chc.ucsb.edu/products/CHIRPS-2.0/)	Daily, 0.05°	~2 days prelim / 3+ weeks final
IMERG Late / Final	Half-hourly	~14 h / ~3.5 months
TMD observation API (data.tmd.go.th/api/index1.php)	3-hourly / daily station obs, XML	Free uid+key registration

---

5. Historical / open-data portals

Portal	Content
https://data.hii.or.th	36 HII datasets (rainfall telemetry the most viewed)
https://data.go.th/dataset?organization=rid	4 RID datasets (API + ZIP)
https://gdcatalog.go.th	Nationwide daily rainfall-station catalogs
https://hydro-1.net	RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download
https://water.rid.go.th/flood/flood/daily.pdf	RID daily flood bulletin (PDF only)

---

6. Integration status & recommended order

Source	Status	Action
RID hourly gauges	✅ Ingested (hourly → PostgreSQL)	—
ThaiWater /v2/waterlevel	🟡 Display-only (src/thaiwater.py, needs THAIWATER_API_KEY, never persisted)	Optionally persist
HII rain_24h	✅ Ingested hourly via src/hii_collector.py → hii_rain_stations + hii_rainfall (Ping-filtered; ~300 stations)	—
HII waterlevel_load	✅ Ingested hourly via src/hii_collector.py → hii_wl_stations + hii_waterlevel (125 Ping stations, m MSL; rid_code column maps mirrors like ridhydro_P.1 → P.1, offset_msl converts MSL → gauge datum)	—
HII waterlevel_graph	✅ Backfill via scripts/backfill_hii_waterlevel.py → hii_waterlevel (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns)	Run once on the box: python scripts/backfill_hii_waterlevel.py (defaults: 2019-01-01 → today, RID-mirror + key stations; --stations P.1,P.67, --all for every Ping station)
Mae Ngat reservoir (app.rid.go.th)	✅ Ingested (2026-08-13)	Daily storage/inflow/outflow for all large dams → rid_reservoir_daily; candidate model features for next retrain
Satellite QPE (GSMaP/IMERG)	❌	Backfill training rainfall (GEE)
Open-Meteo forecasts	❌	Add forecast features (live + 2021 archive for training)
HII API key (dams, forecasts)	❌	Contact HII for sanctioned access

Collector configuration (src/hii_collector.py): runs automatically every scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry mode) from both --web-api and continuous-monitoring modes; one-shot via python -m src.main --collect-hii. Env vars: ENABLE_HII_COLLECTION (default true), HII_BASIN_CODE (default 6 = Ping). Requires a SQL DB_TYPE (sqlite/postgresql/mysql); tables are created automatically.

Caveats: api-v3 is an undocumented backend — no SLA, no ToS, may change without notice. Poll hourly at most, cache aggressively, and pursue official HII access for anything production-critical.