Northern-Thailand-Ping-River-Monitor/docs/DATA_SOURCES.md

# Data Sources & External API Catalog

Catalog of every data source available to the Ping River Monitor — what we ingest
today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for
future model inputs (rainfall, dam releases, forecasts).

All "verified" claims below were empirically probed on **2026-08-11**. Endpoints
marked *(catalog)* were recovered from the thaiwater.net frontend JS bundle but
not exercised (auth required).

---

## 1. Currently ingested — RID hydrology telemetry

The only source persisted to the database and used by the ML pipeline.

| | |
|---|---|
| Endpoint | `POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx` |
| Agency | Royal Irrigation Department (RID) |
| Auth | None |
| Cadence | Hourly (`hourlytime` 1.00–24.00; hour 24 = midnight next day) |
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
| Client | `src/water_scraper_v3.py` |

Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>

---

## 2. ThaiWater / HII ecosystem

ThaiWater (<https://twa.thaiwater.net>) is the National Hydroinformatics
Institute (HII) portal. It sits on **two distinct API layers** with very
different access rules.

### 2.1 `api-v3.thaiwater.net` — open, no authentication ✅

Base: `https://api-v3.thaiwater.net/api/v1/thaiwater30/public/`

Undocumented backend of the portal. No key, no session. No published rate
limits or terms of use — be a good citizen (hourly polls, cache, filter to
Ping basin `basin_code == 6`).

#### `waterlevel_load` — national water-level snapshot (verified)

```
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load
```

- ~2.4 MB, 1,426 stations nationwide; **125 in Ping Basin, 61 in Chiang Mai
  province, 34 with discharge**.
- Per station: `waterlevel_m`, `waterlevel_msl`, `waterlevel_msl_previous`,
  `flow_rate`, `discharge`, `storage_percent`, `situation_level` (1–4 flood
  severity), `diff_wl_bank`, bank/critical levels (`min_bank`,
  `critical_level_msl`, `warning_level_m`, `critical_level_m`, `qmax`),
  `river_name`, basin, geocode, agency, lat/long, `is_key_station`.
- Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A,
  P.7A, P.77, P.2A — plus RID mirrors (`ridhydro_P.1`, …), Tak-reach `TP.`/
  `TUP.` codes, and HII sensor clusters (`PIN001-011`, `CHM001-005`).
- **P.1 = internal station id `3226`** (lat 18.786961, long 99.005089).

#### `waterlevel_graph` — hourly historical time series (verified) ⭐

```
GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08
```

- Returns `{data: {graph_data: [{datetime, value, value_out, discharge}]}}`,
  hourly. `value` is **water level in m MSL** (not gauge datum).
- **Respects arbitrary date ranges. Archive verified back to at least
  2019-08** (P.1 returned data for 2019-08-01). 14-day windows tested OK;
  maximum window size not probed.
- Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00,
  discharge 656 m³/s.
- Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒
  **gauge ≈ MSL − 300.5 m** (verify per station before use; each station has
  its own datum offset).
- Value: independent second historical source for cross-validating / gap-filling
  the RID feed (RID grid is only ~56% filled).

#### `rain_24h` — national rainfall snapshot (verified)

```
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h
```

- ~4.5 MB, 4,445 stations; **321 in Chiang Mai province**; agencies include
  HII, DWR, RID.
- Per station: `rain_1h`, `rain_24h` (mm), `rainfall_datetime`,
  `station.id` (small int — the graph key), `station.tele_station_oldcode`
  (e.g. `CHM005`, `STN0410`, `ridtele_TUP.14`), `sub_basin_id`, lat/long,
  basin, geocode, agency.
- **This is the missing rainfall input** for the flood model — near-real-time
  hourly gauge rain across the upper Ping catchment.

#### `rain_24h_graph` — trailing-window rainfall series (verified, limited) ⚠️

```
GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...
```

- `station_id` is the **small `station.id`** from `rain_24h` (e.g. 418 =
  CHM005 "Chiang Mai 5", Mae Taeng), *not* the top-level record id.
- Returns hourly `{rainfall_datetime, rainfall_value}` — **but the date range
  is IGNORED**: every request returns the same trailing ~36-hour window
  (39 rows). Requests for 2020/2024 return identical data to today.
- Consequence: **no rainfall history via this API**. To build training data,
  persist `rain_24h` from now on and backfill history from satellite QPE or an
  HII data request (§4).

#### Probed and NOT available on api-v3 (all HTTP 404)

`dam_daily`, `dam`, `dam_json`, `big_dam`, `mainstream_dam`, `weather`,
`rain_graph`, `rainfall_graph`, `rain24hr_graph`. Dam data is v2-only (§2.2).

### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒

The layer our existing `src/thaiwater.py` client uses
(`GET /v2/waterlevel` with `x-api-key: $THAIWATER_API_KEY`; wired to
`GET /sensors/thaiwater` in the web API, display-only, never persisted).
Without a key: HTTP 401/500. No public key-registration page was found —
obtain a sanctioned key from HII (<https://hii.or.th>).

Full endpoint catalog *(catalog — recovered from frontend JS, not exercised)*:

- **Water level / discharge**: `/v2/waterlevel`, `/v2/waterlevel/list`,
  `/v2/waterlevel/{id}/detail`, `/v2/waterlevel/canal`,
  `/v2/waterlevel/sea-waterlevel`, `/v2/waterlevel-discharge`,
  `/v2/waterlevel-discharge/list`, `/v2/waterlevel-discharge/{id}/detail`,
  `/v2/waterlevel-discharge/{id}/forecast-table`,
  `/v2/waterlevel-discharge/forecast`, `/v2/waterlevel-discharge/forecast/list`,
  `/v2/watergate`, `/v2/waterload-tide`
- **Dams** (incl. Bhumibol): `/v2/large-dam/daily-geo-json`,
  `/v2/large-dam/daily/list`, `/v2/large-dam/hourly/list`,
  `/v2/large-dam/daily/{id}/detail`, `/v2/large-dam/hourly/{id}/detail`,
  `/v2/medium-dam/daily-geo-json`, `/v2/medium-dam/daily/list`,
  `/v2/medium-dam/{id}/detail`, `/v2/summary/summary4dam`,
  `/v2/summary/dam-summary`, `/v2/summary/dam-crisis`
- **Rainfall**: `/v2/rainfall/{type}`, `/v2/rainfall/{type}/list`,
  `/v2/district-rain/actual-measure`, `/v2/district-rain/forecast`,
  `/v2/district-rain/accumulate`, `/v2/summary/rainfall24h-ranking-province`,
  `/v2/summary/rainfall-24hr-forecast`, `/v2/summary/rainfall-forecast`,
  `/v2/summary/warning-rainfall-24h`, `/v2/summary/warning-rainfall-48h`
- **Weather / hazards**: `/v2/weather`, `/v2/storm`, `/v2/wave`, `/v2/pm25`,
  `/v2/pm10`, `/v2/flood/flash-flood`, `/v2/flood/flash-flood-alert`,
  `/v2/drought/alert`, `/v2/drought/risk-area/list`,
  `/v2/summary/weather-summary`, `/v2/summary/temperature-forecast`,
  `/v2/summary-area/rainfall`
- **Time-series / graph** (base `/data/platform/v1/public/`):
  `tele_waterlevel/graph`, `flow/graph`, `latest_waterlevel/forecast/graph`,
  `latest_watertide/forecast/graph`, `dam_pdaily_sum_by_date`,
  `dam_pdaily_sum_by_region_graph`, `dam_rulecurve/graph`,
  `medium_dam/graph_year`, `monthly_rainfall/stations`,
  `monthly_rainfall/anomaly-stations`, `tele_watergate/graph`,
  `salinity_forecast_cpy/graph`, `sea_waterlevel_forecast/graph`,
  `latest_weather_area`, `latest_weather_area_daily`

### 2.3 Other HII hosts

| Host | What | Access |
|---|---|---|
| `https://standard.thaiwater.net` | **Official water-data standard** — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai) | Open, docs site |
| `https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}` | Daily isohyet rainfall by province | Token in path |
| `https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif` | Rainfall anomaly & 1–6-month forecast GeoTIFF rasters | Open |
| `https://data.hii.or.th` | HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate) | Open browsing |
| `https://tiwrm.hii.or.th` | Legacy reports | Open |

Historical bulk telemetry: HII documents a request channel at
**nhcsoc@hii.or.th**.

---

## 3. Dams & reservoirs

> **Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1** and
> cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.

The predictive upstream reservoir is **Mae Ngat Somboon Chon** (Mae Ngat
tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the
Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.

| Source | What | Access |
|---|---|---|
| `https://app.rid.go.th/reservoir/api/dams` | **INGESTED** — daily snapshot of all ~35 large dams (storage/inflow/outflow MCM, % of usable). `POST` with form field `date=YYYY-MM-DD` (empty = today); GET returns 404 "Unknown method." Archive ≥ 2009; `level_msl` (`DMD_Q`) populated in older years only. Mae Ngat = `DAM_ID 200103` — hit 113% usable capacity, ~19 MCM/day inflow, in Oct 2024. Collected daily by `src/rid_reservoir.py` into `rid_dams` + `rid_reservoir_daily`; backfill via `scripts/backfill_rid_reservoir.py` | Open, no auth |
| `https://lsim.rid.go.th/ForeCast?reservoirid=22` | Mae Ngat daily status/forecast (RID) | **UNREACHABLE — do not plan around it.** Probed 2026-08-13 from a Thai consumer ISP (AIS Fibre, TH) *and* from abroad: DNS resolves (122.154.18.207) but ICMP is 100% loss and ports 80/443/8080 are filtered, while `app.rid.go.th` answers in 0.27 s over the same connection. Down or RID-internal-only — not a geo-block |
| `https://app.rid.go.th/reservoir/api/dam` | **Per-dam daily series in ONE request** — `GET` with `dam_id=200103&date_start=YYYY-MM-DD&date_end=YYYY-MM-DD&percent=`. Archive to 2009 (scattered single-day gaps). Far cheaper than the per-day `api/dams` loop the backfill used (one request vs ~2,900); prefer it for gap repair and for adding other dams. Sibling `api/damgraph` takes the same params | Open, no auth |
| `https://bigdata-api.rid.go.th` (SWOC) | **Intraday reservoir state** — RID SWOC telemetry, hourly with an explicit `hourly_time_utc` stamp; includes Mae Ngat (`TUP.16`) reservoir level m MSL and % capacity. **Snapshot-only — no archive**, so it can only be accumulated forward | Open, no auth |
| ThaiWater `public/waterlevel_load` stations `ridhydro_TUP.16` (at the dam) / `ridhydro_TUP.11` (dam outlet) | Hourly Mae Ngat reservoir level and outlet stage/flow. **Snapshot-only — `waterlevel_graph` returns empty grids for these ids at every era** (verified 2026-08-13). Collected hourly by our HII collector since 2026-08-11; accumulating forward | Open, no auth |
| ThaiWater `public/waterlevel_graph` station `P.75` (id 3253) | **The practical dam-release signal**: hourly stage+discharge 3.8 km below the Mae Ngat dam, history to 2019. Already ingested as a core RID station and a model feature since v1 | Open, no auth |
| ThaiWater `public/waterlevel_graph` station `MOU301` "สะพานน้ำแม่งัด" (id 1475118) | 10-minute stage on the Mae Ngat *above* the reservoir (inflow arm). History only from ~mid-2025; level only, no discharge | Open, no auth |
| ThaiWater `api-v3 .../analyst/dam` (dam.id 53) | EGAT-sourced copy of Mae Ngat carrying reservoir **level in m MSL historically** — the field RID's own API stopped populating (`DMD_Q`) after ~2013. Daily, no observation time | Open, no auth |
| `https://tiwrm.hii.or.th/DATA/REPORT/php/rid_bigcm_raw.php?sdate=YYYY-MM-DD` | HII HTML mirror of the RID large-dam daily table. Daily and *intermittent* (2026 YTD publishes ~108 of 225 days) — a cross-check, not a primary source | Open, no auth |
| `https://water.egat.co.th` | EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level | Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol |
| ThaiWater `/v2/large-dam/*`, `dam_rulecurve/graph` | All large/medium dams incl. hourly | Requires HII API key (§2.2) |

---

## 4. Rainfall & weather (external)

### Near-real-time (usable in the live inference path)

| Source | Cadence / latency | Access | Notes |
|---|---|---|---|
| HII `rain_24h` (§2.1) | Hourly, near-real-time | Open JSON | Primary rain-gauge feed; persist from now on |
| GSMaP NRT (JAXA) | Hourly, ~4 h latency, 0.1° | Free JAXA registration (FTP); or Google Earth Engine `JAXA/GPM_L3/GSMaP/v8/operational` (no registration) | Gauge-corrected `hourlyPrecipRateGC`; catchment-average rain where gauges are sparse |
| NASA IMERG **Early Run** | Half-hourly, ~4 h latency, 0.1° | Free Earthdata login | NASA-stack alternative to GSMaP |

### Forecasts (the only way past the ~17 h physical lead-time cap)

| Source | What | Access |
|---|---|---|
| **Open-Meteo** (<https://open-meteo.com>) — ✅ **INGESTED** since 2026-08-12 (`src/ml/rain.py`): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the `openmeteo_rain` table | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
| TMD NWP API (`https://data.tmd.go.th/nwpapi/v1/forecast/location/...`) | WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (`/nwpapi/doc/main/`) |
| GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |

### Training-only (too slow for live)

| Source | Cadence | Latency |
|---|---|---|
| CHIRPS (`data.chc.ucsb.edu/products/CHIRPS-2.0/`) | Daily, 0.05° | ~2 days prelim / 3+ weeks final |
| IMERG Late / Final | Half-hourly | ~14 h / ~3.5 months |
| TMD observation API (`data.tmd.go.th/api/index1.php`) | 3-hourly / daily station obs, XML | Free uid+key registration |

---

## 5. Historical / open-data portals

| Portal | Content |
|---|---|
| `https://data.hii.or.th` | 36 HII datasets (rainfall telemetry the most viewed) |
| `https://data.go.th/dataset?organization=rid` | 4 RID datasets (API + ZIP) |
| `https://gdcatalog.go.th` | Nationwide daily rainfall-station catalogs |
| `https://hydro-1.net` | RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download |
| `https://water.rid.go.th/flood/flood/daily.pdf` | RID daily flood bulletin (PDF only) |

---

## 6. Integration status & recommended order

| Source | Status | Action |
|---|---|---|
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
| Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |

**Collector configuration** (`src/hii_collector.py`): runs automatically every
scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry
mode) from both `--web-api` and continuous-monitoring modes; one-shot via
`python -m src.main --collect-hii`. Env vars: `ENABLE_HII_COLLECTION`
(default `true`), `HII_BASIN_CODE` (default `6` = Ping). Requires a SQL
`DB_TYPE` (sqlite/postgresql/mysql); tables are created automatically.

**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
without notice. Poll hourly at most, cache aggressively, and pursue official
HII access for anything production-critical.