# Data Sources & External API Catalog

Catalog of every data source available to the Ping River Monitor — what we ingest
today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for
future model inputs (rainfall, dam releases, forecasts).

All "verified" claims below were empirically probed on **2026-08-11**. Endpoints
marked *(catalog)* were recovered from the thaiwater.net frontend JS bundle but
not exercised (auth required).

---

## 1. Currently ingested — RID hydrology telemetry

The only source persisted to the database and used by the ML pipeline.

| | |
|---|---|
| Endpoint | `POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx` |
| Agency | Royal Irrigation Department (RID) |
| Auth | None |
| Cadence | Hourly (`hourlytime` 1.00–24.00; hour 24 = midnight next day) |
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports with holes, `refetch FROM [TO]` any reports) |

Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>

**Spikes.** The report sometimes carries a wrong value for one hour and corrects
it later: a slipped decimal point (P.1 10.6 m between 1.04 and 1.07 on 2026-10-25
00:00, later 1.06; P.85 0.101 for 1.1), a flipped sign (P.4A ±1.18, P.67 ±0.6) or
a placeholder (−5, −6, −9, 50 m). The collector re-fetches a report only while it
has holes, so a later correction never arrived. Since 2026-10-26 every batch is
screened before it is stored (`rid::judge`, `rid::screen`): a reading that leaves
the previous one (within 3 h) by more than 1 m (2 m on the flashy tributaries
P.76, P.77, P.81, P.82, P.84, P.85, P.87) while the next reading comes back near
the previous level is not stored, and a stored copy is deleted. A rising or falling
limb, or a step the next reading confirms, is kept. The newest reading of a gauge
waits for the next one when it jumps more than twice the limit. On the 2018–2026
history the rule finds 16 spikes and would have rejected no genuine change (the
largest genuine 1-h changes: 1.1 m on the mainstem, 3.1 m at P.76). Two wrong hours
in a row are not caught.

Evening hours often arrive level-only (no discharge) and gain the discharge
later; the API's history fills a missing discharge with 20 × level (an old
placeholder, kept for compatibility). `refetch` brings in RID's discharge.

---

## 2. ThaiWater / HII ecosystem

ThaiWater (<https://twa.thaiwater.net>) is the National Hydroinformatics
Institute (HII) portal. It sits on **two distinct API layers** with very
different access rules.

### 2.1 `api-v3.thaiwater.net` — open, no authentication ✅

Base: `https://api-v3.thaiwater.net/api/v1/thaiwater30/public/`

Undocumented backend of the portal. No key, no session. No published rate
limits or terms of use — be a good citizen (hourly polls, cache, filter to
Ping basin `basin_code == 6`).

#### `waterlevel_load` — national water-level snapshot (verified)

```
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load
```

- ~2.4 MB, 1,426 stations nationwide; **125 in Ping Basin, 61 in Chiang Mai
  province, 34 with discharge**.
- Per station: `waterlevel_m`, `waterlevel_msl`, `waterlevel_msl_previous`,
  `flow_rate`, `discharge`, `storage_percent`, `situation_level` (1–4 flood
  severity), `diff_wl_bank`, bank/critical levels (`min_bank`,
  `critical_level_msl`, `warning_level_m`, `critical_level_m`, `qmax`),
  `river_name`, basin, geocode, agency, lat/long, `is_key_station`.
- Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A,
  P.7A, P.77, P.2A — plus RID mirrors (`ridhydro_P.1`, …), Tak-reach `TP.`/
  `TUP.` codes, and HII sensor clusters (`PIN001-011`, `CHM001-005`).
- **P.1 = internal station id `3226`** (lat 18.786961, long 99.005089).

#### `waterlevel_graph` — hourly historical time series (verified) ⭐

```
GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08
```

- Returns `{data: {graph_data: [{datetime, value, value_out, discharge}]}}`,
  hourly. `value` is **water level in m MSL** (not gauge datum).
- **Respects arbitrary date ranges. Archive verified back to at least
  2019-08** (P.1 returned data for 2019-08-01). 14-day windows tested OK;
  maximum window size not probed.
- Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00,
  discharge 656 m³/s.
- Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒
  **gauge ≈ MSL − 300.5 m** (verify per station before use; each station has
  its own datum offset).
- Value: independent second historical source for cross-validating / gap-filling
  the RID feed (RID grid is only ~56% filled).

#### `rain_24h` — national rainfall snapshot (verified)

```
GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h
```

- ~4.5 MB, 4,445 stations; **321 in Chiang Mai province**; agencies include
  HII, DWR, RID.
- Per station: `rain_1h`, `rain_24h` (mm), `rainfall_datetime`,
  `station.id` (small int — the graph key), `station.tele_station_oldcode`
  (e.g. `CHM005`, `STN0410`, `ridtele_TUP.14`), `sub_basin_id`, lat/long,
  basin, geocode, agency.
- **This is the missing rainfall input** for the flood model — near-real-time
  hourly gauge rain across the upper Ping catchment.

#### `rain_24h_graph` — trailing-window rainfall series (verified, limited) ⚠️

```
GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...
```

- `station_id` is the **small `station.id`** from `rain_24h` (e.g. 418 =
  CHM005 "Chiang Mai 5", Mae Taeng), *not* the top-level record id.
- Returns hourly `{rainfall_datetime, rainfall_value}` — **but the date range
  is IGNORED**: every request returns the same trailing ~36-hour window
  (39 rows). Requests for 2020/2024 return identical data to today.
- Consequence: **no rainfall history via this API**. To build training data,
  persist `rain_24h` from now on and backfill history from satellite QPE or an
  HII data request (§4).

#### Probed and NOT available on api-v3 (all HTTP 404)

`dam_daily`, `dam`, `dam_json`, `big_dam`, `mainstream_dam`, `weather`,
`rain_graph`, `rainfall_graph`, `rain24hr_graph`. Dam data is v2-only (§2.2).

### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒

The layer the website's `GET /sensors/thaiwater` uses (`GET /v2/waterlevel`
with an `x-api-key` header from `THAIWATER_API_KEY`;
`rust/collector/src/web/hii.rs`, display-only, never persisted).
Without a key: HTTP 401/500. No public key-registration page was found —
obtain a sanctioned key from HII (<https://hii.or.th>).

Full endpoint catalog *(catalog — recovered from frontend JS, not exercised)*:

- **Water level / discharge**: `/v2/waterlevel`, `/v2/waterlevel/list`,
  `/v2/waterlevel/{id}/detail`, `/v2/waterlevel/canal`,
  `/v2/waterlevel/sea-waterlevel`, `/v2/waterlevel-discharge`,
  `/v2/waterlevel-discharge/list`, `/v2/waterlevel-discharge/{id}/detail`,
  `/v2/waterlevel-discharge/{id}/forecast-table`,
  `/v2/waterlevel-discharge/forecast`, `/v2/waterlevel-discharge/forecast/list`,
  `/v2/watergate`, `/v2/waterload-tide`
- **Dams** (incl. Bhumibol): `/v2/large-dam/daily-geo-json`,
  `/v2/large-dam/daily/list`, `/v2/large-dam/hourly/list`,
  `/v2/large-dam/daily/{id}/detail`, `/v2/large-dam/hourly/{id}/detail`,
  `/v2/medium-dam/daily-geo-json`, `/v2/medium-dam/daily/list`,
  `/v2/medium-dam/{id}/detail`, `/v2/summary/summary4dam`,
  `/v2/summary/dam-summary`, `/v2/summary/dam-crisis`
- **Rainfall**: `/v2/rainfall/{type}`, `/v2/rainfall/{type}/list`,
  `/v2/district-rain/actual-measure`, `/v2/district-rain/forecast`,
  `/v2/district-rain/accumulate`, `/v2/summary/rainfall24h-ranking-province`,
  `/v2/summary/rainfall-24hr-forecast`, `/v2/summary/rainfall-forecast`,
  `/v2/summary/warning-rainfall-24h`, `/v2/summary/warning-rainfall-48h`
- **Weather / hazards**: `/v2/weather`, `/v2/storm`, `/v2/wave`, `/v2/pm25`,
  `/v2/pm10`, `/v2/flood/flash-flood`, `/v2/flood/flash-flood-alert`,
  `/v2/drought/alert`, `/v2/drought/risk-area/list`,
  `/v2/summary/weather-summary`, `/v2/summary/temperature-forecast`,
  `/v2/summary-area/rainfall`
- **Time-series / graph** (base `/data/platform/v1/public/`):
  `tele_waterlevel/graph`, `flow/graph`, `latest_waterlevel/forecast/graph`,
  `latest_watertide/forecast/graph`, `dam_pdaily_sum_by_date`,
  `dam_pdaily_sum_by_region_graph`, `dam_rulecurve/graph`,
  `medium_dam/graph_year`, `monthly_rainfall/stations`,
  `monthly_rainfall/anomaly-stations`, `tele_watergate/graph`,
  `salinity_forecast_cpy/graph`, `sea_waterlevel_forecast/graph`,
  `latest_weather_area`, `latest_weather_area_daily`

### 2.3 Other HII hosts

| Host | What | Access |
|---|---|---|
| `https://standard.thaiwater.net` | **Official water-data standard** — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai) | Open, docs site |
| `https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}` | Daily isohyet rainfall by province | Token in path |
| `https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif` | Rainfall anomaly & 1–6-month forecast GeoTIFF rasters | Open |
| `https://data.hii.or.th` | HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate) | Open browsing |
| `https://tiwrm.hii.or.th` | Legacy reports | Open |

Historical bulk telemetry: HII documents a request channel at
**nhcsoc@hii.or.th**.

---

## 3. Dams & reservoirs

> **Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1** and
> cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.

The predictive upstream reservoir is **Mae Ngat Somboon Chon** (Mae Ngat
tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the
Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.

| Source | What | Access |
|---|---|---|
| `https://app.rid.go.th/reservoir/api/dams` | **INGESTED** — daily snapshot of all ~35 large dams (storage/inflow/outflow MCM, % of usable). `POST` with form field `date=YYYY-MM-DD` (empty = today); GET returns 404 "Unknown method." Archive ≥ 2009; `level_msl` (`DMD_Q`) populated in older years only. Mae Ngat = `DAM_ID 200103` — hit 113% usable capacity, ~19 MCM/day inflow, in Oct 2024. Collected daily by `src/rid_reservoir.py` into `rid_dams` + `rid_reservoir_daily`; backfill via `scripts/backfill_rid_reservoir.py` | Open, no auth |
| `https://lsim.rid.go.th/ForeCast?reservoirid=22` | Mae Ngat daily status/forecast (RID) | **UNREACHABLE — do not plan around it.** Probed 2026-08-13 from a Thai consumer ISP (AIS Fibre, TH) *and* from abroad: DNS resolves (122.154.18.207) but ICMP is 100% loss and ports 80/443/8080 are filtered, while `app.rid.go.th` answers in 0.27 s over the same connection. Down or RID-internal-only — not a geo-block |
| `https://app.rid.go.th/reservoir/api/dam` | **Per-dam daily series in ONE request** — `GET` with `dam_id=200103&date_start=YYYY-MM-DD&date_end=YYYY-MM-DD&percent=`. Archive to 2009 (scattered single-day gaps). Far cheaper than the per-day `api/dams` loop the backfill used (one request vs ~2,900); prefer it for gap repair and for adding other dams. Sibling `api/damgraph` takes the same params | Open, no auth |
| `https://bigdata-api.rid.go.th` (SWOC) | **Intraday reservoir state** — RID SWOC telemetry, hourly with an explicit `hourly_time_utc` stamp; includes Mae Ngat (`TUP.16`) reservoir level m MSL and % capacity. **Snapshot-only — no archive**, so it can only be accumulated forward | Open, no auth |
| ThaiWater `public/waterlevel_load` stations `ridhydro_TUP.16` (at the dam) / `ridhydro_TUP.11` (dam outlet) | Hourly Mae Ngat reservoir level and outlet stage/flow. **Snapshot-only — `waterlevel_graph` returns empty grids for these ids at every era** (verified 2026-08-13). Collected hourly by our HII collector since 2026-08-11; accumulating forward | Open, no auth |
| ThaiWater `public/waterlevel_graph` station `P.75` (id 3253) | **The practical dam-release signal**: hourly stage+discharge 3.8 km below the Mae Ngat dam, history to 2019. Already ingested as a core RID station and a model feature since v1 | Open, no auth |
| ThaiWater `public/waterlevel_graph` station `MOU301` "สะพานน้ำแม่งัด" (id 1475118) | 10-minute stage on the Mae Ngat *above* the reservoir (inflow arm). History only from ~mid-2025; level only, no discharge | Open, no auth |
| ThaiWater `api-v3 .../analyst/dam` (dam.id 53) | EGAT-sourced copy of Mae Ngat carrying reservoir **level in m MSL historically** — the field RID's own API stopped populating (`DMD_Q`) after ~2013. Daily, no observation time | Open, no auth |
| `https://tiwrm.hii.or.th/DATA/REPORT/php/rid_bigcm_raw.php?sdate=YYYY-MM-DD` | HII HTML mirror of the RID large-dam daily table. Daily and *intermittent* (2026 YTD publishes ~108 of 225 days) — a cross-check, not a primary source | Open, no auth |
| `https://water.egat.co.th` | EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level | Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol |
| ThaiWater `/v2/large-dam/*`, `dam_rulecurve/graph` | All large/medium dams incl. hourly | Requires HII API key (§2.2) |

---

## 4. Rainfall & weather (external)

### Near-real-time (usable in the live inference path)

| Source | Cadence / latency | Access | Notes |
|---|---|---|---|
| HII `rain_24h` (§2.1) | Hourly, near-real-time | Open JSON | Primary rain-gauge feed; persist from now on |
| GSMaP NRT (JAXA) | Hourly, ~4 h latency, 0.1° | Free JAXA registration (FTP); or Google Earth Engine `JAXA/GPM_L3/GSMaP/v8/operational` (no registration) | Gauge-corrected `hourlyPrecipRateGC`; catchment-average rain where gauges are sparse |
| NASA IMERG **Early Run** | Half-hourly, ~4 h latency, 0.1° | Free Earthdata login | NASA-stack alternative to GSMaP |

### Forecasts (the only way past the ~17 h physical lead-time cap)

| Source | What | Access |
|---|---|---|
| **Open-Meteo** (<https://open-meteo.com>) — ✅ **INGESTED** since 2026-08-12 (`src/ml/rain.py`): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the `openmeteo_rain` table | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
| TMD NWP API (`https://data.tmd.go.th/nwpapi/v1/forecast/location/...`) | WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (`/nwpapi/doc/main/`) |
| GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |

### Training-only (too slow for live)

| Source | Cadence | Latency |
|---|---|---|
| CHIRPS (`data.chc.ucsb.edu/products/CHIRPS-2.0/`) | Daily, 0.05° | ~2 days prelim / 3+ weeks final |
| IMERG Late / Final | Half-hourly | ~14 h / ~3.5 months |
| TMD observation API (`data.tmd.go.th/api/index1.php`) | 3-hourly / daily station obs, XML | Free uid+key registration |

---

## 5. Historical / open-data portals

| Portal | Content |
|---|---|
| `https://data.hii.or.th` | 36 HII datasets (rainfall telemetry the most viewed) |
| `https://data.go.th/dataset?organization=rid` | 4 RID datasets (API + ZIP) |
| `https://gdcatalog.go.th` | Nationwide daily rainfall-station catalogs |
| `https://hydro-1.net` | RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download |
| `https://water.rid.go.th/flood/flood/daily.pdf` | RID daily flood bulletin (PDF only) |

### 5.1 CMU Water Center — CMFlood (Chiang Mai city) (verified 2026-10-25)

`https://watercenter.cmu.ac.th/chiangmai/cmflood/warning/*`, the Chiang Mai urban
flood warning site of Chiang Mai University's water center (with RID, ONWR, DDPM
and the province). A Next.js app; its data paths are in the page chunks
(`/_next/static/chunks/app/...`). No licence is stated; we credit CMU.

- **Flood hazard map, 2024 flood** — `/data/KML5Layer/L1.kml` … `L5.kml`: the
  flooded extent at P.1 = 4.30 / 4.50 / 4.70 / 5.00 / 5.30 m (2.5 / 3.0 / 22 / 57 /
  156 km², cumulative). CMU: the Ping overflows its banks above P.1 4.20 m, and the
  older municipal map (2011 flood) no longer fits the city's defences. **In use:**
  `P1_FLOOD_STAGES` and `src/static/flood-zones-p1.geojson`
  (`uv run scripts/build_flood_zones.py`).
- **Flood marks 2024** — `/data/New_floodMark_2025_fixed_like_pole.geojson` (4.9 MB):
  4,869 points in Chiang Mai and Lamphun with the October 2024 depth above the road
  in cm (P.1 = 5.30 m), the marker's place and type. **In use:** the dashboard's
  "Flood marks 2024" layer, `src/static/flood-marks-2024.json` (position and depth
  only, 110 KB; `uv run python scripts/build_flood_marks.py`). The depth does not
  scale one-to-one with P.1: marks in the zone that floods only at 5.30 m stood a
  median 65 cm deep in 2024, so the site shows the 2024 depth and no estimate for
  other levels.
- **Flood poles** — `/data/PoleCNX2025_v2.geojson`: 219 poles with ground level and
  maximum flood level (m MSL) and the depth.
- **City-reach gauges** (station page): P.67, P.103, P.1, P.104 (RID, `ST.16`
  every 15 min), FloodBoy FBP.2/FBP.3 (CMU climate data centre) and DDPM CMI01–03
  (camera gauges, ~5 min, public CCTV snapshots), with gauge zero, bank levels
  and chainage. CMU reads them from its own Supabase project with a public browser
  key (tables `disaster_level`, `flagship_hydro`, `water_levels`, `hydro_levels`;
  timestamps are Bangkok time labelled `+00:00`); history starts 2026-08 (FBP
  2025-09). Not ingested: too short to train on, and polling CMU's database is
  using their infrastructure — prefer the original sources or ask CMU.
- **P.1 forecast** (`floodforecast`): login only.

---

## 6. Integration status & recommended order

| Source | Status | Action |
|---|---|---|
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`/sensors/thaiwater`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
| Open-Meteo rain | ✅ Hourly by ping-collector → `openmeteo_rain` (5 catchment points, past 5 days + 48 h forecast); the 2021+ archive trains the hgb-v3 rain features | — |
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
| CMU CMFlood flood map (2024) | ✅ Static (2026-10-25): P.1 stages and the city's flood zones | Rebuild with `scripts/build_flood_zones.py` if CMU revises the map |
| CMU CMFlood flood marks 2024 | ✅ Static (2026-10-25): dashboard layer, nearest mark in zone and marker popups | Rebuild with `scripts/build_flood_marks.py` |
| CMU CMFlood poles, city gauges | ❌ | See 5.1 |

**Collector configuration** (ping-collector): HII, the dams and Open-Meteo are
fetched once per clock hour, also while RID is being retried; one-shot via
`ping-collector once`. Env vars: `ENABLE_HII_COLLECTION`,
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION` (default `true`),
`HII_BASIN_CODE` (default `6` = Ping). Tables are created automatically
(PostgreSQL). The archive backfills stay in Python (`scripts/backfill_*.py`,
using `src/hii_collector.py` and `src/rid_reservoir.py`).

**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
without notice. Poll hourly at most, cache aggressively, and pursue official
HII access for anything production-critical.