feat: hourly HII/ThaiWater rainfall + backup water-level collection

Poll the open api-v3.thaiwater.net public endpoints (rain_24h,
waterlevel_load), filter to the Ping basin, and persist to new
hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables
(auto-created; sqlite/postgresql/mysql). Water levels stay in their own
tables since HII reports m MSL from a different station set; rid_code
maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to
gauge datum. Runs every scraping cycle in web-api and continuous modes
(hourly cadence even during 1-minute RID retry), one-shot via
--collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.

docs/DATA_SOURCES.md

@@ -0,0 +1,256 @@
+# Data Sources & External API Catalog
+
+Catalog of every data source available to the Ping River Monitor — what we ingest
+today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for
+future model inputs (rainfall, dam releases, forecasts).
+
+All "verified" claims below were empirically probed on **2026-08-11**. Endpoints
+marked *(catalog)* were recovered from the thaiwater.net frontend JS bundle but
+not exercised (auth required).
+
+---
+
+## 1. Currently ingested — RID hydrology telemetry
+
+The only source persisted to the database and used by the ML pipeline.
+
+| | |
+|---|---|
+| Endpoint | `POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx` |
+| Agency | Royal Irrigation Department (RID) |
+| Auth | None |
+| Cadence | Hourly (`hourlytime` 1.00–24.00; hour 24 = midnight next day) |
+| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
+| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
+| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
+| Client | `src/water_scraper_v3.py` |
+
+Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
+
+---
+
+## 2. ThaiWater / HII ecosystem
+
+ThaiWater (<https://twa.thaiwater.net>) is the National Hydroinformatics
+Institute (HII) portal. It sits on **two distinct API layers** with very
+different access rules.
+
+### 2.1 `api-v3.thaiwater.net` — open, no authentication ✅
+
+Base: `https://api-v3.thaiwater.net/api/v1/thaiwater30/public/`
+
+Undocumented backend of the portal. No key, no session. No published rate
+limits or terms of use — be a good citizen (hourly polls, cache, filter to
+Ping basin `basin_code == 6`).
+
+#### `waterlevel_load` — national water-level snapshot (verified)
+
+```
+GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load
+```
+
+- ~2.4 MB, 1,426 stations nationwide; **125 in Ping Basin, 61 in Chiang Mai
+  province, 34 with discharge**.
+- Per station: `waterlevel_m`, `waterlevel_msl`, `waterlevel_msl_previous`,
+  `flow_rate`, `discharge`, `storage_percent`, `situation_level` (1–4 flood
+  severity), `diff_wl_bank`, bank/critical levels (`min_bank`,
+  `critical_level_msl`, `warning_level_m`, `critical_level_m`, `qmax`),
+  `river_name`, basin, geocode, agency, lat/long, `is_key_station`.
+- Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A,
+  P.7A, P.77, P.2A — plus RID mirrors (`ridhydro_P.1`, …), Tak-reach `TP.`/
+  `TUP.` codes, and HII sensor clusters (`PIN001-011`, `CHM001-005`).
+- **P.1 = internal station id `3226`** (lat 18.786961, long 99.005089).
+
+#### `waterlevel_graph` — hourly historical time series (verified) ⭐
+
+```
+GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08
+```
+
+- Returns `{data: {graph_data: [{datetime, value, value_out, discharge}]}}`,
+  hourly. `value` is **water level in m MSL** (not gauge datum).
+- **Respects arbitrary date ranges. Archive verified back to at least
+  2019-08** (P.1 returned data for 2019-08-01). 14-day windows tested OK;
+  maximum window size not probed.
+- Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00,
+  discharge 656 m³/s.
+- Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒
+  **gauge ≈ MSL − 300.5 m** (verify per station before use; each station has
+  its own datum offset).
+- Value: independent second historical source for cross-validating / gap-filling
+  the RID feed (RID grid is only ~56% filled).
+
+#### `rain_24h` — national rainfall snapshot (verified)
+
+```
+GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h
+```
+
+- ~4.5 MB, 4,445 stations; **321 in Chiang Mai province**; agencies include
+  HII, DWR, RID.
+- Per station: `rain_1h`, `rain_24h` (mm), `rainfall_datetime`,
+  `station.id` (small int — the graph key), `station.tele_station_oldcode`
+  (e.g. `CHM005`, `STN0410`, `ridtele_TUP.14`), `sub_basin_id`, lat/long,
+  basin, geocode, agency.
+- **This is the missing rainfall input** for the flood model — near-real-time
+  hourly gauge rain across the upper Ping catchment.
+
+#### `rain_24h_graph` — trailing-window rainfall series (verified, limited) ⚠️
+
+```
+GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...
+```
+
+- `station_id` is the **small `station.id`** from `rain_24h` (e.g. 418 =
+  CHM005 "Chiang Mai 5", Mae Taeng), *not* the top-level record id.
+- Returns hourly `{rainfall_datetime, rainfall_value}` — **but the date range
+  is IGNORED**: every request returns the same trailing ~36-hour window
+  (39 rows). Requests for 2020/2024 return identical data to today.
+- Consequence: **no rainfall history via this API**. To build training data,
+  persist `rain_24h` from now on and backfill history from satellite QPE or an
+  HII data request (§4).
+
+#### Probed and NOT available on api-v3 (all HTTP 404)
+
+`dam_daily`, `dam`, `dam_json`, `big_dam`, `mainstream_dam`, `weather`,
+`rain_graph`, `rainfall_graph`, `rain24hr_graph`. Dam data is v2-only (§2.2).
+
+### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒
+
+The layer our existing `src/thaiwater.py` client uses
+(`GET /v2/waterlevel` with `x-api-key: $THAIWATER_API_KEY`; wired to
+`GET /sensors/thaiwater` in the web API, display-only, never persisted).
+Without a key: HTTP 401/500. No public key-registration page was found —
+obtain a sanctioned key from HII (<https://hii.or.th>).
+
+Full endpoint catalog *(catalog — recovered from frontend JS, not exercised)*:
+
+- **Water level / discharge**: `/v2/waterlevel`, `/v2/waterlevel/list`,
+  `/v2/waterlevel/{id}/detail`, `/v2/waterlevel/canal`,
+  `/v2/waterlevel/sea-waterlevel`, `/v2/waterlevel-discharge`,
+  `/v2/waterlevel-discharge/list`, `/v2/waterlevel-discharge/{id}/detail`,
+  `/v2/waterlevel-discharge/{id}/forecast-table`,
+  `/v2/waterlevel-discharge/forecast`, `/v2/waterlevel-discharge/forecast/list`,
+  `/v2/watergate`, `/v2/waterload-tide`
+- **Dams** (incl. Bhumibol): `/v2/large-dam/daily-geo-json`,
+  `/v2/large-dam/daily/list`, `/v2/large-dam/hourly/list`,
+  `/v2/large-dam/daily/{id}/detail`, `/v2/large-dam/hourly/{id}/detail`,
+  `/v2/medium-dam/daily-geo-json`, `/v2/medium-dam/daily/list`,
+  `/v2/medium-dam/{id}/detail`, `/v2/summary/summary4dam`,
+  `/v2/summary/dam-summary`, `/v2/summary/dam-crisis`
+- **Rainfall**: `/v2/rainfall/{type}`, `/v2/rainfall/{type}/list`,
+  `/v2/district-rain/actual-measure`, `/v2/district-rain/forecast`,
+  `/v2/district-rain/accumulate`, `/v2/summary/rainfall24h-ranking-province`,
+  `/v2/summary/rainfall-24hr-forecast`, `/v2/summary/rainfall-forecast`,
+  `/v2/summary/warning-rainfall-24h`, `/v2/summary/warning-rainfall-48h`
+- **Weather / hazards**: `/v2/weather`, `/v2/storm`, `/v2/wave`, `/v2/pm25`,
+  `/v2/pm10`, `/v2/flood/flash-flood`, `/v2/flood/flash-flood-alert`,
+  `/v2/drought/alert`, `/v2/drought/risk-area/list`,
+  `/v2/summary/weather-summary`, `/v2/summary/temperature-forecast`,
+  `/v2/summary-area/rainfall`
+- **Time-series / graph** (base `/data/platform/v1/public/`):
+  `tele_waterlevel/graph`, `flow/graph`, `latest_waterlevel/forecast/graph`,
+  `latest_watertide/forecast/graph`, `dam_pdaily_sum_by_date`,
+  `dam_pdaily_sum_by_region_graph`, `dam_rulecurve/graph`,
+  `medium_dam/graph_year`, `monthly_rainfall/stations`,
+  `monthly_rainfall/anomaly-stations`, `tele_watergate/graph`,
+  `salinity_forecast_cpy/graph`, `sea_waterlevel_forecast/graph`,
+  `latest_weather_area`, `latest_weather_area_daily`
+
+### 2.3 Other HII hosts
+
+| Host | What | Access |
+|---|---|---|
+| `https://standard.thaiwater.net` | **Official water-data standard** — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai) | Open, docs site |
+| `https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}` | Daily isohyet rainfall by province | Token in path |
+| `https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif` | Rainfall anomaly & 1–6-month forecast GeoTIFF rasters | Open |
+| `https://data.hii.or.th` | HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate) | Open browsing |
+| `https://tiwrm.hii.or.th` | Legacy reports | Open |
+
+Historical bulk telemetry: HII documents a request channel at
+**nhcsoc@hii.or.th**.
+
+---
+
+## 3. Dams & reservoirs
+
+> **Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1** and
+> cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.
+
+The predictive upstream reservoir is **Mae Ngat Somboon Chon** (Mae Ngat
+tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the
+Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
+
+| Source | What | Access |
+|---|---|---|
+| `https://lsim.rid.go.th/ForeCast?reservoirid=22` | Mae Ngat daily status/forecast (RID) | Open, scrape |
+| `https://app.rid.go.th/reservoir/` | RID reservoir DB, per-reservoir daily detail with date-range URLs | Open, scrape (confirm URL pattern before hard-coding) |
+| `https://water.egat.co.th` | EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level | Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol |
+| ThaiWater `/v2/large-dam/*`, `dam_rulecurve/graph` | All large/medium dams incl. hourly | Requires HII API key (§2.2) |
+
+---
+
+## 4. Rainfall & weather (external)
+
+### Near-real-time (usable in the live inference path)
+
+| Source | Cadence / latency | Access | Notes |
+|---|---|---|---|
+| HII `rain_24h` (§2.1) | Hourly, near-real-time | Open JSON | Primary rain-gauge feed; persist from now on |
+| GSMaP NRT (JAXA) | Hourly, ~4 h latency, 0.1° | Free JAXA registration (FTP); or Google Earth Engine `JAXA/GPM_L3/GSMaP/v8/operational` (no registration) | Gauge-corrected `hourlyPrecipRateGC`; catchment-average rain where gauges are sparse |
+| NASA IMERG **Early Run** | Half-hourly, ~4 h latency, 0.1° | Free Earthdata login | NASA-stack alternative to GSMaP |
+
+### Forecasts (the only way past the ~17 h physical lead-time cap)
+
+| Source | What | Access |
+|---|---|---|
+| **Open-Meteo** (<https://open-meteo.com>) | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
+| TMD NWP API (`https://data.tmd.go.th/nwpapi/v1/forecast/location/...`) | WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (`/nwpapi/doc/main/`) |
+| GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |
+
+### Training-only (too slow for live)
+
+| Source | Cadence | Latency |
+|---|---|---|
+| CHIRPS (`data.chc.ucsb.edu/products/CHIRPS-2.0/`) | Daily, 0.05° | ~2 days prelim / 3+ weeks final |
+| IMERG Late / Final | Half-hourly | ~14 h / ~3.5 months |
+| TMD observation API (`data.tmd.go.th/api/index1.php`) | 3-hourly / daily station obs, XML | Free uid+key registration |
+
+---
+
+## 5. Historical / open-data portals
+
+| Portal | Content |
+|---|---|
+| `https://data.hii.or.th` | 36 HII datasets (rainfall telemetry the most viewed) |
+| `https://data.go.th/dataset?organization=rid` | 4 RID datasets (API + ZIP) |
+| `https://gdcatalog.go.th` | Nationwide daily rainfall-station catalogs |
+| `https://hydro-1.net` | RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download |
+| `https://water.rid.go.th/flood/flood/daily.pdf` | RID daily flood bulletin (PDF only) |
+
+---
+
+## 6. Integration status & recommended order
+
+| Source | Status | Action |
+|---|---|---|
+| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
+| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
+| HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
+| HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
+| HII `waterlevel_graph` | ❌ Not ingested | Backfill script to cross-validate / gap-fill RID history (≥2019) |
+| Mae Ngat reservoir (lsim.rid.go.th) | ❌ Not ingested | Add daily storage/release scrape |
+| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
+| Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |
+| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
+
+**Collector configuration** (`src/hii_collector.py`): runs automatically every
+scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry
+mode) from both `--web-api` and continuous-monitoring modes; one-shot via
+`python -m src.main --collect-hii`. Env vars: `ENABLE_HII_COLLECTION`
+(default `true`), `HII_BASIN_CODE` (default `6` = Ping). Requires a SQL
+`DB_TYPE` (sqlite/postgresql/mysql); tables are created automatically.
+
+**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
+without notice. Poll hourly at most, cache aggressively, and pursue official
+HII access for anything production-critical.

docs/references/NOTABLE_DOCUMENTS.md

@@ -34,6 +34,20 @@ This document contains important references and external resources related to th
 - **Usage**: Reference for individual station characteristics and historical data patterns
 - **Station**: P.76 - บ้านแม่อีไฮ (Banb Mae I Hai)
 
+### **ThaiWater / HII (Hydro-Informatics Institute) Resources**
+
+#### **4. ThaiWater Portal**
+- **URL**: https://twa.thaiwater.net
+- **Description**: National water situation portal (rainfall, water level, dams, warnings)
+- **Language**: Thai/English
+- **Usage**: Backed by open and auth-gated APIs — full endpoint catalog in [DATA_SOURCES.md](../DATA_SOURCES.md)
+
+#### **5. ThaiWater Data Standard**
+- **URL**: https://standard.thaiwater.net
+- **Description**: Official water-data standard for exchange and warning — canonical station/basin/province code registries, data formats, warning-level definitions
+- **Language**: Thai
+- **Usage**: Reference for station metadata and warning-level semantics
+
 ## 📊 **Data Sources and APIs**
 
 ### **Primary Data Source**