B4L/Northern-Thailand-Ping-River-Monitor
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

feat: backfill hii_waterlevel from the HII waterlevel_graph archive
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 31s
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Details
Documentation / Documentation Summary (push) Successful in 2s
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Details
Documentation / Validate Documentation (push) Failing after 9s
Details
Documentation / Generate API Documentation (push) Successful in 9s
Details
Documentation / Build Sphinx Documentation (push) Successful in 15s
Details

Browse Source 
scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph
endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year
windows per station and upserts into hii_waterlevel. Defaults to the
RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days
override. History upserts touch only wl_msl and discharge so colliding
live-snapshot rows keep storage_percent/situation_level. Idempotent and
safe to re-run.
This commit is contained in:
grabowski
2026-08-11 15:11:08 +07:00
parent 33d8baa8dd
commit d72496f404
5 changed files with 383 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DATA_SOURCES.md
+1 -1
View File
@@ -238,7 +238,7 @@ Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| HII `waterlevel_graph` | ❌ Not ingested | Backfill script to cross-validate / gap-fill RID history (≥2019) |
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
| Mae Ngat reservoir (lsim.rid.go.th) | ❌ Not ingested | Add daily storage/release scrape |
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
| Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |