10 Commits

Include renames

Author	SHA1	Message	Date
grabowski	9bf904b23c	feat(rid): screen one-hour spikes; refetch and redo-from to repair the store ... RID's report sometimes carries a wrong value for one hour and corrects it later. P.1 read 10.6 m at 2026-10-25 00:00 between 1.04 and 1.07 (RID now says 1.06); the store kept it, because a report is re-fetched only while it has holes, and the model issued a 10.8 m forecast for that hour. - rid::judge / rid::screen: a reading that leaves the previous one (within 3 h) by more than 1 m (2 m on the flashy tributaries P.76, P.77, P.81, P.82, P.84, P.85, P.87) while the next reading comes back near the previous level is a spike: not stored, a stored copy is deleted. Limbs and steps the next reading confirms are kept; the newest reading waits for the next one when it jumps more than twice the limit. On 2018-2026 history: 16 spikes found (slipped decimals, sign flips, -5/-6/-9 m placeholders), no genuine change rejected. - Every store goes through it (rid::store: run_cycle, fill-gaps, refetch). - `ping-collector refetch FROM [TO]`: re-fetch reports regardless of holes (corrects revised readings, fills the discharge of level-only evening hours, removes stored spikes) and print the catch-up command when readings changed. - `catch-up-forecasts --redo-from T`: delete the model forecasts issued from T on, then recompute them. - Tests: judge/screen unit tests; a DB test of refetch, the live hold and the forecast deletion. Checked end to end against RID on a scratch DB. - Docs: DATA_SOURCES (spikes, level-only hours), FLOOD_FORECASTING, rust/README.	2026-10-26 15:04:01 +07:00
grabowski	a610abcdb8	feat(dashboard): CMU's 2024 flood marks as a map layer ... CMU Water Center marked the October 2024 high-water line at ~4,900 places in Chiang Mai and Lamphun (depth above the road while P.1 stood at 5.30 m). - src/static/flood-marks-2024.json (position and depth, 110 KB), built by scripts/build_flood_marks.py from CMU's GeoJSON. - "Flood marks 2024" layer: canvas dots from zoom 12 in CMU's five depth classes, legend row, CMU credit; turning it on below zoom 12 flies to the city. A mark's popup gives the depth and its flood zone. - Zone and marker popups name the nearest mark within 250 m (loaded with the zones). - No depth estimate for other P.1 levels: the marks do not scale one-to-one with P.1 (marks in the 5.30 m zone stood a median 65 cm deep in 2024). - The layer bar wraps on narrow maps; popups pan clear of it. - Docs: DATA_SOURCES 5.1, DASHBOARD_REDESIGN, README.	2026-10-26 14:30:35 +07:00
grabowski	701bb87d34	feat(p1): city flood stages and zones from CMU Water Center's 2024 flood map ... CMU Water Center's Chiang Mai flood hazard map (CMFlood) replaces the older municipal map. It is built on the October 2024 flood and has the Ping leave its banks above P.1 4.20 m, with five stages: 4.30, 4.50, 4.70, 5.00 and 5.30 m (the 2024 record). The old map's seven stages (3.70-4.60 m) said the city floods from 3.70 m; CMU calls that map out of date. - P1_FLOOD_STAGES: the five CMU stages, discharge on RID's 2024 rating. /forecast's P.1 `stages` now has five entries (same format). Thresholds are unchanged: 4.20 m danger is the bank top, 3.70 m warning half a metre below (as on CMU's gauge page). flood_config.json and the Rust parity fixtures regenerated (only the stages changed). - flood-zones-p1.geojson: CMU's KML (L1..L5) as five disjoint zones, zone k floods first at stage k; scripts/build_flood_zones.py rebuilds it. - Dashboard: CMU's colours and credit (map attribution); the stage grid follows the number of stages; zone lookup for markers honours holes. The city verdict uses statusOf's classes: flooding (over the banks) from 4.20 m, high water from 3.70 m. Zones show by themselves when the river is over its banks or stage 1 is >= 50 % likely. Alert, replay and verdict texts no longer call 3.70 m city flooding (EN + TH). - Docs: DATA_SOURCES 5.1 (CMFlood: the map, flood marks, poles, city gauges), FLOOD_FORECASTING, DASHBOARD_REDESIGN, NOTIFICATIONS, README credits.	2026-10-26 14:19:52 +07:00
grabowski	7e642555b4	chore: remove the Python service; Python is the training stack only ... Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.	2026-10-18 17:15:07 +07:00
grabowski	5dc5850df6	docs: source sweep — P.75 already is the Mae Ngat release signal ... Verified from a Thai ISP that lsim.rid.go.th is unreachable (not geo-blocked), then swept for any better-than-daily Mae Ngat source. Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has been a model feature since v1 — the model has always read the dam's actual outflow, hourly and directly. That is the likelier reason the daily reservoir table adds nothing, beyond its publication lag. Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with no archive, so they cannot retrain against past events; the HII collector accumulates them from 2026-08-11 for a post-monsoon revisit. Also documents api/dam (whole per-dam range in one request, vs the ~2,900-request per-day loop) and several cross-check mirrors.	2026-08-13 21:58:47 +07:00
grabowski	70e4da07a0	docs: lsim.rid.go.th is unreachable, not geo-blocked ... Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Correct the earlier note that assumed the timeout was a foreign-network block, and stop pointing the dam-feature follow-up at a host that cannot be reached.	2026-08-13 21:00:24 +07:00
grabowski	6eafb353b1	feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow ... POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain.	2026-08-13 10:29:34 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00