Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.
Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.
Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.
- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
the stores lost their sqlite/mysql branches (and the 2026-08 column
migration). The database tests run against a throwaway schema from
TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
a checkout without touching its ownership): binary, units, uv sync, .env
defaults, removes the retired water-monitor.service and any old
--alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
and Rust doc comments updated to the Rust service.
Verified from a Thai ISP that lsim.rid.go.th is unreachable (not
geo-blocked), then swept for any better-than-daily Mae Ngat source.
Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has
been a model feature since v1 — the model has always read the dam's
actual outflow, hourly and directly. That is the likelier reason the
daily reservoir table adds nothing, beyond its publication lag.
Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th
SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with
no archive, so they cannot retrain against past events; the HII
collector accumulates them from 2026-08-11 for a post-monsoon revisit.
Also documents api/dam (whole per-dam range in one request, vs the
~2,900-request per-day loop) and several cross-check mirrors.
Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and
ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s
over the same connection. Correct the earlier note that assumed the
timeout was a foreign-network block, and stop pointing the dam-feature
follow-up at a host that cannot be reached.
POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected
hourly into rid_dams + rid_reservoir_daily; backfill script fetches only
missing days so reruns repair holes and are safe alongside the live
collector. /api/stats counts the new table via an engine fallback that
works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113%
usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate
features for the next retrain.
The rolling-origin harness (models/eval_rain.json) showed catchment rain
halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and
extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at
P.103). Ported: train_all loads the catchment-mean series (use_rain /
--no-rain to opt out; without it bundles train as v2), predict fetches
live rain hourly and passes an empty series on failure so rain-trained
bundles serve with NaN features instead of tripping the feature guard,
and the leader worker persists hourly per-point + catchment-mean rows to
a new openmeteo_rain table.
Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING
(alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h
acceptance gate PASSES for the first time. Journey on that crossing:
v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its
artifact +46h latch for a calibrated +2h with zero false alarms. P.1
MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.
scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph
endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year
windows per station and upserts into hii_waterlevel. Defaults to the
RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days
override. History upserts touch only wl_msl and discharge so colliding
live-snapshot rows keep storage_percent/situation_level. Idempotent and
safe to re-run.
Poll the open api-v3.thaiwater.net public endpoints (rain_24h,
waterlevel_load), filter to the Ping basin, and persist to new
hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables
(auto-created; sqlite/postgresql/mysql). Water levels stay in their own
tables since HII reports m MSL from a different station set; rid_code
maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to
gauge datum. Runs every scraping cycle in web-api and continuous modes
(hourly cadence even during 1-minute RID retry), one-shot via
--collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.
grabowski