chore: remove the Python service; Python is the training stack only

Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558

No files matched your search

+15 -14
View File
@@ -23,7 +23,7 @@ The only source persisted to the database and used by the ML pipeline.
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
| Client | `src/water_scraper_v3.py` |
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports) |
Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
@@ -117,9 +117,9 @@ GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&
### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒
The layer our existing `src/thaiwater.py` client uses
(`GET /v2/waterlevel` with `x-api-key: $THAIWATER_API_KEY`; wired to
`GET /sensors/thaiwater` in the web API, display-only, never persisted).
The layer the website's `GET /sensors/thaiwater` uses (`GET /v2/waterlevel`
with an `x-api-key` header from `THAIWATER_API_KEY`;
`rust/collector/src/web/hii.rs`, display-only, never persisted).
Without a key: HTTP 401/500. No public key-registration page was found —
obtain a sanctioned key from HII (<https://hii.or.th>).
@@ -242,21 +242,22 @@ Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
| Source | Status | Action |
|---|---|---|
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`/sensors/thaiwater`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
| Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |
| Open-Meteo rain | ✅ Hourly by ping-collector → `openmeteo_rain` (5 catchment points, past 5 days + 48 h forecast); the 2021+ archive trains the hgb-v3 rain features | — |
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
**Collector configuration** (`src/hii_collector.py`): runs automatically every
scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry
mode) from both `--web-api` and continuous-monitoring modes; one-shot via
`python -m src.main --collect-hii`. Env vars: `ENABLE_HII_COLLECTION`
(default `true`), `HII_BASIN_CODE` (default `6` = Ping). Requires a SQL
`DB_TYPE` (sqlite/postgresql/mysql); tables are created automatically.
**Collector configuration** (ping-collector): HII, the dams and Open-Meteo are
fetched once per clock hour, also while RID is being retried; one-shot via
`ping-collector once`. Env vars: `ENABLE_HII_COLLECTION`,
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION` (default `true`),
`HII_BASIN_CODE` (default `6` = Ping). Tables are created automatically
(PostgreSQL). The archive backfills stay in Python (`scripts/backfill_*.py`,
using `src/hii_collector.py` and `src/rid_reservoir.py`).
**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
without notice. Poll hourly at most, cache aggressively, and pursue official