chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
+15
-14
@@ -23,7 +23,7 @@ The only source persisted to the database and used by the ML pipeline.
|
||||
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
|
||||
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
|
||||
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
|
||||
| Client | `src/water_scraper_v3.py` |
|
||||
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports) |
|
||||
|
||||
Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
|
||||
|
||||
@@ -117,9 +117,9 @@ GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&
|
||||
|
||||
### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒
|
||||
|
||||
The layer our existing `src/thaiwater.py` client uses
|
||||
(`GET /v2/waterlevel` with `x-api-key: $THAIWATER_API_KEY`; wired to
|
||||
`GET /sensors/thaiwater` in the web API, display-only, never persisted).
|
||||
The layer the website's `GET /sensors/thaiwater` uses (`GET /v2/waterlevel`
|
||||
with an `x-api-key` header from `THAIWATER_API_KEY`;
|
||||
`rust/collector/src/web/hii.rs`, display-only, never persisted).
|
||||
Without a key: HTTP 401/500. No public key-registration page was found —
|
||||
obtain a sanctioned key from HII (<https://hii.or.th>).
|
||||
|
||||
@@ -242,21 +242,22 @@ Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
|
||||
| Source | Status | Action |
|
||||
|---|---|---|
|
||||
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
|
||||
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
|
||||
| HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
|
||||
| HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
|
||||
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`/sensors/thaiwater`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
|
||||
| HII `rain_24h` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
|
||||
| HII `waterlevel_load` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
|
||||
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
|
||||
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
|
||||
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
|
||||
| Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |
|
||||
| Open-Meteo rain | ✅ Hourly by ping-collector → `openmeteo_rain` (5 catchment points, past 5 days + 48 h forecast); the 2021+ archive trains the hgb-v3 rain features | — |
|
||||
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
|
||||
|
||||
**Collector configuration** (`src/hii_collector.py`): runs automatically every
|
||||
scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry
|
||||
mode) from both `--web-api` and continuous-monitoring modes; one-shot via
|
||||
`python -m src.main --collect-hii`. Env vars: `ENABLE_HII_COLLECTION`
|
||||
(default `true`), `HII_BASIN_CODE` (default `6` = Ping). Requires a SQL
|
||||
`DB_TYPE` (sqlite/postgresql/mysql); tables are created automatically.
|
||||
**Collector configuration** (ping-collector): HII, the dams and Open-Meteo are
|
||||
fetched once per clock hour, also while RID is being retried; one-shot via
|
||||
`ping-collector once`. Env vars: `ENABLE_HII_COLLECTION`,
|
||||
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION` (default `true`),
|
||||
`HII_BASIN_CODE` (default `6` = Ping). Tables are created automatically
|
||||
(PostgreSQL). The archive backfills stay in Python (`scripts/backfill_*.py`,
|
||||
using `src/hii_collector.py` and `src/rid_reservoir.py`).
|
||||
|
||||
**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
|
||||
without notice. Poll hourly at most, cache aggressively, and pursue official
|
||||
|
||||
Reference in new issue
Block a user