chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
@@ -5,16 +5,20 @@ Guidance for AI coding agents working in this repository.
|
||||
## What this is
|
||||
|
||||
Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
|
||||
API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
|
||||
Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
|
||||
`water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
|
||||
API at https://water.buildfor.life/ (never publish the server's private/mesh IP).
|
||||
Production: one systemd unit, `ping-collector.service` (the Rust binary in `rust/`),
|
||||
on a small VPS in `/opt/thailand-water-monitor` (a git checkout), user `water-monitor`;
|
||||
plus the monthly `water-monitor-retrain.timer` (Python, `.venv`, uv-managed). Updated by
|
||||
`git pull`, `cargo build --release`, `sudo bash scripts/install.sh`.
|
||||
|
||||
## Rules
|
||||
|
||||
- Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
|
||||
- `make format` (black 88 / isort black profile, config in pyproject.toml) before
|
||||
committing; CI fails on formatting. `make test` must stay green — tests are
|
||||
synthetic-data only, never add one that needs the DB or network.
|
||||
- `make format` (black 88 / isort black profile, config in pyproject.toml; cargo fmt)
|
||||
before committing; CI fails on formatting. `make test` must stay green. Tests use
|
||||
synthetic data, never the network; database tests (both languages) run against a
|
||||
throwaway PostgreSQL schema from `TEST_DATABASE_URL` and skip without it.
|
||||
- PostgreSQL only. No other database backends.
|
||||
- Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
|
||||
parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
|
||||
- Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
|
||||
@@ -27,37 +31,41 @@ Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
|
||||
before. Stage files by name.
|
||||
- Do not add Co-Authored-By trailers.
|
||||
- The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
|
||||
table: every user-visible string needs both languages.
|
||||
table: every user-visible string needs both languages. Every path it fetches must be
|
||||
a route in `rust/collector/src/web/mod.rs` (tests/test_dashboard.py checks).
|
||||
- Rust (`rust/`, see rust/README.md): `cargo fmt --all`, `cargo clippy --all-targets
|
||||
-- -D warnings`, `cargo test` (database tests need `TEST_DATABASE_URL`). The Rust
|
||||
collector writes the same tables as the Python collectors: a change to a table,
|
||||
upsert or parser on one side must be made on the other until the Python side is
|
||||
retired (check with a live run of both into two databases).
|
||||
- The forecast is served by Rust (`rust/forecast`) from models trained in Python.
|
||||
A change to `src/ml/features.py`, `predict.py`, the HII fill or the dam frame
|
||||
must be ported in the same change: regenerate
|
||||
`rust/forecast/testdata/gen_forecast_fixtures.py` and keep
|
||||
`cargo test -p ping-forecast` green. Changing thresholds, stages, leads or
|
||||
notify constants: `uv run python -m src.ml.export --config` (a test fails
|
||||
otherwise) and rebuild the collector. Notify texts:
|
||||
`rust/collector/testdata/gen_notify_cases.py`.
|
||||
- API routes live in `rust/collector/src/web` (phase 3). Until the FastAPI app is
|
||||
removed, a route change goes into both, and `scripts/api_parity.py` (both
|
||||
servers on one database) must report no differences. A route's parameters are
|
||||
declared once (`web/params.rs`); that list validates and documents (/docs).
|
||||
-- -D warnings`, `cargo test`; minimum version 1.88
|
||||
(`CARGO_TARGET_DIR=target/msrv cargo +1.88 check --all-targets --locked`). Read
|
||||
`.env` only through `rust/collector/src/envfile.rs` (python-dotenv semantics, never
|
||||
dotenvy: the Python tools read the same file).
|
||||
- The forecast is served by Rust (`rust/forecast`) from models trained in Python;
|
||||
`src/ml/predict.py` is the reference it is tested against. A change to
|
||||
`src/ml/features.py`, `predict.py`, the HII fill or the dam frame must be ported in
|
||||
the same change: regenerate `rust/forecast/testdata/gen_forecast_fixtures.py` and keep
|
||||
`cargo test -p ping-forecast` green. Changing thresholds, stages, leads or the
|
||||
notifier settings (`src/ml/export.py`): `uv run python -m src.ml.export --config`
|
||||
(a test fails otherwise) and rebuild the collector.
|
||||
- Notifier texts are pinned by `rust/collector/testdata/notify_python_cases.json.gz`;
|
||||
a deliberate wording change updates that fixture in the same commit.
|
||||
- API routes live in `rust/collector/src/web`. A route's parameters are declared once
|
||||
(`web/params.rs`); that list validates and documents (/docs). Keep response formats
|
||||
stable: external clients use them.
|
||||
- The Python backfill stores (`src/hii_collector.py`, `src/rid_reservoir.py`,
|
||||
`src/ml/rain.py`) write the same tables as `rust/collector/src/db.rs`: a table or
|
||||
upsert change goes into both.
|
||||
|
||||
## Where things are
|
||||
|
||||
- `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
|
||||
`src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
|
||||
`rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
|
||||
- `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
|
||||
promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
|
||||
- `rust/collector` (`ping-collector`): Rust data collector; with it deployed the
|
||||
Python app runs `COLLECTION_ENABLED=false`. With `FORECAST_IN_COLLECTOR=true` it
|
||||
also forecasts (`rust/forecast`) and sends ntfy + Matrix; with `WEB_LISTEN` it
|
||||
serves the website and API (`rust/collector/src/web`, the port of
|
||||
`src/web_api.py`; the dashboard stays `src/static/dashboard.html`). Rewrite
|
||||
plan: `docs/RUST_REWRITE.md`.
|
||||
- `rust/collector` (`ping-collector`): collection (`rid.rs`, `hii.rs`, `dams.rs`,
|
||||
`rain.rs`), forecast run (`forecasting.rs`), ntfy/Matrix (`notify.rs`), website/API
|
||||
(`web/`), tables (`db.rs`). `rust/forecast`: inference.
|
||||
- `src/ml/`: features/train/evaluate/predict/export, `rain.py` (Open-Meteo), `dam.py`,
|
||||
`hii_rain.py`. `src/hii_backfill.py`, `src/hii_collector.py`, `src/rid_reservoir.py`:
|
||||
archive backfills (`scripts/backfill_*.py`).
|
||||
- `scripts/install.sh`: install/update on the server. `scripts/retrain.sh` +
|
||||
`water-monitor-retrain.timer`: monthly retrain with staged promote.
|
||||
`scripts/forecast_parity.py`: Rust vs Python forecast on a live database.
|
||||
`scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
|
||||
- `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
|
||||
the source catalog.
|
||||
the source catalog; `docs/NOTIFICATIONS.md` the alerts; `docs/RUST_REWRITE.md` the
|
||||
history of the Python-to-Rust move.
|
||||
Reference in new issue
Block a user