B4L/Northern-Thailand-Ping-River-Monitor

0 Fork 0

Files

T

grabowski 7e642555b4 chore: remove the Python service; Python is the training stack only

Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.

2026-10-18 17:15:07 +07:00

4.4 KiB

Raw Blame History

CLAUDE.md

Guidance for AI coding agents working in this repository.

What this is

Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and API at https://water.buildfor.life/ (never publish the server's private/mesh IP). Production: one systemd unit, ping-collector.service (the Rust binary in rust/), on a small VPS in /opt/thailand-water-monitor (a git checkout), user water-monitor; plus the monthly water-monitor-retrain.timer (Python, .venv, uv-managed). Updated by git pull, cargo build --release, sudo bash scripts/install.sh.

Rules

Python 3.11 only. uv sync --python 3.11; run everything as uv run ....
make format (black 88 / isort black profile, config in pyproject.toml; cargo fmt) before committing; CI fails on formatting. make test must stay green. Tests use synthetic data, never the network; database tests (both languages) run against a throwaway PostgreSQL schema from TEST_DATABASE_URL and skip without it.
PostgreSQL only. No other database backends.
Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard parses them with parseTs() and renders with timeZone: TZ; keep it that way.
Model changes go through the rolling-origin harness (scripts/evaluate_variants.py) and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive or negative, in docs/FLOOD_FORECASTING.md section 5. Do not change what is deployed (rise_rain / hgb-v3) without a harness result that beats it on lead.
train_all() must never silently produce a gauge-only (v2) model; the guard that raises RainUnavailableError stays.
No git add -A: zero-byte shell-accident files (#, $(wc, ...) have been committed before. Stage files by name.
Do not add Co-Authored-By trailers.
The dashboard is a single file, src/static/dashboard.html, EN + TH via the t() table: every user-visible string needs both languages. Every path it fetches must be a route in rust/collector/src/web/mod.rs (tests/test_dashboard.py checks).
Rust (rust/, see rust/README.md): cargo fmt --all, cargo clippy --all-targets -- -D warnings, cargo test; minimum version 1.88 (CARGO_TARGET_DIR=target/msrv cargo +1.88 check --all-targets --locked). Read .env only through rust/collector/src/envfile.rs (python-dotenv semantics, never dotenvy: the Python tools read the same file).
The forecast is served by Rust (rust/forecast) from models trained in Python; src/ml/predict.py is the reference it is tested against. A change to src/ml/features.py, predict.py, the HII fill or the dam frame must be ported in the same change: regenerate rust/forecast/testdata/gen_forecast_fixtures.py and keep cargo test -p ping-forecast green. Changing thresholds, stages, leads or the notifier settings (src/ml/export.py): uv run python -m src.ml.export --config (a test fails otherwise) and rebuild the collector.
Notifier texts are pinned by rust/collector/testdata/notify_python_cases.json.gz; a deliberate wording change updates that fixture in the same commit.
API routes live in rust/collector/src/web. A route's parameters are declared once (web/params.rs); that list validates and documents (/docs). Keep response formats stable: external clients use them.
The Python backfill stores (src/hii_collector.py, src/rid_reservoir.py, src/ml/rain.py) write the same tables as rust/collector/src/db.rs: a table or upsert change goes into both.

Where things are

rust/collector (ping-collector): collection (rid.rs, hii.rs, dams.rs, rain.rs), forecast run (forecasting.rs), ntfy/Matrix (notify.rs), website/API (web/), tables (db.rs). rust/forecast: inference.
src/ml/: features/train/evaluate/predict/export, rain.py (Open-Meteo), dam.py, hii_rain.py. src/hii_backfill.py, src/hii_collector.py, src/rid_reservoir.py: archive backfills (scripts/backfill_*.py).
scripts/install.sh: install/update on the server. scripts/retrain.sh + water-monitor-retrain.timer: monthly retrain with staged promote. scripts/forecast_parity.py: Rust vs Python forecast on a live database. scripts/dev_proxy.py: serve the working-copy dashboard against the live API.
docs/FLOOD_FORECASTING.md is the authoritative model write-up; docs/DATA_SOURCES.md the source catalog; docs/NOTIFICATIONS.md the alerts; docs/RUST_REWRITE.md the history of the Python-to-Rust move.

4.4 KiB Raw Blame History

CLAUDE.md

What this is

Rules

Where things are

4.4 KiB

Raw Blame History