grabowski
779728e38b
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
64 lines
3.8 KiB
Markdown
64 lines
3.8 KiB
Markdown
# CLAUDE.md
|
|
|
|
Guidance for AI coding agents working in this repository.
|
|
|
|
## What this is
|
|
|
|
Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
|
|
API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
|
|
Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
|
|
`water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
|
|
|
|
## Rules
|
|
|
|
- Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
|
|
- `make format` (black 88 / isort black profile, config in pyproject.toml) before
|
|
committing; CI fails on formatting. `make test` must stay green — tests are
|
|
synthetic-data only, never add one that needs the DB or network.
|
|
- Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
|
|
parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
|
|
- Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
|
|
and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive
|
|
or negative, in `docs/FLOOD_FORECASTING.md` section 5. Do not change what is deployed
|
|
(`rise_rain` / hgb-v3) without a harness result that beats it on lead.
|
|
- `train_all()` must never silently produce a gauge-only (v2) model; the guard that
|
|
raises `RainUnavailableError` stays.
|
|
- No `git add -A`: zero-byte shell-accident files (`#`, `$(wc`, ...) have been committed
|
|
before. Stage files by name.
|
|
- Do not add Co-Authored-By trailers.
|
|
- The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
|
|
table: every user-visible string needs both languages.
|
|
- Rust (`rust/`, see rust/README.md): `cargo fmt --all`, `cargo clippy --all-targets
|
|
-- -D warnings`, `cargo test` (database tests need `TEST_DATABASE_URL`). The Rust
|
|
collector writes the same tables as the Python collectors: a change to a table,
|
|
upsert or parser on one side must be made on the other until the Python side is
|
|
retired (check with a live run of both into two databases).
|
|
- The forecast is served by Rust (`rust/forecast`) from models trained in Python.
|
|
A change to `src/ml/features.py`, `predict.py`, the HII fill or the dam frame
|
|
must be ported in the same change: regenerate
|
|
`rust/forecast/testdata/gen_forecast_fixtures.py` and keep
|
|
`cargo test -p ping-forecast` green. Changing thresholds, stages, leads or
|
|
notify constants: `uv run python -m src.ml.export --config` (a test fails
|
|
otherwise) and rebuild the collector. Notify texts:
|
|
`rust/collector/testdata/gen_notify_cases.py`.
|
|
- API routes live in `rust/collector/src/web` (phase 3). Until the FastAPI app is
|
|
removed, a route change goes into both, and `scripts/api_parity.py` (both
|
|
servers on one database) must report no differences. A route's parameters are
|
|
declared once (`web/params.rs`); that list validates and documents (/docs).
|
|
|
|
## Where things are
|
|
|
|
- `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
|
|
`src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
|
|
`rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
|
|
- `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
|
|
promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
|
|
- `rust/collector` (`ping-collector`): Rust data collector; with it deployed the
|
|
Python app runs `COLLECTION_ENABLED=false`. With `FORECAST_IN_COLLECTOR=true` it
|
|
also forecasts (`rust/forecast`) and sends ntfy + Matrix; with `WEB_LISTEN` it
|
|
serves the website and API (`rust/collector/src/web`, the port of
|
|
`src/web_api.py`; the dashboard stays `src/static/dashboard.html`). Rewrite
|
|
plan: `docs/RUST_REWRITE.md`.
|
|
- `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
|
|
the source catalog.
|