grabowski
779728e38b
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
3.8 KiB
3.8 KiB
CLAUDE.md
Guidance for AI coding agents working in this repository.
What this is
Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
Production: one systemd unit on a small VPS, /opt/thailand-water-monitor, user
water-monitor, interpreter .venv/bin/python (uv-managed), updated by git pull.
Rules
- Python 3.11 only.
uv sync --python 3.11; run everything asuv run .... make format(black 88 / isort black profile, config in pyproject.toml) before committing; CI fails on formatting.make testmust stay green — tests are synthetic-data only, never add one that needs the DB or network.- Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
parses them with
parseTs()and renders withtimeZone: TZ; keep it that way. - Model changes go through the rolling-origin harness (
scripts/evaluate_variants.py) and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive or negative, indocs/FLOOD_FORECASTING.mdsection 5. Do not change what is deployed (rise_rain/ hgb-v3) without a harness result that beats it on lead. train_all()must never silently produce a gauge-only (v2) model; the guard that raisesRainUnavailableErrorstays.- No
git add -A: zero-byte shell-accident files (#,$(wc, ...) have been committed before. Stage files by name. - Do not add Co-Authored-By trailers.
- The dashboard is a single file,
src/static/dashboard.html, EN + TH via thet()table: every user-visible string needs both languages. - Rust (
rust/, see rust/README.md):cargo fmt --all,cargo clippy --all-targets -- -D warnings,cargo test(database tests needTEST_DATABASE_URL). The Rust collector writes the same tables as the Python collectors: a change to a table, upsert or parser on one side must be made on the other until the Python side is retired (check with a live run of both into two databases). - The forecast is served by Rust (
rust/forecast) from models trained in Python. A change tosrc/ml/features.py,predict.py, the HII fill or the dam frame must be ported in the same change: regeneraterust/forecast/testdata/gen_forecast_fixtures.pyand keepcargo test -p ping-forecastgreen. Changing thresholds, stages, leads or notify constants:uv run python -m src.ml.export --config(a test fails otherwise) and rebuild the collector. Notify texts:rust/collector/testdata/gen_notify_cases.py. - API routes live in
rust/collector/src/web(phase 3). Until the FastAPI app is removed, a route change goes into both, andscripts/api_parity.py(both servers on one database) must report no differences. A route's parameters are declared once (web/params.rs); that list validates and documents (/docs).
Where things are
src/web_api.pyFastAPI app;src/water_scraper_v3.pyRID collector;src/hii_collector.pyThaiWater/HII;src/ml/features/train/evaluate/predict,rain.py(Open-Meteo),dam.py,hii_rain.py.scripts/retrain.sh+water-monitor-retrain.timer: monthly retrain with staged promote.scripts/dev_proxy.py: serve the working-copy dashboard against the live API.rust/collector(ping-collector): Rust data collector; with it deployed the Python app runsCOLLECTION_ENABLED=false. WithFORECAST_IN_COLLECTOR=trueit also forecasts (rust/forecast) and sends ntfy + Matrix; withWEB_LISTENit serves the website and API (rust/collector/src/web, the port ofsrc/web_api.py; the dashboard stayssrc/static/dashboard.html). Rewrite plan:docs/RUST_REWRITE.md.docs/FLOOD_FORECASTING.mdis the authoritative model write-up;docs/DATA_SOURCES.mdthe source catalog.