feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions
CI / Format & lint (push) Successful in 26s
CI / Test suite (push) Successful in 1m9s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 11m46s

Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.

Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).

scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.

Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).

install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
This commit is contained in:
grabowski committed 2026-10-18 16:08:06 +07:00
1 parent 1d1c000922
commit 779728e38b
32 files changed
+5692 -160

No files matched your search

+40 -2
View File
@@ -7,6 +7,7 @@ costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md
| --- | --- | --- |
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
| `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: in production since 2026-10-18 |
| `collector`: `src/web` | the FastAPI app (`src/web_api.py`): dashboard, read API, `/docs` | phase 3: built, cutover pending |
## ping-collector
@@ -19,7 +20,10 @@ the database advances.
```text
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
# while RID is behind), HII/dams/rain once per clock hour;
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy/Matrix;
# with WEB_LISTEN the website and API
ping-collector serve [--listen A] [--no-analytics]
# the website and API only (default 127.0.0.1:8001)
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
# and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
@@ -39,7 +43,7 @@ only its line number, never its value. Variables already set (e.g. by systemd's
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
`SCRAPING_STATUS_FILE` (status JSON for `/scraping/status` when the website runs elsewhere),
`LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false),
`MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX`
/ `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18,
@@ -105,6 +109,37 @@ Verified:
`src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy
outages); the Rust notifier sends the same 751 messages word for word.
## Website and API (phase 3)
`src/web` serves what the FastAPI app served: the dashboard
(`STATIC_DIR`/dashboard.html, default `src/static`, loaded at start), the
crawler files, `/static`, `/docs` + `/redoc` + `/openapi.json`, and every API
route with the same parameters, JSON (field order included), status codes,
`{"detail": ...}` errors, Cache-Control, gzip, CORS (`CORS_ALLOW_ORIGINS`),
Umami API tracking (`UMAMI_*`) and response caches (`*_CACHE_TTL_SECONDS`).
Admin routes need `ADMIN_API_KEY`. With `WEB_LISTEN=0.0.0.0:8000` it runs
inside `ping-collector run` (one process collects, forecasts, notifies and
serves; `/scraping/status` and `POST /scrape/trigger` then talk to the
collection loop directly); `serve` runs it alone.
- Parameters are declared once per route (`web/params.rs`): the same list
validates the request (pydantic-style 422s) and builds the OpenAPI
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
The component schemas are FastAPI's (`web/openapi_components.json`).
- Python's output conventions are kept where they show: `isoformat()`
timestamps, `round()`, numpy's pairwise sums for the catchment means
(`web/pyfmt.rs`), the skill metrics' summation order.
- `scripts/api_parity.py --python URL --rust URL` compares the two live on
one database; `install_collector.sh --check-api` / `--cutover-api` run it
on the server against `ping-collector serve` on port 8001.
Matrix (`MATRIX_HOMESERVER`, `MATRIX_ACCESS_TOKEN`, `MATRIX_ROOM_ID`): the
notifier sends the room the same transitions as ntfy (per station, outlook,
feed; not ntfy's per-topic summaries), each once, with its own `matrix:` keys
in `notification_state` (seeded from the ntfy ones on first use, so nothing
already announced is repeated). It replaces the `run.py --alert-check` cron
job, which the install script comments out.
### Develop
```bash
@@ -142,6 +177,9 @@ cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI ar
sudo bash scripts/install_collector.sh # install + checks, nothing switched
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies
sudo bash scripts/install_collector.sh --check-api # compare the Rust API with FastAPI
sudo bash scripts/install_collector.sh --cutover-api # Rust serves the website on :8000
sudo bash scripts/install_collector.sh --rollback-api # FastAPI serves the website again
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
sudo bash scripts/install_collector.sh --rollback # Python does everything again
journalctl -u ping-collector -f