feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
This commit is contained in:
1 parent
1d1c000922
commit
779728e38b
32 files changed
+5692
-160
No files matched your search
+40
-2
@@ -7,6 +7,7 @@ costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md
|
||||
| --- | --- | --- |
|
||||
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
|
||||
| `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: in production since 2026-10-18 |
|
||||
| `collector`: `src/web` | the FastAPI app (`src/web_api.py`): dashboard, read API, `/docs` | phase 3: built, cutover pending |
|
||||
|
||||
## ping-collector
|
||||
|
||||
@@ -19,7 +20,10 @@ the database advances.
|
||||
```text
|
||||
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
|
||||
# while RID is behind), HII/dams/rain once per clock hour;
|
||||
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy
|
||||
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy/Matrix;
|
||||
# with WEB_LISTEN the website and API
|
||||
ping-collector serve [--listen A] [--no-analytics]
|
||||
# the website and API only (default 127.0.0.1:8001)
|
||||
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
|
||||
# and compares the RID fetch with the database
|
||||
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
|
||||
@@ -39,7 +43,7 @@ only its line number, never its value. Variables already set (e.g. by systemd's
|
||||
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
|
||||
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
|
||||
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
|
||||
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
|
||||
`SCRAPING_STATUS_FILE` (status JSON for `/scraping/status` when the website runs elsewhere),
|
||||
`LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false),
|
||||
`MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX`
|
||||
/ `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18,
|
||||
@@ -105,6 +109,37 @@ Verified:
|
||||
`src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy
|
||||
outages); the Rust notifier sends the same 751 messages word for word.
|
||||
|
||||
## Website and API (phase 3)
|
||||
|
||||
`src/web` serves what the FastAPI app served: the dashboard
|
||||
(`STATIC_DIR`/dashboard.html, default `src/static`, loaded at start), the
|
||||
crawler files, `/static`, `/docs` + `/redoc` + `/openapi.json`, and every API
|
||||
route with the same parameters, JSON (field order included), status codes,
|
||||
`{"detail": ...}` errors, Cache-Control, gzip, CORS (`CORS_ALLOW_ORIGINS`),
|
||||
Umami API tracking (`UMAMI_*`) and response caches (`*_CACHE_TTL_SECONDS`).
|
||||
Admin routes need `ADMIN_API_KEY`. With `WEB_LISTEN=0.0.0.0:8000` it runs
|
||||
inside `ping-collector run` (one process collects, forecasts, notifies and
|
||||
serves; `/scraping/status` and `POST /scrape/trigger` then talk to the
|
||||
collection loop directly); `serve` runs it alone.
|
||||
|
||||
- Parameters are declared once per route (`web/params.rs`): the same list
|
||||
validates the request (pydantic-style 422s) and builds the OpenAPI
|
||||
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
|
||||
The component schemas are FastAPI's (`web/openapi_components.json`).
|
||||
- Python's output conventions are kept where they show: `isoformat()`
|
||||
timestamps, `round()`, numpy's pairwise sums for the catchment means
|
||||
(`web/pyfmt.rs`), the skill metrics' summation order.
|
||||
- `scripts/api_parity.py --python URL --rust URL` compares the two live on
|
||||
one database; `install_collector.sh --check-api` / `--cutover-api` run it
|
||||
on the server against `ping-collector serve` on port 8001.
|
||||
|
||||
Matrix (`MATRIX_HOMESERVER`, `MATRIX_ACCESS_TOKEN`, `MATRIX_ROOM_ID`): the
|
||||
notifier sends the room the same transitions as ntfy (per station, outlook,
|
||||
feed; not ntfy's per-topic summaries), each once, with its own `matrix:` keys
|
||||
in `notification_state` (seeded from the ntfy ones on first use, so nothing
|
||||
already announced is repeated). It replaces the `run.py --alert-check` cron
|
||||
job, which the install script comments out.
|
||||
|
||||
### Develop
|
||||
|
||||
```bash
|
||||
@@ -142,6 +177,9 @@ cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI ar
|
||||
sudo bash scripts/install_collector.sh # install + checks, nothing switched
|
||||
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
|
||||
sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies
|
||||
sudo bash scripts/install_collector.sh --check-api # compare the Rust API with FastAPI
|
||||
sudo bash scripts/install_collector.sh --cutover-api # Rust serves the website on :8000
|
||||
sudo bash scripts/install_collector.sh --rollback-api # FastAPI serves the website again
|
||||
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
|
||||
sudo bash scripts/install_collector.sh --rollback # Python does everything again
|
||||
journalctl -u ping-collector -f
|
||||
|
||||
Reference in new issue
Block a user