feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions
CI / Format & lint (push) Successful in 26s
CI / Test suite (push) Successful in 1m9s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 11m46s

Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.

Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).

scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.

Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).

install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
This commit is contained in:
grabowski committed 2026-10-18 16:08:06 +07:00
1 parent 1d1c000922
commit 779728e38b
32 files changed
+5692 -160

No files matched your search

+31 -33
View File
@@ -41,45 +41,43 @@ MATRIX_ROOM_ID=!your_room_id:matrix.org
GRAFANA_URL=http://localhost:3000
```
## Step 5: Test Configuration
## Step 5: Restart and check
The Matrix messages are sent by `ping-collector` (the Rust service, with
`FORECAST_IN_COLLECTOR=true`), together with the ntfy notifications:
```bash
# Test Matrix connection
uv run python run.py --alert-test
# Check system status (shows Matrix config)
uv run python run.py --status
# Run alert check
uv run python run.py --alert-check
sudo systemctl restart ping-collector
journalctl -u ping-collector | grep -i matrix # "Matrix notifications: room ... on ..."
# what would be sent right now (ntfy and Matrix), changing nothing:
sudo -u water-monitor /opt/thailand-water-monitor/bin/ping-collector notify --dry-run
```
## Example Alert Message
When thresholds are exceeded, you'll receive messages like:
## What the room receives
The same transitions as the ntfy topics (docs/NOTIFICATIONS.md), each once:
a station crossing its warning or danger level and back (0.10 m hysteresis),
the P.1 early-warning outlook turning on and off, and the gauge feed going
stale or recovering. ntfy's per-topic summaries ("Ping River: warning level
reached") are not repeated in the room. For example:
```
🌊 **WATER LEVEL ALERT**
⚠️ **Warning level at P.1**
Nawarat Bridge, Chiang Mai city: 3.75 m at 18 Oct 14:00, above the warning level of 3.70 m.
**Station:** P.1 (สถานีเชียงใหม่)
**Alert Type:** Critical Water Level
**Severity:** CRITICAL
**Current Level:** 6.75m
**Threshold:** 6.0m
**Difference:** +0.75m
**Discharge:** 450.2 cms
**Time:** 2025-09-26 14:30:00
📈 View dashboard: http://localhost:3000
📈 Dashboard: https://water.buildfor.life/
```
## Cron Job Setup (Optional)
Add to crontab for automatic alerting:
```bash
# Check water levels every 15 minutes
*/15 * * * * cd /path/to/monitor && uv run python run.py --alert-check >> alerts.log 2>&1
```
The room has its own delivery state (`matrix:` keys in `notification_state`),
so a Matrix outage is retried on Matrix only and does not re-send ntfy.
The old `run.py --alert-check` cron job (every 15 minutes, repeating an alert
for as long as a level stayed high) is replaced by this;
`scripts/install_collector.sh` comments it out of the crontab once
ping-collector sends Matrix, keeping a backup next to the app.
## Troubleshooting
- **403 Error**: Check Matrix access token is valid
- **Room Not Found**: Verify room ID includes `!` prefix and `:homeserver.com` suffix
- **No Alerts**: Check database has recent data with `uv run python run.py --status`
- **HTTP 401/403** in `journalctl -u ping-collector`: the access token is invalid
or the account is not in the room.
- **Room not found**: the room ID needs the `!` prefix and the `:homeserver`
suffix.
- **Nothing sent**: there was no transition; `ping-collector notify --dry-run`
shows the readings judged and what would be sent.
+6 -3
View File
@@ -48,7 +48,9 @@ message then says "ThaiWater telemetry; the RID report is not in yet". RID's
report regularly lags 2-10 h at night, which used to send "gauge feed stale"
around midnight and "recovered" in the morning. Now the feed is stale only
when neither source has a reading for 3 h, and RID alone lagging is reported
once after `RID_STALE_ALERT_HOURS` (18, 0 = off).
once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the
same transitions also go to a Matrix room, each once (per station, outlook,
feed; not the per-topic summaries), with their own `matrix:` state keys.
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
(leader only), right after the forecast precompute, so it sees exactly the
@@ -136,8 +138,9 @@ template.
## Why ntfy and not …
- **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
subscriber and a room invite; fine for a team, wrong for the public.
- **Matrix**: needs a homeserver account per subscriber and a room invite;
fine for a team, wrong for the public. The team room gets the same
transitions from ping-collector (docs/MATRIX_QUICK_START.md).
- **Gotify**: also self-hosted and light, but Android-only client and one
account per subscriber.
- **Email / SMS**: deliverability work, cost per message, no priority
+32 -9
View File
@@ -1,7 +1,8 @@
# A Rust rewrite: assessment
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17,
phase 2 (forecast + ntfy) since 2026-10-18; see
phase 2 (forecast + ntfy) since 2026-10-18; phase 3 (website/API + Matrix) is
built and parity-checked, cutover pending; see
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
## Short version
@@ -85,7 +86,7 @@ This slows model iteration, which is the most active part of the repo.
| 0 | Spike: P.1 features + HGB inference | done, bit-identical |
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
| 2 | Forecast crate + ntfy | in production since 2026-10-18 15:08 (`FORECAST_IN_COLLECTOR=true`); live check on production data: 48/48 rows identical, 96/96 probabilities bit-identical |
| 3 | Read API | |
| 3 | Website + read API in `ping-collector`, Matrix like ntfy | built 2026-10-18; locally 53 of 53 compared routes identical to FastAPI (`scripts/api_parity.py`); cutover with `install_collector.sh --cutover-api` |
| 4 | Cleanup | |
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
@@ -106,14 +107,36 @@ This slows model iteration, which is the most active part of the repo.
NaN at serving time (docs/FLOOD_FORECASTING.md section 5). The ntfy change:
ThaiWater mirror readings keep level alerts and the feed check current
while RID lags at night. The Python API then only serves.
3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
then a Caddy switch. Python keeps only training/evaluation.
4. **Cleanup:** drop the multi-backend adapters, demo databases and the
Python web stack.
3. **Website + read API** (`rust/collector/src/web`, axum): every FastAPI
route with the same parameters, JSON fields and order, status codes,
FastAPI-style errors (`{"detail": ...}`, pydantic's 422 list), cache
headers, gzip, CORS, Umami API tracking and the stale-while-revalidate
response caches (now with byte budgets). `/docs` is the same Swagger UI;
its OpenAPI document is generated from the route table whose parameter
declarations also do the validation. One process serves, collects,
forecasts and notifies (`WEB_LISTEN`), so the status file, the leader
election and the `forecast_current` hand-off between processes are gone
in effect. Checked by `scripts/api_parity.py`, which runs `ping-collector
serve` next to the FastAPI app on the same database and compares every
route (query values vary per run so FastAPI's caches answer fresh);
`--cutover-api` refuses to switch on a difference. Intended differences:
`/health` counts ThaiWater telemetry of the same gauges while RID lags (as
the notifier); `/scraping/status` adds the collector version and the last
forecast run; `/forecast` serves an old stored forecast marked
`X-Data-Stale` instead of recomputing (same process); `/sensors/thaiwater`
is cached for a minute; database errors serve the last good response
where FastAPI answered `[]`. Matrix: the notifier's transitions go to the
room too (per station, outlook, feed; not the per-topic summaries), with
their own `matrix:` state keys seeded from ntfy's, replacing the
`run.py --alert-check` cron job that repeated alerts every 15 minutes.
Python keeps training, evaluation and the monthly retrain timer.
4. **Cleanup:** drop FastAPI/uvicorn, the Python collector, notifier, CLI and
the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) and
demo databases; the Python environment shrinks to the training stack.
Open questions before phase 3: is Matrix alerting still used? Must `/docs`
(OpenAPI UI) stay? Is SQLite dev mode needed? Where does the monthly retrain
run if the server stops having a Python environment for the API?
Decided before phase 3 (2026-10-18): Matrix stays and behaves like ntfy;
`/docs` stays; SQLite dev mode goes; the monthly retrain stays on the same
machine (Python, `water-monitor-retrain.timer`).
## Cheaper alternative (stay Python)