diff --git a/.env.example b/.env.example index 93a54b7..cf53a1f 100644 --- a/.env.example +++ b/.env.example @@ -116,11 +116,19 @@ PUBLIC_URL=https://water.buildfor.life/ # telemetry is current (0 = never) RID_STALE_ALERT_HOURS=18 -# Matrix Alerting Configuration +# Matrix: ping-collector sends the ntfy transitions to this room too +# (docs/MATRIX_QUICK_START.md); off while token or room is empty MATRIX_HOMESERVER=https://matrix.org MATRIX_ACCESS_TOKEN= MATRIX_ROOM_ID= +# Website and API served by ping-collector (rust/collector/src/web): the +# address to listen on; empty = the Python app serves (port 8000). Set by +# scripts/install_collector.sh --cutover-api. +WEB_LISTEN= +# STATIC_DIR=src/static +# ADMIN_API_KEY= # X-API-Key for station editing, /config, /scrape/trigger + # Grafana Integration GRAFANA_URL=http://localhost:3000 diff --git a/CLAUDE.md b/CLAUDE.md index 0b6abc5..9b44b34 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,6 +41,10 @@ Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user notify constants: `uv run python -m src.ml.export --config` (a test fails otherwise) and rebuild the collector. Notify texts: `rust/collector/testdata/gen_notify_cases.py`. +- API routes live in `rust/collector/src/web` (phase 3). Until the FastAPI app is + removed, a route change goes into both, and `scripts/api_parity.py` (both + servers on one database) must report no differences. A route's parameters are + declared once (`web/params.rs`); that list validates and documents (/docs). ## Where things are @@ -51,7 +55,9 @@ Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API. - `rust/collector` (`ping-collector`): Rust data collector; with it deployed the Python app runs `COLLECTION_ENABLED=false`. With `FORECAST_IN_COLLECTOR=true` it - also forecasts (`rust/forecast`) and sends ntfy; the API serves its - `forecast_current`. Rewrite plan: `docs/RUST_REWRITE.md`. + also forecasts (`rust/forecast`) and sends ntfy + Matrix; with `WEB_LISTEN` it + serves the website and API (`rust/collector/src/web`, the port of + `src/web_api.py`; the dashboard stays `src/static/dashboard.html`). Rewrite + plan: `docs/RUST_REWRITE.md`. - `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md` the source catalog. diff --git a/docs/MATRIX_QUICK_START.md b/docs/MATRIX_QUICK_START.md index efcdb64..2e72d17 100644 --- a/docs/MATRIX_QUICK_START.md +++ b/docs/MATRIX_QUICK_START.md @@ -41,45 +41,43 @@ MATRIX_ROOM_ID=!your_room_id:matrix.org GRAFANA_URL=http://localhost:3000 ``` -## Step 5: Test Configuration +## Step 5: Restart and check +The Matrix messages are sent by `ping-collector` (the Rust service, with +`FORECAST_IN_COLLECTOR=true`), together with the ntfy notifications: + ```bash -# Test Matrix connection -uv run python run.py --alert-test - -# Check system status (shows Matrix config) -uv run python run.py --status - -# Run alert check -uv run python run.py --alert-check +sudo systemctl restart ping-collector +journalctl -u ping-collector | grep -i matrix # "Matrix notifications: room ... on ..." +# what would be sent right now (ntfy and Matrix), changing nothing: +sudo -u water-monitor /opt/thailand-water-monitor/bin/ping-collector notify --dry-run ``` -## Example Alert Message -When thresholds are exceeded, you'll receive messages like: +## What the room receives +The same transitions as the ntfy topics (docs/NOTIFICATIONS.md), each once: +a station crossing its warning or danger level and back (0.10 m hysteresis), +the P.1 early-warning outlook turning on and off, and the gauge feed going +stale or recovering. ntfy's per-topic summaries ("Ping River: warning level +reached") are not repeated in the room. For example: + ``` -🌊 **WATER LEVEL ALERT** +⚠️ **Warning level at P.1** +Nawarat Bridge, Chiang Mai city: 3.75 m at 18 Oct 14:00, above the warning level of 3.70 m. -**Station:** P.1 (สถานีเชียงใหม่) -**Alert Type:** Critical Water Level -**Severity:** CRITICAL - -**Current Level:** 6.75m -**Threshold:** 6.0m -**Difference:** +0.75m -**Discharge:** 450.2 cms - -**Time:** 2025-09-26 14:30:00 - -📈 View dashboard: http://localhost:3000 +📈 Dashboard: https://water.buildfor.life/ ``` -## Cron Job Setup (Optional) -Add to crontab for automatic alerting: -```bash -# Check water levels every 15 minutes -*/15 * * * * cd /path/to/monitor && uv run python run.py --alert-check >> alerts.log 2>&1 -``` +The room has its own delivery state (`matrix:` keys in `notification_state`), +so a Matrix outage is retried on Matrix only and does not re-send ntfy. + +The old `run.py --alert-check` cron job (every 15 minutes, repeating an alert +for as long as a level stayed high) is replaced by this; +`scripts/install_collector.sh` comments it out of the crontab once +ping-collector sends Matrix, keeping a backup next to the app. ## Troubleshooting -- **403 Error**: Check Matrix access token is valid -- **Room Not Found**: Verify room ID includes `!` prefix and `:homeserver.com` suffix -- **No Alerts**: Check database has recent data with `uv run python run.py --status` \ No newline at end of file +- **HTTP 401/403** in `journalctl -u ping-collector`: the access token is invalid + or the account is not in the room. +- **Room not found**: the room ID needs the `!` prefix and the `:homeserver` + suffix. +- **Nothing sent**: there was no transition; `ping-collector notify --dry-run` + shows the readings judged and what would be sent. diff --git a/docs/NOTIFICATIONS.md b/docs/NOTIFICATIONS.md index 52a5ce3..57d9db5 100644 --- a/docs/NOTIFICATIONS.md +++ b/docs/NOTIFICATIONS.md @@ -48,7 +48,9 @@ message then says "ThaiWater telemetry; the RID report is not in yet". RID's report regularly lags 2-10 h at night, which used to send "gauge feed stale" around midnight and "recovered" in the morning. Now the feed is stale only when neither source has a reading for 3 h, and RID alone lagging is reported -once after `RID_STALE_ALERT_HOURS` (18, 0 = off). +once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the +same transitions also go to a Matrix room, each once (per station, outlook, +feed; not the per-topic summaries), with their own `matrix:` state keys. With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process (leader only), right after the forecast precompute, so it sees exactly the @@ -136,8 +138,9 @@ template. ## Why ntfy and not … -- **Matrix** (`src/alerting.py`, still there): needs a homeserver account per - subscriber and a room invite; fine for a team, wrong for the public. +- **Matrix**: needs a homeserver account per subscriber and a room invite; + fine for a team, wrong for the public. The team room gets the same + transitions from ping-collector (docs/MATRIX_QUICK_START.md). - **Gotify**: also self-hosted and light, but Android-only client and one account per subscriber. - **Email / SMS**: deliverability work, cost per message, no priority diff --git a/docs/RUST_REWRITE.md b/docs/RUST_REWRITE.md index 9ceac16..288577c 100644 --- a/docs/RUST_REWRITE.md +++ b/docs/RUST_REWRITE.md @@ -1,7 +1,8 @@ # A Rust rewrite: assessment Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17, -phase 2 (forecast + ntfy) since 2026-10-18; see +phase 2 (forecast + ntfy) since 2026-10-18; phase 3 (website/API + Matrix) is +built and parity-checked, cutover pending; see [rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md). ## Short version @@ -85,7 +86,7 @@ This slows model iteration, which is the most active part of the repo. | 0 | Spike: P.1 features + HGB inference | done, bit-identical | | 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) | | 2 | Forecast crate + ntfy | in production since 2026-10-18 15:08 (`FORECAST_IN_COLLECTOR=true`); live check on production data: 48/48 rows identical, 96/96 probabilities bit-identical | -| 3 | Read API | | +| 3 | Website + read API in `ping-collector`, Matrix like ntfy | built 2026-10-18; locally 53 of 53 compared routes identical to FastAPI (`scripts/api_parity.py`); cutover with `install_collector.sh --cutover-api` | | 4 | Cleanup | | 0. **Spike: done.** P.1 features + HGB inference, bit-identical. @@ -106,14 +107,36 @@ This slows model iteration, which is the most active part of the repo. NaN at serving time (docs/FLOOD_FORECASTING.md section 5). The ntfy change: ThaiWater mirror readings keep level alerts and the feed check current while RID lags at night. The Python API then only serves. -3. **Read API** (~2k LOC): routes recorded from the Python API as golden files, - then a Caddy switch. Python keeps only training/evaluation. -4. **Cleanup:** drop the multi-backend adapters, demo databases and the - Python web stack. +3. **Website + read API** (`rust/collector/src/web`, axum): every FastAPI + route with the same parameters, JSON fields and order, status codes, + FastAPI-style errors (`{"detail": ...}`, pydantic's 422 list), cache + headers, gzip, CORS, Umami API tracking and the stale-while-revalidate + response caches (now with byte budgets). `/docs` is the same Swagger UI; + its OpenAPI document is generated from the route table whose parameter + declarations also do the validation. One process serves, collects, + forecasts and notifies (`WEB_LISTEN`), so the status file, the leader + election and the `forecast_current` hand-off between processes are gone + in effect. Checked by `scripts/api_parity.py`, which runs `ping-collector + serve` next to the FastAPI app on the same database and compares every + route (query values vary per run so FastAPI's caches answer fresh); + `--cutover-api` refuses to switch on a difference. Intended differences: + `/health` counts ThaiWater telemetry of the same gauges while RID lags (as + the notifier); `/scraping/status` adds the collector version and the last + forecast run; `/forecast` serves an old stored forecast marked + `X-Data-Stale` instead of recomputing (same process); `/sensors/thaiwater` + is cached for a minute; database errors serve the last good response + where FastAPI answered `[]`. Matrix: the notifier's transitions go to the + room too (per station, outlook, feed; not the per-topic summaries), with + their own `matrix:` state keys seeded from ntfy's, replacing the + `run.py --alert-check` cron job that repeated alerts every 15 minutes. + Python keeps training, evaluation and the monthly retrain timer. +4. **Cleanup:** drop FastAPI/uvicorn, the Python collector, notifier, CLI and + the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) and + demo databases; the Python environment shrinks to the training stack. -Open questions before phase 3: is Matrix alerting still used? Must `/docs` -(OpenAPI UI) stay? Is SQLite dev mode needed? Where does the monthly retrain -run if the server stops having a Python environment for the API? +Decided before phase 3 (2026-10-18): Matrix stays and behaves like ntfy; +`/docs` stays; SQLite dev mode goes; the monthly retrain stays on the same +machine (Python, `water-monitor-retrain.timer`). ## Cheaper alternative (stay Python) diff --git a/rust/Cargo.lock b/rust/Cargo.lock index 66a6dce..387056c 100644 --- a/rust/Cargo.lock +++ b/rust/Cargo.lock @@ -88,6 +88,18 @@ version = "1.0.104" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" +[[package]] +name = "async-compression" +version = "0.4.51" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d5755640120b24961f8231e05892e4a4f7e0d04e8aeca161508c9c397946970" +dependencies = [ + "compression-codecs", + "compression-core", + "pin-project-lite", + "tokio", +] + [[package]] name = "atoi" version = "2.0.0" @@ -109,6 +121,56 @@ version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" +[[package]] +name = "axum" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" +dependencies = [ + "axum-core", + "bytes", + "form_urlencoded", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "itoa", + "matchit", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "serde_core", + "serde_json", + "serde_path_to_error", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tower", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum-core" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "http-body-util", + "mime", + "pin-project-lite", + "sync_wrapper", + "tower-layer", + "tower-service", +] + [[package]] name = "base64" version = "0.22.1" @@ -237,6 +299,23 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "compression-codecs" +version = "0.4.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cae36683101b2899f48bd31f8db9839437356188e3b1ff23096e73dab09ba8fe" +dependencies = [ + "compression-core", + "flate2", + "memchr", +] + +[[package]] +name = "compression-core" +version = "0.4.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ebdda89edc534bf8d5dd2857562633b6495557e5846552ade994b99ed13c36a" + [[package]] name = "const-oid" version = "0.9.6" @@ -443,6 +522,12 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + [[package]] name = "foreign-types" version = "0.3.2" @@ -578,7 +663,7 @@ checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ "allocator-api2", "equivalent", - "foldhash", + "foldhash 0.1.5", ] [[package]] @@ -668,12 +753,24 @@ dependencies = [ "pin-project-lite", ] +[[package]] +name = "http-range-header" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9171a2ea8a68358193d15dd5d70c1c10a2afc3e7e4c5bc92bc9f025cebd7359c" + [[package]] name = "httparse" version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + [[package]] name = "hyper" version = "1.12.0" @@ -686,6 +783,7 @@ dependencies = [ "http", "http-body", "httparse", + "httpdate", "itoa", "pin-project-lite", "smallvec", @@ -979,6 +1077,12 @@ dependencies = [ "regex-automata", ] +[[package]] +name = "matchit" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" + [[package]] name = "md-5" version = "0.10.6" @@ -995,6 +1099,22 @@ version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mime_guess" +version = "2.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" +dependencies = [ + "mime", + "unicase", +] + [[package]] name = "miniz_oxide" version = "0.9.1" @@ -1197,9 +1317,11 @@ name = "ping-collector" version = "0.1.0" dependencies = [ "anyhow", + "axum", "chrono", "clap", "flate2", + "form_urlencoded", "log", "ping-forecast", "reqwest", @@ -1207,6 +1329,8 @@ dependencies = [ "serde_json", "sqlx", "tokio", + "tower", + "tower-http", "tracing", "tracing-subscriber", ] @@ -1566,6 +1690,8 @@ version = "1.0.154" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" dependencies = [ + "foldhash 0.2.0", + "indexmap", "itoa", "memchr", "serde", @@ -1573,6 +1699,17 @@ dependencies = [ "zmij", ] +[[package]] +name = "serde_path_to_error" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457" +dependencies = [ + "itoa", + "serde", + "serde_core", +] + [[package]] name = "serde_urlencoded" version = "0.7.1" @@ -2063,6 +2200,19 @@ dependencies = [ "tokio", ] +[[package]] +name = "tokio-util" +version = "0.7.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + [[package]] name = "tower" version = "0.5.3" @@ -2084,12 +2234,22 @@ version = "0.6.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" dependencies = [ + "async-compression", "bitflags", "bytes", + "futures-core", "futures-util", "http", "http-body", + "http-body-util", + "http-range-header", + "httpdate", + "mime", + "mime_guess", + "percent-encoding", "pin-project-lite", + "tokio", + "tokio-util", "tower", "tower-layer", "tower-service", @@ -2182,6 +2342,12 @@ version = "1.20.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" +[[package]] +name = "unicase" +version = "2.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "357cc3acc6a036009fd6c973ed009037c732d60d0b4f6c673e9041497482a28f" + [[package]] name = "unicode-bidi" version = "0.3.18" diff --git a/rust/Cargo.toml b/rust/Cargo.toml index 33f5529..6259c3c 100644 --- a/rust/Cargo.toml +++ b/rust/Cargo.toml @@ -13,17 +13,23 @@ repository = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor" [workspace.dependencies] anyhow = "1" +axum = { version = "0.8", default-features = false, features = ["http1", "tokio", "query", "json", "matched-path", "original-uri"] } chrono = { version = "0.4", default-features = false, features = ["std", "clock", "serde"] } clap = { version = "4", features = ["derive"] } flate2 = "1" +form_urlencoded = "1" log = "0.4" # native-tls (OpenSSL on Linux, SChannel on Windows), not rustls: app.rid.go.th (dams) # offers only TLS 1.2 CBC suites (ECDHE-RSA-AES256-SHA384), which rustls refuses. reqwest = { version = "0.12", default-features = false, features = ["native-tls", "json"] } serde = { version = "1", features = ["derive"] } -serde_json = { version = "1", features = ["float_roundtrip"] } +# preserve_order: JSON objects keep the order they were built in (API +# responses list their fields in the same order as the FastAPI app did) +serde_json = { version = "1", features = ["float_roundtrip", "preserve_order"] } sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "chrono", "macros"] } -tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "signal"] } +tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "signal", "net", "sync", "fs"] } +tower = { version = "0.5", default-features = false, features = ["util"] } +tower-http = { version = "0.6", features = ["compression-gzip", "cors", "fs"] } tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } diff --git a/rust/README.md b/rust/README.md index 021a9de..1a4a0de 100644 --- a/rust/README.md +++ b/rust/README.md @@ -7,6 +7,7 @@ costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md | --- | --- | --- | | `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 | | `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: in production since 2026-10-18 | +| `collector`: `src/web` | the FastAPI app (`src/web_api.py`): dashboard, read API, `/docs` | phase 3: built, cutover pending | ## ping-collector @@ -19,7 +20,10 @@ the database advances. ```text ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries # while RID is behind), HII/dams/rain once per clock hour; - # with FORECAST_IN_COLLECTOR=true also forecast + ntfy + # with FORECAST_IN_COLLECTOR=true also forecast + ntfy/Matrix; + # with WEB_LISTEN the website and API +ping-collector serve [--listen A] [--no-analytics] + # the website and API only (default 127.0.0.1:8001) ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing # and compares the RID fetch with the database ping-collector fill-gaps # re-fetch RID reports with holes @@ -39,7 +43,7 @@ only its line number, never its value. Variables already set (e.g. by systemd's `GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`, `ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6), `STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`), -`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`), +`SCRAPING_STATUS_FILE` (status JSON for `/scraping/status` when the website runs elsewhere), `LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false), `MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX` / `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18, @@ -105,6 +109,37 @@ Verified: `src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy outages); the Rust notifier sends the same 751 messages word for word. +## Website and API (phase 3) + +`src/web` serves what the FastAPI app served: the dashboard +(`STATIC_DIR`/dashboard.html, default `src/static`, loaded at start), the +crawler files, `/static`, `/docs` + `/redoc` + `/openapi.json`, and every API +route with the same parameters, JSON (field order included), status codes, +`{"detail": ...}` errors, Cache-Control, gzip, CORS (`CORS_ALLOW_ORIGINS`), +Umami API tracking (`UMAMI_*`) and response caches (`*_CACHE_TTL_SECONDS`). +Admin routes need `ADMIN_API_KEY`. With `WEB_LISTEN=0.0.0.0:8000` it runs +inside `ping-collector run` (one process collects, forecasts, notifies and +serves; `/scraping/status` and `POST /scrape/trigger` then talk to the +collection loop directly); `serve` runs it alone. + +- Parameters are declared once per route (`web/params.rs`): the same list + validates the request (pydantic-style 422s) and builds the OpenAPI + document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted. + The component schemas are FastAPI's (`web/openapi_components.json`). +- Python's output conventions are kept where they show: `isoformat()` + timestamps, `round()`, numpy's pairwise sums for the catchment means + (`web/pyfmt.rs`), the skill metrics' summation order. +- `scripts/api_parity.py --python URL --rust URL` compares the two live on + one database; `install_collector.sh --check-api` / `--cutover-api` run it + on the server against `ping-collector serve` on port 8001. + +Matrix (`MATRIX_HOMESERVER`, `MATRIX_ACCESS_TOKEN`, `MATRIX_ROOM_ID`): the +notifier sends the room the same transitions as ntfy (per station, outlook, +feed; not ntfy's per-topic summaries), each once, with its own `matrix:` keys +in `notification_state` (seeded from the ntfy ones on first use, so nothing +already announced is repeated). It replaces the `run.py --alert-check` cron +job, which the install script comments out. + ### Develop ```bash @@ -142,6 +177,9 @@ cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=, + /// MATRIX_*: the notifications also go to this room (off when unset). + pub matrix: Option, + pub web: WebConfig, } -fn var(name: &str) -> Option { - env::var(name) - .ok() - .map(|v| v.trim().to_string()) - .filter(|v| !v.is_empty()) +#[derive(Clone, Debug)] +pub struct MatrixConfig { + pub homeserver: String, + pub access_token: String, + pub room_id: String, } -fn flag(name: &str, default: bool) -> bool { - var(name).map_or(default, |v| { - matches!(v.to_lowercase().as_str(), "1" | "true" | "yes") - }) +/// The website and read API (src/web_api.py's settings, same names). +#[derive(Clone, Debug)] +pub struct WebConfig { + /// WEB_LISTEN: `run` also serves the website on this address + /// (e.g. 0.0.0.0:8000); unset = collection only. + pub listen: Option, + /// Dashboard, robots/llms/sitemap and /static files. + pub static_dir: PathBuf, + /// X-API-Key for the station editing, /config and /scrape/trigger; + /// unset = those endpoints are disabled. + pub admin_api_key: Option, + pub cors_allow_origins: Vec, + pub umami_api_url: String, + /// Empty = no server-side analytics. + pub umami_website_id: String, + pub umami_track_api: bool, + pub hii_cache_ttl_s: u64, + pub latest_cache_ttl_s: u64, + pub health_cache_ttl_s: u64, + /// A stored forecast older than this is served marked `X-Data-Stale`. + pub forecast_max_age_s: u64, + pub thaiwater_api_key: Option, + pub request_timeout_s: u64, } -fn number(name: &str, default: T) -> Result -where - T::Err: std::fmt::Display, -{ - match var(name) { - None => Ok(default), - Some(v) => v.parse().map_err(|e| anyhow::anyhow!("{name}={v:?}: {e}")), +/// Variable lookup (the process environment, or a map in tests). +struct Vars<'a>(&'a dyn Fn(&str) -> Option); + +impl Vars<'_> { + fn var(&self, name: &str) -> Option { + (self.0)(name) + .map(|v| v.trim().to_string()) + .filter(|v| !v.is_empty()) + } + + /// `os.getenv(name, default)`: a variable set to "" stays "". + fn raw_var(&self, name: &str, default: &str) -> String { + (self.0)(name).map_or_else(|| default.to_string(), |v| v.trim().to_string()) + } + + fn flag(&self, name: &str, default: bool) -> bool { + self.var(name).map_or(default, |v| { + matches!(v.to_lowercase().as_str(), "1" | "true" | "yes") + }) + } + + fn number(&self, name: &str, default: T) -> Result + where + T::Err: std::fmt::Display, + { + match self.var(name) { + None => Ok(default), + Some(v) => v.parse().map_err(|e| anyhow::anyhow!("{name}={v:?}: {e}")), + } } } @@ -101,52 +147,106 @@ pub fn normalize_database_url(url: &str) -> String { impl Config { pub fn from_env() -> Result { - let database_url = match var("POSTGRES_CONNECTION_STRING") { + Self::from_vars(&|name| env::var(name).ok()) + } + + pub fn from_vars(get: &dyn Fn(&str) -> Option) -> Result { + let v = Vars(get); + let database_url = match v.var("POSTGRES_CONNECTION_STRING") { Some(url) => normalize_database_url(&url), None => { - let password = var("POSTGRES_PASSWORD") + let password = v + .var("POSTGRES_PASSWORD") .context("POSTGRES_CONNECTION_STRING or POSTGRES_PASSWORD must be set")?; format!( "postgres://{}:{}@{}:{}/{}", - encode_userinfo(&var("POSTGRES_USER").unwrap_or_else(|| "postgres".into())), + encode_userinfo(&v.var("POSTGRES_USER").unwrap_or_else(|| "postgres".into())), encode_userinfo(&password), - var("POSTGRES_HOST").unwrap_or_else(|| "localhost".into()), - var("POSTGRES_PORT").unwrap_or_else(|| "5432".into()), - var("POSTGRES_DB").unwrap_or_else(|| "water_monitoring".into()), + v.var("POSTGRES_HOST").unwrap_or_else(|| "localhost".into()), + v.var("POSTGRES_PORT").unwrap_or_else(|| "5432".into()), + v.var("POSTGRES_DB") + .unwrap_or_else(|| "water_monitoring".into()), ) } }; - if let Some(db_type) = var("DB_TYPE") + if let Some(db_type) = v.var("DB_TYPE") && db_type != "postgresql" { bail!("ping-collector supports PostgreSQL only (DB_TYPE={db_type})"); } let cfg = Config { database_url, - rid_api_url: var("RID_API_URL").unwrap_or_else(|| RID_API_URL.into()), - hii_api_base: var("HII_API_BASE").unwrap_or_else(|| HII_API_BASE.into()), - dams_url: var("RID_DAMS_URL").unwrap_or_else(|| RID_DAMS_URL.into()), - openmeteo_url: var("OPENMETEO_URL").unwrap_or_else(|| OPENMETEO_URL.into()), - hii_enabled: flag("ENABLE_HII_COLLECTION", true), - dams_enabled: flag("ENABLE_RESERVOIR_COLLECTION", true), - rain_enabled: flag("ENABLE_RAIN_COLLECTION", true), - hii_basin_code: number("HII_BASIN_CODE", 6)?, - poll_offset_minutes: number("RID_POLL_OFFSET_MINUTES", 5)?, - retry_minutes: number("RID_RETRY_MINUTES", 10)?, - gap_fill_days_on_start: number("GAP_FILL_DAYS_ON_START", 14)?, - station_config_path: var("STATION_CONFIG_PATH") + rid_api_url: v.var("RID_API_URL").unwrap_or_else(|| RID_API_URL.into()), + hii_api_base: v.var("HII_API_BASE").unwrap_or_else(|| HII_API_BASE.into()), + dams_url: v.var("RID_DAMS_URL").unwrap_or_else(|| RID_DAMS_URL.into()), + openmeteo_url: v + .var("OPENMETEO_URL") + .unwrap_or_else(|| OPENMETEO_URL.into()), + hii_enabled: v.flag("ENABLE_HII_COLLECTION", true), + dams_enabled: v.flag("ENABLE_RESERVOIR_COLLECTION", true), + rain_enabled: v.flag("ENABLE_RAIN_COLLECTION", true), + hii_basin_code: v.number("HII_BASIN_CODE", 6)?, + poll_offset_minutes: v.number("RID_POLL_OFFSET_MINUTES", 5)?, + retry_minutes: v.number("RID_RETRY_MINUTES", 10)?, + gap_fill_days_on_start: v.number("GAP_FILL_DAYS_ON_START", 14)?, + station_config_path: v + .var("STATION_CONFIG_PATH") .unwrap_or_else(|| "stations.json".into()) .into(), - status_file: var("SCRAPING_STATUS_FILE").map(PathBuf::from), - forecast_enabled: flag("FORECAST_IN_COLLECTOR", false), - models_dir: var("MODELS_DIR").unwrap_or_else(|| "models".into()).into(), + status_file: v.var("SCRAPING_STATUS_FILE").map(PathBuf::from), + forecast_enabled: v.flag("FORECAST_IN_COLLECTOR", false), + models_dir: v + .var("MODELS_DIR") + .unwrap_or_else(|| "models".into()) + .into(), // as src/config.py: off unless NTFY_SERVER is set - ntfy_publish_url: var("NTFY_SERVER") - .map(|server| var("NTFY_PUBLISH_URL").unwrap_or(server)), - ntfy_token: var("NTFY_TOKEN"), - ntfy_topic_prefix: var("NTFY_TOPIC_PREFIX").unwrap_or_else(|| "ping".into()), - public_url: var("PUBLIC_URL").unwrap_or_else(|| "https://water.buildfor.life/".into()), - rid_stale_alert_hours: number("RID_STALE_ALERT_HOURS", 18.0)?, + ntfy_publish_url: v + .var("NTFY_SERVER") + .map(|server| v.var("NTFY_PUBLISH_URL").unwrap_or(server)), + ntfy_token: v.var("NTFY_TOKEN"), + ntfy_topic_prefix: v.var("NTFY_TOPIC_PREFIX").unwrap_or_else(|| "ping".into()), + public_url: v + .var("PUBLIC_URL") + .unwrap_or_else(|| "https://water.buildfor.life/".into()), + rid_stale_alert_hours: v.number("RID_STALE_ALERT_HOURS", 18.0)?, + ntfy_server: v.var("NTFY_SERVER"), + matrix: match (v.var("MATRIX_ACCESS_TOKEN"), v.var("MATRIX_ROOM_ID")) { + (Some(access_token), Some(room_id)) => Some(MatrixConfig { + homeserver: v + .var("MATRIX_HOMESERVER") + .unwrap_or_else(|| "https://matrix.org".into()), + access_token, + room_id, + }), + _ => None, + }, + web: WebConfig { + listen: v.var("WEB_LISTEN"), + static_dir: v + .var("STATIC_DIR") + .unwrap_or_else(|| "src/static".into()) + .into(), + admin_api_key: v.var("ADMIN_API_KEY"), + cors_allow_origins: v + .raw_var("CORS_ALLOW_ORIGINS", "") + .split(',') + .map(str::trim) + .filter(|o| !o.is_empty()) + .map(String::from) + .collect(), + umami_api_url: v + .var("UMAMI_API_URL") + .unwrap_or_else(|| "https://stats.buildfor.life/api/send".into()), + umami_website_id: v + .raw_var("UMAMI_WEBSITE_ID", "00b2be73-8f5f-4400-9029-3be852eb08f7"), + umami_track_api: v.flag("UMAMI_TRACK_API", true), + hii_cache_ttl_s: v.number("HII_CACHE_TTL_SECONDS", 120)?, + latest_cache_ttl_s: v.number("LATEST_CACHE_TTL_SECONDS", 45)?, + health_cache_ttl_s: v.number("HEALTH_CACHE_TTL_SECONDS", 30)?, + forecast_max_age_s: v.number("FORECAST_MAX_AGE_SECONDS", 7200)?, + thaiwater_api_key: v.var("THAIWATER_API_KEY"), + request_timeout_s: v.number("REQUEST_TIMEOUT", 30)?, + }, }; if cfg.poll_offset_minutes >= 60 { bail!("RID_POLL_OFFSET_MINUTES must be between 0 and 59"); diff --git a/rust/collector/src/db.rs b/rust/collector/src/db.rs index 63aa074..40cdab9 100644 --- a/rust/collector/src/db.rs +++ b/rust/collector/src/db.rs @@ -14,7 +14,7 @@ use sqlx::PgPool; use sqlx::postgres::PgPoolOptions; use std::collections::{BTreeMap, HashMap, HashSet}; -pub async fn connect(url: &str) -> Result { +pub async fn connect(url: &str, max_connections: u32) -> Result { use sqlx::ConnectOptions; use std::str::FromStr; // sqlx warns about any statement over 1 s; the whole-history gap scan @@ -23,7 +23,7 @@ pub async fn connect(url: &str) -> Result { .context("invalid database URL")? .log_slow_statements(log::LevelFilter::Warn, std::time::Duration::from_secs(30)); PgPoolOptions::new() - .max_connections(4) + .max_connections(max_connections) .acquire_timeout(std::time::Duration::from_secs(30)) .connect_with(options) .await diff --git a/rust/collector/src/db_tests.rs b/rust/collector/src/db_tests.rs index bf30fdb..04f714e 100644 --- a/rust/collector/src/db_tests.rs +++ b/rust/collector/src/db_tests.rs @@ -17,14 +17,14 @@ use std::collections::HashMap; use std::str::FromStr; use std::sync::Mutex; -struct TestDb { - pool: PgPool, +pub(crate) struct TestDb { + pub(crate) pool: PgPool, admin: PgPool, schema: String, } impl TestDb { - async fn new() -> Option { + pub(crate) async fn new() -> Option { let Ok(url) = std::env::var("TEST_DATABASE_URL") else { eprintln!("TEST_DATABASE_URL not set: skipping database test"); return None; @@ -58,7 +58,7 @@ impl TestDb { }) } - async fn drop(self) { + pub(crate) async fn drop(self) { self.pool.close().await; sqlx::query(&format!("DROP SCHEMA {} CASCADE", self.schema)) .execute(&self.admin) diff --git a/rust/collector/src/forecasting.rs b/rust/collector/src/forecasting.rs index f8ba940..e77d75b 100644 --- a/rust/collector/src/forecasting.rs +++ b/rust/collector/src/forecasting.rs @@ -130,12 +130,23 @@ pub fn summarize(s: &Snapshot) -> String { ) } +/// ntfy (NTFY_SERVER) and/or Matrix (MATRIX_ACCESS_TOKEN + MATRIX_ROOM_ID); +/// `None` when neither is configured. pub fn notifier(cfg: &Config, http: reqwest::Client) -> Option { - let url = cfg.ntfy_publish_url.clone()?; + let ntfy = cfg + .ntfy_publish_url + .clone() + .map(|publish_url| notify::Ntfy { + publish_url, + token: cfg.ntfy_token.clone(), + }); + if ntfy.is_none() && cfg.matrix.is_none() { + return None; + } Some(Notifier { http, - publish_url: url, - token: cfg.ntfy_token.clone(), + ntfy, + matrix: cfg.matrix.clone(), dashboard_url: cfg.public_url.clone(), settings: Settings { prefix: cfg.ntfy_topic_prefix.clone(), @@ -164,14 +175,27 @@ pub struct Forecaster { impl Forecaster { pub fn new(cfg: &Config, http: reqwest::Client) -> Self { let notifier = notifier(cfg, http); - match ¬ifier { - Some(n) => tracing::info!( + match notifier + .as_ref() + .and_then(|n| n.ntfy.as_ref().map(|t| (n, t))) + { + Some((n, t)) => tracing::info!( "ntfy notifications: publish to {}, topics {}-*", - n.publish_url, + t.publish_url, n.settings.prefix ), None => tracing::info!("ntfy notifications off (NTFY_SERVER not set)"), } + match &cfg.matrix { + Some(m) => tracing::info!( + "Matrix notifications: room {} on {}", + m.room_id, + m.homeserver + ), + None => tracing::info!( + "Matrix notifications off (MATRIX_ACCESS_TOKEN/MATRIX_ROOM_ID not set)" + ), + } Self { store: ModelStore::new(&cfg.models_dir), notifier, @@ -236,7 +260,7 @@ impl Forecaster { notified = sent.len(); if !sent.is_empty() { tracing::info!( - "ntfy: published {}", + "notifications: published {}", sent.iter() .map(|s| format!("{}: {}", s.topic, s.title)) .collect::>() @@ -244,7 +268,7 @@ impl Forecaster { ); } } - Err(e) => tracing::warn!("ntfy notify cycle failed: {e:#}"), + Err(e) => tracing::warn!("notify cycle failed: {e:#}"), } } Ok(RunInfo { diff --git a/rust/collector/src/main.rs b/rust/collector/src/main.rs index f7e683a..5994930 100644 --- a/rust/collector/src/main.rs +++ b/rust/collector/src/main.rs @@ -3,8 +3,9 @@ //! Replaces the collection half of the Python web app (RID hourly gauges, //! HII/ThaiWater rain + water level, RID dams, Open-Meteo rain) and, with //! FORECAST_IN_COLLECTOR=true, the flood forecast (rust/forecast) and the -//! ntfy notifications. Writes the same PostgreSQL tables as the Python app, -//! which then runs with `COLLECTION_ENABLED=false`. See docs/RUST_REWRITE.md. +//! ntfy/Matrix notifications, and with WEB_LISTEN the website and read API +//! (the port of the FastAPI app). Writes the same PostgreSQL tables as the +//! Python app did. See docs/RUST_REWRITE.md. mod config; mod dams; @@ -20,6 +21,7 @@ mod rid; mod service; mod stations; mod util; +mod web; use anyhow::{Result, bail}; use clap::{Parser, Subcommand}; @@ -38,8 +40,20 @@ struct Cli { #[derive(Subcommand)] enum Command { - /// Run the collection service (what systemd starts) + /// Run the service (what systemd starts): collection, with + /// FORECAST_IN_COLLECTOR=true forecasts and notifications, with WEB_LISTEN + /// the website and API Run, + /// Serve only the website and API (no collection), e.g. next to the + /// running service for a comparison + Serve { + /// Address to listen on [default: WEB_LISTEN, else 127.0.0.1:8001] + #[arg(long)] + listen: Option, + /// Do not report requests to Umami (comparison runs) + #[arg(long)] + no_analytics: bool, + }, /// One cycle of every source, then exit Once { /// Fetch and compare with the database, write nothing @@ -132,18 +146,33 @@ fn main() -> Result<()> { } async fn run(cli: Cli) -> Result<()> { - let cfg = config::Config::from_env()?; + let mut cfg = config::Config::from_env()?; + if let Command::Serve { + no_analytics: true, .. + } = cli.command + { + cfg.web.umami_track_api = false; + } tracing::info!( "ping-collector {} -> {}", env!("CARGO_PKG_VERSION"), cfg.redacted_database_url() ); - let pool = db::connect(&cfg.database_url).await?; + // the website answers concurrent requests; collection alone needs few + let serving = matches!(cli.command, Command::Serve { .. }) + || (matches!(cli.command, Command::Run) && cfg.web.listen.is_some()); + let pool = db::connect(&cfg.database_url, if serving { 12 } else { 4 }).await?; let stations = stations::Stations::load(&cfg.station_config_path)?; let collector = service::Collector::new(cfg, pool, stations)?; match cli.command { Command::Run => collector.run().await, + Command::Serve { listen, .. } => { + let listen = listen + .or_else(|| collector.cfg.web.listen.clone()) + .unwrap_or_else(|| "127.0.0.1:8001".into()); + collector.serve(listen).await + } Command::Once { dry_run } => collector.once(dry_run).await, Command::FillGaps { days } => { let days = match days.as_str() { @@ -157,7 +186,7 @@ async fn run(cli: Cli) -> Result<()> { let filled = rid::fill_gaps( &collector.rid, &collector.pool, - &collector.stations, + &collector.stations(), days, util::bangkok_now(), ) diff --git a/rust/collector/src/notify.rs b/rust/collector/src/notify.rs index 88a39c2..fe5e366 100644 --- a/rust/collector/src/notify.rs +++ b/rust/collector/src/notify.rs @@ -1,5 +1,8 @@ -//! Public flood notifications over ntfy: the port of src/notify.py, run by -//! the collector when FORECAST_IN_COLLECTOR=true. +//! Public flood notifications over ntfy, and the same transitions to a +//! Matrix room: the port of src/notify.py, run by the collector when +//! FORECAST_IN_COLLECTOR=true. (Matrix replaces the old `run.py +//! --alert-check` cron job, which repeated an alert on every run while a +//! level stayed high.) //! //! Same topics, texts, hysteresis, capacity guard, outlook and persisted //! state keys (`notification_state`) as Python, so a cutover neither repeats @@ -9,6 +12,7 @@ //! "stale" only when neither source has a reading for 3 h; RID lagging alone //! is reported separately after RID_STALE_ALERT_HOURS. +use crate::config::MatrixConfig; use crate::db::{self, LatestReading}; use anyhow::Result; use chrono::{Duration, NaiveDateTime}; @@ -425,22 +429,105 @@ pub fn evaluate( plan } -/// Posts to ntfy and records delivered transitions. -pub struct Notifier { - pub http: reqwest::Client, +/// ntfy publishing target. +pub struct Ntfy { pub publish_url: String, pub token: Option, +} + +/// State keys of the Matrix channel: the same transitions as ntfy, tracked +/// separately so a failed delivery on one channel is retried on that one +/// only. +const MATRIX_PREFIX: &str = "matrix:"; + +/// Posts to ntfy and Matrix and records delivered transitions. +pub struct Notifier { + pub http: reqwest::Client, + pub ntfy: Option, + pub matrix: Option, pub dashboard_url: String, pub settings: Settings, } +/// ntfy tag shortcodes the notifications use, as emoji for Matrix. +fn emoji(tags: &[String]) -> &'static str { + match tags.first().map(String::as_str) { + Some("rotating_light") => "🚨", + Some("warning") => "⚠️", + Some("arrow_down") => "⬇️", + Some("white_check_mark") => "✅", + Some("crystal_ball") => "🔮", + Some("hourglass") => "⏳", + _ => "📊", + } +} + +/// The Matrix message: the ntfy title and text, in the small Markdown +/// subset `markdown_to_html` renders. +pub fn matrix_text(n: &Notification, dashboard_url: &str) -> String { + format!( + "{} **{}**\n{}\n\n📈 Dashboard: {dashboard_url}", + emoji(&n.tags), + n.title, + n.message + ) +} + +/// `**bold**` removed, for Matrix's plain-text `body`. +pub fn strip_markdown(text: &str) -> String { + text.replace("**", "") +} + +/// `alerting.markdown_to_matrix_html`: escape, `**bold**`, bare URLs as +/// links, newlines as `
`. +pub fn markdown_to_html(text: &str) -> String { + let escaped = text + .replace('&', "&") + .replace('<', "<") + .replace('>', ">"); + let mut bold = String::new(); + let mut parts = escaped.split("**").peekable(); + let mut open = false; + while let Some(part) = parts.next() { + bold.push_str(part); + if parts.peek().is_some() { + bold.push_str(if open { "" } else { "" }); + open = !open; + } + } + if open { + // an unpaired ** stays literal + if let Some(i) = bold.rfind("") { + bold.replace_range(i..i + "".len(), "**"); + } + } + let mut out = String::new(); + let mut rest = bold.as_str(); + while let Some(i) = rest + .find("http://") + .into_iter() + .chain(rest.find("https://")) + .min() + { + out.push_str(&rest[..i]); + let url_len = rest[i..] + .find(|c: char| c.is_whitespace() || c == '<') + .unwrap_or(rest.len() - i); + let url = &rest[i..i + url_len]; + out.push_str(&format!("{url}")); + rest = &rest[i + url_len..]; + } + out.push_str(rest); + out.replace('\n', "
") +} + impl Notifier { - pub async fn publish(&self, n: &Notification) -> bool { + pub async fn publish(&self, ntfy: &Ntfy, n: &Notification) -> bool { let mut req = self .http .post(format!( "{}/{}", - self.publish_url.trim_end_matches('/'), + ntfy.publish_url.trim_end_matches('/'), n.topic )) .timeout(std::time::Duration::from_secs(10)) @@ -455,7 +542,7 @@ impl Notifier { if !n.tags.is_empty() { req = req.header("Tags", n.tags.join(",")); } - if let Some(t) = &self.token { + if let Some(t) = &ntfy.token { req = req.bearer_auth(t); } match req.send().await { @@ -477,7 +564,106 @@ impl Notifier { } } - /// Evaluate and publish; returns the notifications delivered. + /// One Matrix room message (`MatrixNotifier.send_message`). + pub async fn send_matrix(&self, m: &MatrixConfig, n: &Notification) -> bool { + static TXN: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + let text = matrix_text(n, &self.dashboard_url); + let txn = format!( + "ping-{}-{}", + chrono::Utc::now().timestamp_micros(), + TXN.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + ); + let url = format!( + "{}/_matrix/client/v3/rooms/{}/send/m.room.message/{txn}", + m.homeserver.trim_end_matches('/'), + m.room_id + ); + let body = serde_json::json!({ + "msgtype": "m.text", + "body": strip_markdown(&text), + "format": "org.matrix.custom.html", + "formatted_body": markdown_to_html(&text), + }); + match self + .http + .put(url) + .bearer_auth(&m.access_token) + .timeout(std::time::Duration::from_secs(10)) + .json(&body) + .send() + .await + { + Ok(r) if r.status().is_success() => true, + Ok(r) => { + let status = r.status(); + let body = r.text().await.unwrap_or_default(); + tracing::warn!( + "Matrix: HTTP {status} {}", + body.chars().take(120).collect::() + ); + false + } + Err(e) => { + tracing::warn!("Matrix: {e}"); + false + } + } + } + + /// The Matrix channel's states (keys without the prefix). On first use + /// they start as a copy of the ntfy states, so what ntfy already + /// announced is not sent again. + pub async fn matrix_states( + &self, + pool: &PgPool, + states: &HashMap, + now: NaiveDateTime, + seed: bool, + ) -> Result> { + let own: HashMap = states + .iter() + .filter_map(|(k, v)| { + k.strip_prefix(MATRIX_PREFIX) + .map(|k| (k.to_string(), v.clone())) + }) + .collect(); + if !own.is_empty() { + return Ok(own); + } + let copy: HashMap = states + .iter() + .filter(|(k, _)| !k.starts_with(MATRIX_PREFIX)) + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + if seed { + for (k, v) in © { + db::set_notification_state(pool, &format!("{MATRIX_PREFIX}{k}"), v, None, now) + .await?; + } + db::set_notification_state(pool, "matrix:seeded", "yes", None, now).await?; + } + Ok(copy) + } + + /// What the Matrix channel would send: the transitions that carry a + /// state (per station, outlook, feed). The per-topic summaries ntfy also + /// gets ("Ping River: warning level reached") repeat those, so the room + /// does not get them. + pub fn matrix_plan( + &self, + readings: &[Reading], + forecasts: &[Forecast], + states: &HashMap, + now: NaiveDateTime, + ) -> Vec { + evaluate(readings, forecasts, states, &self.settings, now) + .into_iter() + .filter(|p| p.state.is_some()) + .collect() + } + + /// Evaluate and publish; returns the notifications delivered (Matrix + /// messages with topic "matrix"). pub async fn run( &self, pool: &PgPool, @@ -487,14 +673,32 @@ impl Notifier { ) -> Result> { let states = db::notification_states(pool).await?; let mut sent = vec![]; - for p in evaluate(readings, forecasts, &states, &self.settings, now) { - if !self.publish(&p.notification).await { - continue; + if let Some(ntfy) = &self.ntfy { + for p in evaluate(readings, forecasts, &states, &self.settings, now) { + if !self.publish(ntfy, &p.notification).await { + continue; + } + if let Some((key, state, value)) = &p.state { + db::set_notification_state(pool, key, state, *value, now).await?; + } + sent.push(p.notification); } - if let Some((key, state, value)) = &p.state { - db::set_notification_state(pool, key, state, *value, now).await?; + } + if let Some(m) = &self.matrix { + let mstates = self.matrix_states(pool, &states, now, true).await?; + for p in self.matrix_plan(readings, forecasts, &mstates, now) { + if !self.send_matrix(m, &p.notification).await { + continue; + } + if let Some((key, state, value)) = &p.state { + let key = format!("{MATRIX_PREFIX}{key}"); + db::set_notification_state(pool, &key, state, *value, now).await?; + } + sent.push(Notification { + topic: "matrix".into(), + ..p.notification + }); } - sent.push(p.notification); } Ok(sent) } @@ -505,6 +709,28 @@ mod tests { use super::*; use chrono::NaiveDate; + #[test] + fn matrix_message_rendering() { + let n = Notification { + topic: "ping-p1-warning".into(), + title: "Warning level at P.1".into(), + message: + "Nawarat Bridge: 3.75 m at 18 Oct 14:00, above the warning level of 3.70 m & y." + .into(), + priority: 4, + tags: vec!["warning".into(), "P.1".into()], + }; + let text = matrix_text(&n, "https://water.buildfor.life/"); + assert_eq!( + markdown_to_html(&text), + "⚠️ Warning level at P.1
Nawarat Bridge: 3.75 m at 18 Oct 14:00, \ + above the warning level of 3.70 m <x> & y.

📈 Dashboard: \ + https://water.buildfor.life/" + ); + assert!(strip_markdown(&text).starts_with("⚠️ Warning level at P.1\nNawarat")); + assert_eq!(markdown_to_html("a ** b"), "a ** b"); + } + fn now() -> NaiveDateTime { NaiveDate::from_ymd_opt(2026, 10, 1) .unwrap() diff --git a/rust/collector/src/service.rs b/rust/collector/src/service.rs index a81149a..ae9114f 100644 --- a/rust/collector/src/service.rs +++ b/rust/collector/src/service.rs @@ -16,11 +16,17 @@ use ping_forecast::ModelStore; use ping_forecast::rain::RainSeries; use serde_json::json; use sqlx::PgPool; +use std::sync::{Arc, RwLock}; pub struct Collector { pub cfg: Config, pub pool: PgPool, - pub stations: Stations, + /// Shared with the website (the station CRUD API edits it). + pub stations: Arc>, + /// The last status written, shared with /scraping/status. + pub status: Arc>, + /// POST /scrape/trigger: poll now. + pub trigger: Arc, pub rid: RidClient, hii: HiiClient, dams: DamsClient, @@ -61,10 +67,36 @@ impl Collector { http, cfg, pool, - stations, + stations: Arc::new(RwLock::new(stations)), + status: Arc::new(RwLock::new(json!({"is_running": false}))), + trigger: Arc::new(tokio::sync::Notify::new()), }) } + /// The station mapping as of now (the API may edit it between cycles). + pub fn stations(&self) -> Stations { + self.stations.read().unwrap().clone() + } + + /// The website/API state over this collector's pool, stations and status. + pub fn web_state(&self, in_process: bool) -> crate::web::Shared { + crate::web::AppState::new( + self.cfg.clone(), + self.pool.clone(), + self.http.clone(), + self.stations.clone(), + in_process.then(|| self.status.clone()), + in_process.then(|| self.trigger.clone()), + ) + } + + /// `ping-collector serve`: the website and API only; /scraping/status + /// reads the status file the collector service writes. + pub async fn serve(&self, listen: String) -> Result<()> { + db::ensure_schema(&self.pool).await?; + crate::web::serve(self.web_state(false), listen, shutdown_signal()).await + } + /// HII rain + water level, RID dams (today and yesterday), Open-Meteo. /// Each source is failure-isolated. pub async fn collect_hourly(&self, now: NaiveDateTime) { @@ -107,9 +139,6 @@ impl Collector { } fn write_status(&self, stats: &Stats, running: bool) { - let Some(path) = &self.cfg.status_file else { - return; - }; let body = json!({ "is_running": running, "total_runs": stats.total_runs, @@ -129,6 +158,10 @@ impl Collector { "notifications_sent": f.notified, })), }); + *self.status.write().unwrap() = body.clone(); + let Some(path) = &self.cfg.status_file else { + return; + }; let tmp = path.with_extension("json.tmp"); let result = std::fs::write(&tmp, body.to_string()).and_then(|_| std::fs::rename(&tmp, path)); @@ -144,14 +177,40 @@ impl Collector { db::ensure_schema(&self.pool).await?; let shutdown = shutdown_signal(); tokio::pin!(shutdown); + // the website, when WEB_LISTEN is set: stopped with the collector + let (stop_web, web_stopped) = tokio::sync::watch::channel(false); + let web = self.cfg.web.listen.clone().map(|listen| { + let state = self.web_state(true); + let mut rx = web_stopped.clone(); + tokio::spawn(async move { + let stop = async move { + let _ = rx.wait_for(|stop| *stop).await; + }; + if let Err(e) = crate::web::serve(state, listen, stop).await { + tracing::error!("website stopped: {e:#}"); + } + }) + }); + let result = self.collect_loop(&mut shutdown).await; + let _ = stop_web.send(true); + if let Some(handle) = web { + let _ = tokio::time::timeout(std::time::Duration::from_secs(10), handle).await; + } + result + } + async fn collect_loop( + &self, + shutdown: &mut std::pin::Pin<&mut impl std::future::Future>, + ) -> Result<()> { if self.cfg.gap_fill_days_on_start > 0 { let days = Some(self.cfg.gap_fill_days_on_start); + let stations = self.stations(); tokio::select! { - r = rid::fill_gaps(&self.rid, &self.pool, &self.stations, days, bangkok_now()) => { + r = rid::fill_gaps(&self.rid, &self.pool, &stations, days, bangkok_now()) => { if let Err(e) = r { tracing::error!("startup gap fill failed: {e:#}") } } - _ = &mut shutdown => return Ok(()), + _ = &mut *shutdown => return Ok(()), } } @@ -168,7 +227,7 @@ impl Collector { let start = bangkok_now(); self.write_status(&stats, true); let outcome: CycleOutcome = - rid::run_cycle(&self.rid, &self.pool, &self.stations, start).await; + rid::run_cycle(&self.rid, &self.pool, &self.stations(), start).await; stats.total_runs += 1; stats.last_run = Some(start); if outcome.current { @@ -217,7 +276,8 @@ impl Collector { .max(std::time::Duration::from_secs(30)); tokio::select! { _ = tokio::time::sleep(wait) => {} - _ = &mut shutdown => { + _ = self.trigger.notified() => tracing::info!("collection triggered via the API"), + _ = &mut *shutdown => { tracing::info!("shutdown signal received"); return Ok(()); } @@ -231,7 +291,7 @@ impl Collector { let now = bangkok_now(); if !dry_run { db::ensure_schema(&self.pool).await?; - let o = rid::run_cycle(&self.rid, &self.pool, &self.stations, now).await; + let o = rid::run_cycle(&self.rid, &self.pool, &self.stations(), now).await; println!( "RID: reports {:?}, {} fetched, {} saved, latest {:?}, current hour in: {}{}", o.dates, @@ -348,7 +408,9 @@ impl Collector { ); } let Some(notifier) = forecasting::notifier(&self.cfg, self.http.clone()) else { - bail!("ntfy is off: NTFY_SERVER is not set"); + bail!( + "notifications are off: neither NTFY_SERVER nor MATRIX_ACCESS_TOKEN/MATRIX_ROOM_ID is set" + ); }; let mut store = ModelStore::new(&self.cfg.models_dir); let models = store.refresh()?; @@ -370,15 +432,32 @@ impl Collector { db::ensure_schema(&self.pool).await?; if dry_run { let states = db::notification_states(&self.pool).await?; - let plan = - notify::evaluate(&readings, &snap.forecasts, &states, ¬ifier.settings, now); + let mut plan = vec![]; + if notifier.ntfy.is_some() { + plan.extend( + notify::evaluate(&readings, &snap.forecasts, &states, ¬ifier.settings, now) + .into_iter() + .map(|p| (p.notification.topic.clone(), p)), + ); + } + if notifier.matrix.is_some() { + let mstates = notifier + .matrix_states(&self.pool, &states, now, false) + .await?; + plan.extend( + notifier + .matrix_plan(&readings, &snap.forecasts, &mstates, now) + .into_iter() + .map(|p| ("matrix".to_string(), p)), + ); + } if plan.is_empty() { println!("no transitions: nothing would be published"); } - for p in plan { + for (to, p) in plan { println!( - "would publish {}: {} -- {}", - p.notification.topic, p.notification.title, p.notification.message + "would publish {to}: {} -- {}", + p.notification.title, p.notification.message ); } } else { diff --git a/rust/collector/src/stations.rs b/rust/collector/src/stations.rs index 4bfa9cc..faafb64 100644 --- a/rust/collector/src/stations.rs +++ b/rust/collector/src/stations.rs @@ -4,13 +4,13 @@ //! else the defaults bundled in src/data/stations.json. use anyhow::{Context, Result}; -use serde::Deserialize; +use serde::{Deserialize, Serialize}; use std::collections::BTreeMap; use std::path::Path; const BUNDLED: &str = include_str!("../../../src/data/stations.json"); -#[derive(Clone, Debug, Deserialize)] +#[derive(Clone, Debug, Deserialize, Serialize)] pub struct StationInfo { pub code: String, pub thai_name: String, @@ -55,6 +55,46 @@ impl Stations { Ok(Stations(map)) } + pub fn iter(&self) -> impl Iterator { + self.0.iter() + } + + pub fn info(&self, n: u32) -> Option<&StationInfo> { + self.0.get(&n) + } + + pub fn info_mut(&mut self, n: u32) -> Option<&mut StationInfo> { + self.0.get_mut(&n) + } + + /// The id a new station gets (the station CRUD API): one past the highest. + pub fn next_id(&self) -> u32 { + self.0.keys().max().map_or(1, |m| m + 1) + } + + pub fn insert(&mut self, n: u32, info: StationInfo) { + self.0.insert(n, info); + } + + pub fn remove(&mut self, n: u32) -> Option { + self.0.remove(&n) + } + + /// Persist to the runtime file as the Python scraper's `save_stations` + /// (indent 2, UTF-8, trailing newline), atomically. + pub fn save(&self, path: &Path) -> Result<()> { + // numeric key order ("10" after "9"), as the Python mapping kept it + let mut obj = serde_json::Map::new(); + for (k, v) in &self.0 { + obj.insert(k.to_string(), serde_json::to_value(v)?); + } + let text = serde_json::to_string_pretty(&serde_json::Value::Object(obj))? + "\n"; + let tmp = path.with_extension("json.tmp"); + std::fs::write(&tmp, text).with_context(|| format!("writing {}", tmp.display()))?; + std::fs::rename(&tmp, path).with_context(|| format!("replacing {}", path.display()))?; + Ok(()) + } + /// Metadata for report column `n`, with the Python fallback for unknown /// columns (`P.{19+n}` / `Station n`). pub fn get(&self, n: u32) -> StationInfo { @@ -81,4 +121,28 @@ mod tests { assert_eq!(s.get(8).code, "P.1"); assert_eq!(s.get(99).code, "P.118"); // Python fallback } + + #[test] + fn save_round_trips() { + let mut s = Stations::bundled(); + let id = s.next_id(); + assert_eq!(id, 17); + let mut extra = s.get(8); + extra.code = "P.999".into(); + s.insert(id, extra); + let dir = std::env::temp_dir().join(format!("ping-stations-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("stations.json"); + s.save(&path).unwrap(); + let text = std::fs::read_to_string(&path).unwrap(); + assert!( + text.starts_with("{\n \"1\": {\n \"code\": \"P.20\","), + "{text}" + ); + assert!(text.contains("\"thai_name\": \"บ้านเชียงดาว\"")); + let back = Stations::load(&path).unwrap(); + assert_eq!(back.info(17).unwrap().code, "P.999"); + assert_eq!(back.iter().count(), 17); + std::fs::remove_dir_all(&dir).unwrap(); + } } diff --git a/rust/collector/src/web/cache.rs b/rust/collector/src/web/cache.rs new file mode 100644 index 0000000..58a32ba --- /dev/null +++ b/rust/collector/src/web/cache.rs @@ -0,0 +1,244 @@ +//! Stale-while-revalidate response cache, as `_cached_swr` in src/web_api.py: +//! +//! - fresh entry: served; +//! - expired entry (no older than `max_stale`): served at once, marked stale, +//! and ONE background refresh is started; +//! - no entry: computed inline, single-flight per key (concurrent misses +//! wait for the one computation instead of piling queries onto Postgres); +//! - a failed computation serves the last good value, whatever its age; +//! - empty results are not cached, so recovery is immediate; +//! - a byte budget per cache: the oldest entries go first (query parameters +//! are client-controlled, so keys are unbounded, and one history response +//! can be 10 MB). + +use super::error::ApiError; +use std::collections::{HashMap, HashSet}; +use std::future::Future; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +/// The size of a cached value, for the budget. +pub trait Weigh { + fn weight(&self) -> usize; +} + +impl Weigh for axum::body::Bytes { + fn weight(&self) -> usize { + self.len() + } +} + +impl Weigh for (axum::body::Bytes, bool) { + fn weight(&self) -> usize { + self.0.len() + } +} + +#[cfg(test)] +impl Weigh for u32 { + fn weight(&self) -> usize { + 4 + } +} + +pub struct Swr { + budget: usize, + entries: Mutex>, + flights: Mutex>>>, + refreshing: Mutex>, +} + +/// What a computation returns: the value and whether to keep it. +pub type Computed = Result<(T, bool), ApiError>; + +impl Swr { + /// A cache holding at most `budget` bytes. + pub fn new(budget: usize) -> Arc { + Arc::new(Self { + budget, + entries: Mutex::new(HashMap::new()), + flights: Mutex::new(HashMap::new()), + refreshing: Mutex::new(HashSet::new()), + }) + } + + fn entry(&self, key: &str) -> Option<(Instant, T)> { + self.entries.lock().unwrap().get(key).cloned() + } + + /// `(value, served_stale)` + pub async fn get( + self: &Arc, + key: String, + ttl: Duration, + max_stale: Option, + compute: F, + ) -> Result<(T, bool), ApiError> + where + F: Fn() -> Fut + Send + Sync + 'static, + Fut: Future> + Send + 'static, + { + if let Some((at, v)) = self.entry(&key) { + let age = at.elapsed(); + if age < ttl { + return Ok((v, false)); + } + if max_stale.is_none_or(|m| age <= m) { + let start = self.refreshing.lock().unwrap().insert(key.clone()); + if start { + let me = self.clone(); + tokio::spawn(async move { + if let Err(e) = me.compute(&key, ttl, &compute).await { + tracing::warn!("{key}: background refresh failed: {}", e.detail); + } + me.refreshing.lock().unwrap().remove(&key); + }); + } + return Ok((v, true)); + } + } + self.compute(&key, ttl, &compute).await + } + + async fn compute( + &self, + key: &str, + ttl: Duration, + compute: &F, + ) -> Result<(T, bool), ApiError> + where + F: Fn() -> Fut, + Fut: Future>, + { + let flight = self + .flights + .lock() + .unwrap() + .entry(key.to_string()) + .or_default() + .clone(); + let _guard = flight.lock().await; + let previous = self.entry(key); + if let Some((at, v)) = &previous + && at.elapsed() < ttl + { + return Ok((v.clone(), false)); + } + match compute().await { + Ok((v, keep)) => { + if keep && v.weight() <= self.budget { + let mut entries = self.entries.lock().unwrap(); + entries.insert(key.to_string(), (Instant::now(), v.clone())); + let mut total: usize = entries.values().map(|(_, v)| v.weight()).sum(); + if total > self.budget { + let mut ages: Vec<(Instant, String)> = entries + .iter() + .map(|(k, (at, _))| (*at, k.clone())) + .collect(); + ages.sort(); + for (_, k) in ages { + if total <= self.budget { + break; + } + if let Some((_, old)) = entries.remove(&k) { + total -= old.weight(); + } + } + } + } + Ok((v, false)) + } + Err(e) => match previous { + Some((at, v)) => { + tracing::warn!( + "{key}: recompute failed ({}); serving {}s-stale copy", + e.detail, + at.elapsed().as_secs() + ); + Ok((v, true)) + } + None => Err(e), + }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicU32, Ordering}; + + #[tokio::test] + async fn fresh_stale_and_error_fallback() { + let cache: Arc> = Swr::new(1024); + let calls = Arc::new(AtomicU32::new(0)); + let fail = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let compute = { + let (calls, fail) = (calls.clone(), fail.clone()); + move || { + let (calls, fail) = (calls.clone(), fail.clone()); + async move { + let n = calls.fetch_add(1, Ordering::SeqCst) + 1; + if fail.load(Ordering::SeqCst) { + Err(ApiError::new(500, "down")) + } else { + Ok((n, true)) + } + } + } + }; + let ttl = Duration::from_millis(40); + let key = || "k".to_string(); + assert_eq!( + cache.get(key(), ttl, None, compute.clone()).await.unwrap(), + (1, false) + ); + assert_eq!( + cache.get(key(), ttl, None, compute.clone()).await.unwrap(), + (1, false) + ); + tokio::time::sleep(Duration::from_millis(60)).await; + // expired: the old value at once, refreshed in the background + assert_eq!( + cache.get(key(), ttl, None, compute.clone()).await.unwrap(), + (1, true) + ); + tokio::time::sleep(Duration::from_millis(10)).await; + assert_eq!( + cache.get(key(), ttl, None, compute.clone()).await.unwrap(), + (2, false) + ); + // older than max_stale and the database is down: last good value + tokio::time::sleep(Duration::from_millis(60)).await; + fail.store(true, Ordering::SeqCst); + let max = Some(Duration::from_millis(1)); + assert_eq!( + cache.get(key(), ttl, max, compute.clone()).await.unwrap(), + (2, true) + ); + // nothing cached yet: the error + assert!(cache.get("other".into(), ttl, max, compute).await.is_err()); + } + + #[tokio::test] + async fn budget_evicts_the_oldest() { + let cache: Arc> = Swr::new(8); // two u32 entries + let ttl = Duration::from_secs(60); + for (i, k) in ["a", "b", "c"].iter().enumerate() { + let v = i as u32; + cache + .get( + k.to_string(), + ttl, + None, + move || async move { Ok((v, true)) }, + ) + .await + .unwrap(); + tokio::time::sleep(Duration::from_millis(2)).await; + } + let entries = cache.entries.lock().unwrap(); + assert_eq!(entries.len(), 2); + assert!(!entries.contains_key("a")); + } +} diff --git a/rust/collector/src/web/data.rs b/rust/collector/src/web/data.rs new file mode 100644 index 0000000..50038bc --- /dev/null +++ b/rust/collector/src/web/data.rs @@ -0,0 +1,357 @@ +//! RID gauge readings: /measurements/*, /api/stats. + +use super::error::{ApiError, ApiResult, ok, to_body}; +use super::params::{self, Param}; +use super::pyfmt::{end_of_day, isoformat, py_round, start_of_day}; +use super::{Shared, secs}; +use crate::util::bangkok_now; +use axum::extract::{Path, RawQuery, State}; +use axum::http::HeaderMap; +use chrono::{NaiveDateTime, TimeDelta}; +use serde_json::{Map, Value, json}; +use sqlx::postgres::PgRow; +use sqlx::{Column, Row, TypeInfo}; +use std::time::Duration; + +/// A result row as a JSON object: column names as keys (SQL aliases), in +/// select order. NUMERIC columns must be cast to float8 in the query (as +/// Python's `float(Decimal)`), timestamps become `isoformat()` strings. +pub fn row_json(row: &PgRow) -> Map { + let mut m = Map::new(); + for (i, col) in row.columns().iter().enumerate() { + let v = match col.type_info().name() { + "INT2" => json!(row.try_get::, _>(i).ok().flatten()), + "INT4" => json!(row.try_get::, _>(i).ok().flatten()), + "INT8" => json!(row.try_get::, _>(i).ok().flatten()), + "FLOAT4" => super::pyfmt::num( + row.try_get::, _>(i) + .ok() + .flatten() + .map(f64::from), + ), + "FLOAT8" => super::pyfmt::num(row.try_get::, _>(i).ok().flatten()), + "BOOL" => json!(row.try_get::, _>(i).ok().flatten()), + "TIMESTAMP" => json!( + row.try_get::, _>(i) + .ok() + .flatten() + .map(isoformat) + ), + "DATE" => json!( + row.try_get::, _>(i) + .ok() + .flatten() + .map(|d| d.format("%Y-%m-%d").to_string()) + ), + _ => json!(row.try_get::, _>(i).ok().flatten()), + }; + m.insert(col.name().to_string(), v); + } + m +} + +pub fn rows_json(rows: &[PgRow]) -> Vec { + rows.iter().map(|r| Value::Object(row_json(r))).collect() +} + +fn query(q: &Option) -> std::collections::HashMap { + params::query_map(q.as_deref()) +} + +/// `datetime.now() - timedelta(hours=h)`; `None` where Python overflows. +pub fn hours_before(t: NaiveDateTime, hours: i64) -> Option { + t.checked_sub_signed(TimeDelta::try_hours(hours)?) +} + +const MEASUREMENT_COLUMNS: &str = "m.timestamp, s.station_code, s.english_name AS station_name_en, + s.thai_name AS station_name_th, m.water_level::float8 AS water_level, + m.discharge::float8 AS discharge, m.discharge_percent::float8 AS discharge_percent, m.status"; + +pub const LATEST: &[Param] = &[Param::int("limit", 100)]; + +/// `SQLAdapter.get_latest_measurements`: the newest reading per station. +pub async fn latest_rows(pool: &sqlx::PgPool, limit: i64) -> Result, sqlx::Error> { + if limit < 0 { + // Postgres refuses a negative LIMIT; the Python adapter answered [] + return Ok(vec![]); + } + sqlx::query(&format!( + "SELECT {MEASUREMENT_COLUMNS} + FROM water_measurements m + JOIN stations s ON m.station_id = s.id + INNER JOIN ( + SELECT station_id, MAX(timestamp) as max_timestamp + FROM water_measurements + GROUP BY station_id + ) latest ON m.station_id = latest.station_id AND m.timestamp = latest.max_timestamp + ORDER BY s.station_code + LIMIT $1" + )) + .bind(limit) + .fetch_all(pool) + .await +} + +/// GET /measurements/latest: the most frequently hit endpoint (every +/// dashboard poll), stale-while-revalidate cached. +pub async fn latest(State(st): State, RawQuery(q): RawQuery, h: HeaderMap) -> ApiResult { + st.metrics.count("measurements_latest"); + let a = params::parse(LATEST, &query(&q), &[], &h)?; + let limit = a.int("limit"); + let pool = st.pool.clone(); + let (body, stale) = st + .caches + .latest + .get( + format!("latest:{limit}"), + secs(st.cfg.web.latest_cache_ttl_s), + None, + move || { + let pool = pool.clone(); + async move { + let rows = rows_json(&latest_rows(&pool, limit).await?); + let keep = !rows.is_empty(); + Ok((to_body(&rows), keep)) + } + }, + ) + .await?; + ok(body, stale) +} + +type HistoryRow = (NaiveDateTime, String, Option, Option, Option); + +pub const HISTORY: &[Param] = &[ + Param::path_str("station_code"), + Param::int("hours", 168).ge(1), + Param::int("limit", 50000).ge(1).le(100000), + Param::opt_date("start", "First day (overrides hours)"), + Param::opt_date("end", "Last day, inclusive"), +]; + +/// GET /measurements/history/{station_code} (`PostgresHistory.station_history`) +pub async fn history( + State(st): State, + Path(code): Path, + RawQuery(q): RawQuery, + h: HeaderMap, +) -> ApiResult { + let a = params::parse(HISTORY, &query(&q), &[("station_code", &code)], &h)?; + let key = format!( + "{code}:{}:{}:{}:{}", + a.py_str("hours"), + a.py_str("limit"), + a.py_str("start"), + a.py_str("end") + ); + let (hours, limit, start, end) = ( + a.int("hours"), + a.int("limit"), + a.date("start"), + a.date("end"), + ); + let pool = st.pool.clone(); + let unavailable = || ApiError::new(502, "Measurement history unavailable"); + let (body, stale) = st + .caches + .history + .get( + key, + Duration::from_secs(300), + Some(Duration::ZERO), + move || { + let (pool, code) = (pool.clone(), code.clone()); + async move { + let end_time = end.map_or_else(bangkok_now, end_of_day); + let start_time = match start { + Some(d) => start_of_day(d), + None => hours_before(end_time, hours).ok_or_else(unavailable)?, + }; + if start_time >= end_time { + return Err(unavailable()); + } + let rows: Vec = sqlx::query_as( + "SELECT m.timestamp, s.station_code, m.water_level::float8, + m.discharge::float8, m.discharge_percent::float8 + FROM water_measurements m + JOIN stations s ON m.station_id = s.id + WHERE s.station_code = $1 + AND m.timestamp >= $2 + AND m.timestamp <= $3 + ORDER BY m.timestamp ASC + LIMIT $4", + ) + .bind(&code) + .bind(start_time) + .bind(end_time) + .bind(limit) + .fetch_all(&pool) + .await + .map_err(|e| { + tracing::error!("Error fetching measurement history: {e}"); + unavailable() + })?; + let out: Vec = rows + .into_iter() + .map(|(t, code, level, discharge, pct)| { + // no rating curves are defined: Q = 20 * H, as postgres_history.py + let discharge = discharge.or(level.map(|h| py_round(20.0 * h, 2))); + json!({ + "timestamp": isoformat(t), + "station_code": code, + "water_level": level, + "discharge": discharge, + "discharge_percent": pct, + }) + }) + .collect(); + Ok((to_body(&out), true)) + } + }, + ) + .await?; + ok(body, stale) +} + +pub const STATION: &[Param] = &[ + Param::path_str("station_code"), + Param::int("hours", 24), + Param::int("limit", 1000), +]; + +/// GET /measurements/station/{station_code}: newest first. +pub async fn station( + State(st): State, + Path(code): Path, + RawQuery(q): RawQuery, + h: HeaderMap, +) -> ApiResult { + st.metrics.count("measurements_station"); + let a = params::parse(STATION, &query(&q), &[("station_code", &code)], &h)?; + let end = bangkok_now(); + let start = hours_before(end, a.int("hours")) + .ok_or_else(|| ApiError::new(500, "date value out of range"))?; + let rows = sqlx::query(&format!( + "SELECT {MEASUREMENT_COLUMNS} + FROM water_measurements m + JOIN stations s ON m.station_id = s.id + WHERE m.timestamp BETWEEN $1 AND $2 AND s.station_code IN ($3) + ORDER BY m.timestamp DESC, s.station_code" + )) + .bind(start) + .bind(end) + .bind(&code) + .fetch_all(&st.pool) + .await?; + let mut out = rows_json(&rows); + // measurements[:limit], Python slice semantics + let limit = a.int("limit"); + let keep = if limit >= 0 { + (limit as usize).min(out.len()) + } else { + out.len().saturating_sub(limit.unsigned_abs() as usize) + }; + out.truncate(keep); + ok(to_body(&out), false) +} + +async fn count(pool: &sqlx::PgPool, table: &str) -> i64 { + sqlx::query_scalar::<_, i64>(&format!("SELECT COUNT(*) FROM {table}")) + .fetch_one(pool) + .await + .unwrap_or(0) +} + +type HiiTotals = ( + i64, + i64, + i64, + i64, + Option, + Option, + Option, + Option, +); + +async fn compute_stats(pool: sqlx::PgPool) -> super::cache::Computed { + // Heavy: full-table counts and coverage; runs at most once per TTL + type Stats = (i64, i64, Option, Option, i64); + let (total, stations, first, last, recorded): Stats = sqlx::query_as( + "SELECT COUNT(*), COUNT(DISTINCT station_id), MIN(timestamp), MAX(timestamp), + COUNT(DISTINCT TO_CHAR(timestamp, 'YYYY-MM-DD HH24')) + FROM water_measurements", + ) + .fetch_one(&pool) + .await?; + let (Some(mut first_ts), Some(mut last_ts)) = (first, last) else { + return Err(ApiError::new(503, "Database statistics unavailable")); + }; + if total == 0 { + return Err(ApiError::new(503, "Database statistics unavailable")); + } + let slot = crate::util::floor_hour; + let expected = (slot(last_ts) - slot(first_ts)).num_hours() + 1; + let coverage = py_round(100.0 * recorded as f64 / expected as f64, 1); + + let hii: Option = sqlx::query_as( + "SELECT (SELECT COUNT(*) FROM hii_rainfall), + (SELECT COUNT(*) FROM hii_waterlevel), + (SELECT COUNT(*) FROM hii_rain_stations), + (SELECT COUNT(*) FROM hii_wl_stations), + (SELECT MIN(timestamp) FROM hii_rainfall), + (SELECT MAX(timestamp) FROM hii_rainfall), + (SELECT MIN(timestamp) FROM hii_waterlevel), + (SELECT MAX(timestamp) FROM hii_waterlevel)", + ) + .fetch_one(&pool) + .await + .map_err(|e| tracing::warn!("HII stats unavailable: {e}")) + .ok(); + let openmeteo_n = count(&pool, "openmeteo_rain").await; + let reservoir_n = count(&pool, "rid_reservoir_daily").await; + let (mut rain_n, mut wl_n, mut hii_stations) = (0, 0, 0); + if let Some((rn, wn, rs, ws, rain_lo, rain_hi, wl_lo, wl_hi)) = hii { + (rain_n, wl_n, hii_stations) = (rn, wn, rs + ws); + for lo in [rain_lo, wl_lo].into_iter().flatten() { + if lo < first_ts { + first_ts = lo; + } + } + for hi in [rain_hi, wl_hi].into_iter().flatten() { + if hi > last_ts { + last_ts = hi; + } + } + } + let body = json!({ + // whole-DB totals (RID + HII + Open-Meteo + dams), breakdown alongside + "total_measurements": total + rain_n + wl_n + openmeteo_n + reservoir_n, + "rid_measurements": total, + "hii_rainfall_measurements": rain_n, + "hii_waterlevel_measurements": wl_n, + "openmeteo_rain_measurements": openmeteo_n, + "reservoir_measurements": reservoir_n, + "station_count": stations + hii_stations, + "rid_station_count": stations, + "hii_station_count": hii_stations, + "first_timestamp": isoformat(first_ts), + "last_timestamp": isoformat(last_ts), + "days_spanned": (last_ts.date() - first_ts.date()).num_days() + 1, + "coverage_percent": coverage, + }); + Ok((to_body(&body), true)) +} + +/// GET /api/stats: database coverage (totals, date range, hourly coverage). +pub async fn stats(State(st): State) -> ApiResult { + st.metrics.count("api_stats"); + let pool = st.pool.clone(); + let (body, _) = st + .caches + .stats + .get("all".into(), Duration::from_secs(300), None, move || { + compute_stats(pool.clone()) + }) + .await?; + ok(body, false) +} diff --git a/rust/collector/src/web/error.rs b/rust/collector/src/web/error.rs new file mode 100644 index 0000000..ddcb410 --- /dev/null +++ b/rust/collector/src/web/error.rs @@ -0,0 +1,88 @@ +//! FastAPI-shaped errors (`{"detail": ...}`) and JSON responses. + +use axum::body::Bytes; +use axum::http::{HeaderValue, StatusCode, header}; +use axum::response::{IntoResponse, Response}; +use serde::Serialize; +use serde_json::{Value, json}; + +#[derive(Debug, Clone)] +pub struct ApiError { + pub status: StatusCode, + pub detail: Value, +} + +impl ApiError { + pub fn new(status: u16, detail: impl Into) -> Self { + Self { + status: StatusCode::from_u16(status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR), + detail: Value::String(detail.into()), + } + } + + /// 422 with pydantic's error list. + pub fn validation(errors: Vec) -> Self { + Self { + status: StatusCode::UNPROCESSABLE_ENTITY, + detail: Value::Array(errors), + } + } + + /// An unexpected failure (FastAPI handlers answer `500 str(e)`). + pub fn internal(e: impl std::fmt::Display) -> Self { + Self::new(500, e.to_string()) + } +} + +impl From for ApiError { + fn from(e: sqlx::Error) -> Self { + tracing::error!("database: {e}"); + Self::internal(e) + } +} + +impl From for ApiError { + fn from(e: anyhow::Error) -> Self { + tracing::error!("{e:#}"); + Self::new(500, format!("{e:#}")) + } +} + +impl IntoResponse for ApiError { + fn into_response(self) -> Response { + json_response( + self.status, + to_body(&json!({ "detail": self.detail })), + false, + ) + } +} + +pub type ApiResult = Result; + +pub fn to_body(v: &impl Serialize) -> Bytes { + Bytes::from(serde_json::to_vec(v).expect("JSON-serializable value")) +} + +/// A JSON body; `stale` adds `X-Data-Stale: true` (served from the cache +/// after a failed refresh, or an old stored forecast). +pub fn json_response(status: StatusCode, body: Bytes, stale: bool) -> Response { + let mut r = (status, body).into_response(); + let h = r.headers_mut(); + h.insert( + header::CONTENT_TYPE, + HeaderValue::from_static("application/json"), + ); + if stale { + h.insert("x-data-stale", HeaderValue::from_static("true")); + } + r +} + +pub fn ok(body: Bytes, stale: bool) -> ApiResult { + Ok(json_response(StatusCode::OK, body, stale)) +} + +pub fn ok_json(v: &impl Serialize) -> ApiResult { + ok(to_body(v), false) +} diff --git a/rust/collector/src/web/forecasts.rs b/rust/collector/src/web/forecasts.rs new file mode 100644 index 0000000..714f275 --- /dev/null +++ b/rust/collector/src/web/forecasts.rs @@ -0,0 +1,333 @@ +//! Flood forecasts: /forecast, /api/forecast/*, /api/notifications. + +use super::data::{hours_before, rows_json}; +use super::error::{ApiError, ApiResult, ok, ok_json, to_body}; +use super::params::{self, Param}; +use super::pyfmt::{end_of_day, isoformat, py_round, start_of_day}; +use super::{Shared, secs}; +use crate::util::bangkok_now; +use axum::extract::{Path, RawQuery, State}; +use axum::http::HeaderMap; +use chrono::NaiveDateTime; +use serde_json::{Map, Value, json}; +use std::time::Duration; + +/// GET /forecast: flood-risk forecasts per station for the 6/12/24 h +/// horizons, as the forecaster last stored them (forecast_current). +/// +/// The Python app recomputed in-process when the stored forecast was older +/// than FORECAST_MAX_AGE_SECONDS; here the forecaster runs in the same +/// process, so an old forecast is served marked `X-Data-Stale: true`. +pub async fn forecast(State(st): State) -> ApiResult { + st.metrics.count("forecast"); + let pool = st.pool.clone(); + let max_age = st.cfg.web.forecast_max_age_s as i64; + let ((body, old), _) = st + .caches + .forecast + .get( + "stored".into(), + Duration::from_secs(60), + Some(Duration::ZERO), + move || { + let pool = pool.clone(); + async move { + let row: Option<(NaiveDateTime, String)> = sqlx::query_as( + "SELECT computed_at, payload FROM forecast_current WHERE id = 'all'", + ) + .fetch_optional(&pool) + .await?; + let Some((computed_at, payload)) = row else { + return Err(ApiError::new( + 503, + "No forecast stored yet (the forecaster has not run)", + )); + }; + let age = (bangkok_now() - computed_at).num_seconds(); + if age > max_age { + tracing::warn!("stored forecast is {:.1} h old", age as f64 / 3600.0); + } + Ok(((payload.into(), age > max_age), true)) + } + }, + ) + .await?; + ok(body, old) +} + +pub const HISTORY: &[Param] = &[ + Param::path_str("station_code"), + Param::int("hours", 168).ge(1), + Param::opt_date("start", "First day (overrides hours)"), + Param::opt_date("end", "Last day, inclusive"), + Param::opt_int("horizon").ge(1).le(48), +]; + +/// GET /api/forecast/history/{station_code}: issued model forecasts, what +/// the model predicted when (`ForecastHistoryStore.fetch`). +pub async fn history( + State(st): State, + Path(code): Path, + RawQuery(q): RawQuery, + h: HeaderMap, +) -> ApiResult { + st.metrics.count("forecast_history"); + let a = params::parse( + HISTORY, + ¶ms::query_map(q.as_deref()), + &[("station_code", &code)], + &h, + )?; + let end = a.date("end").map_or_else(bangkok_now, end_of_day); + let start = match a.date("start") { + Some(d) => start_of_day(d), + None => hours_before(end, a.int("hours")) + .ok_or_else(|| ApiError::new(500, "date value out of range"))?, + }; + let horizon = a.opt_int("horizon"); + let rows = sqlx::query( + "SELECT as_of, station_code, horizon_hours, + predicted_max_level::float8 AS predicted_max_level, + p_warning::float8 AS p_warning, p_danger::float8 AS p_danger, + current_level::float8 AS current_level, model_version, source + FROM forecast_history + WHERE station_code = $1 AND as_of >= $2 AND as_of <= $3 + AND ($4::int IS NULL OR horizon_hours = $4) + ORDER BY as_of ASC, horizon_hours ASC LIMIT 5000", + ) + .bind(&code) + .bind(start) + .bind(end) + .bind(horizon.map(|v| v as i32)) + .fetch_all(&st.pool) + .await; + match rows { + Ok(rows) => ok_json(&rows_json(&rows)), + Err(e) => { + // as the Python store: an empty history rather than an error + tracing::error!("forecast history fetch failed: {e}"); + ok_json(&Vec::::new()) + } + } +} + +/// GET /api/notifications: public ntfy settings for the dashboard's +/// subscribe links. +pub async fn notifications(State(st): State) -> ApiResult { + let Some(server) = &st.cfg.ntfy_server else { + return ok_json(&json!({"enabled": false})); + }; + let p = &st.cfg.ntfy_topic_prefix; + ok_json(&json!({ + "enabled": true, + "server": server, + "prefix": p, + "topics": { + "warning": format!("{p}-warning"), + "danger": format!("{p}-danger"), + "p1_outlook": format!("{p}-p1-outlook"), + "status": format!("{p}-status"), + "station_pattern": format!("{p}--warning | {p}--danger (station code lowercase, no dot: p1, p103)"), + }, + "semantics": "transitions only: one message on crossing up, one all-clear on the way down (0.10 m hysteresis)", + })) +} + +pub const SKILL: &[Param] = &[ + Param::str("station_code", "P.1"), + Param::int("horizon", 24).ge(1).le(48), +]; + +const MIN_VERIFIED: usize = 24; // fewer than a day of verified hours is not a number + +struct Verified { + as_of: NaiveDateTime, + predicted: f64, + current: Option, + observed: f64, +} + +/// Python's `sum()` (3.11: left to right) / len +fn mean(v: impl Iterator) -> f64 { + let (mut s, mut n) = (0.0, 0usize); + for x in v { + s += x; + n += 1; + } + s / n as f64 +} + +/// `src/ml/skill.compute_skill`: issued forecasts verified against what the +/// river did, per model version, with a persistence baseline. +pub async fn compute_skill( + pool: &sqlx::PgPool, + code: &str, + horizon: i64, + now: NaiveDateTime, +) -> Result { + let verifiable_before = now - chrono::TimeDelta::hours(horizon); + type Row = ( + NaiveDateTime, + Option, + f64, + Option, + Option, + Option, + ); + let rows: Vec = sqlx::query_as( + "SELECT f.as_of, f.model_version, f.predicted_max_level::float8, f.current_level::float8, + (SELECT MAX(m.water_level) FROM water_measurements m + JOIN stations s ON s.id = m.station_id + WHERE s.station_code = f.station_code + AND m.timestamp > f.as_of + AND m.timestamp <= f.as_of + (f.horizon_hours || ' hours')::interval)::float8, + (SELECT COUNT(m.water_level) FROM water_measurements m + JOIN stations s ON s.id = m.station_id + WHERE s.station_code = f.station_code + AND m.timestamp > f.as_of + AND m.timestamp <= f.as_of + (f.horizon_hours || ' hours')::interval) + FROM forecast_history f + WHERE f.station_code = $1 AND f.horizon_hours = $2 + AND f.source = 'model' AND f.predicted_max_level IS NOT NULL + AND f.as_of <= $3 + ORDER BY f.as_of", + ) + .bind(code) + .bind(horizon as i32) + .bind(verifiable_before) + .fetch_all(pool) + .await?; + + // versions in order of first appearance, as Python's dict + let mut by_version: Vec<(String, Vec)> = vec![]; + for (as_of, version, predicted, current, observed, n) in rows { + // most of the window must be observed, or the "max" is not the peak + let Some(observed) = observed else { continue }; + if (n.unwrap_or(0) as f64) < horizon as f64 * 0.75 { + continue; + } + let version = version.unwrap_or_else(|| "unknown".into()); + let row = Verified { + as_of, + predicted, + current, + observed, + }; + match by_version.iter_mut().find(|(v, _)| *v == version) { + Some((_, list)) => list.push(row), + None => by_version.push((version, vec![row])), + } + } + + let mut versions: Vec> = vec![]; + for (version, rows) in &by_version { + let err: Vec = rows.iter().map(|r| r.predicted - r.observed).collect(); + let mae = mean(err.iter().map(|e| e.abs())); + let bias = mean(err.iter().copied()); + let pers: Vec<(f64, f64)> = rows + .iter() + .filter_map(|r| r.current.map(|c| (c, r.observed))) + .collect(); + let persistence = (!pers.is_empty()).then(|| mean(pers.iter().map(|(c, o)| (c - o).abs()))); + let high: Vec<(f64, f64)> = rows + .iter() + .filter(|r| r.observed >= 2.0) + .map(|r| (r.predicted, r.observed)) + .collect(); + let first = rows.iter().map(|r| r.as_of).min().expect("non-empty"); + let last = rows.iter().map(|r| r.as_of).max().expect("non-empty"); + let mut v = Map::new(); + v.insert("model_version".into(), json!(version)); + v.insert("first_issued".into(), json!(isoformat(first))); + v.insert("last_issued".into(), json!(isoformat(last))); + v.insert("n".into(), json!(rows.len())); + v.insert("mae_m".into(), json!(py_round(mae, 3))); + v.insert("bias_m".into(), json!(py_round(bias, 3))); + v.insert( + "persistence_mae_m".into(), + json!(persistence.map(|p| py_round(p, 3))), + ); + v.insert( + "skill".into(), + json!( + persistence + .filter(|p| *p != 0.0) + .map(|p| py_round(1.0 - mae / p, 3)) + ), + ); + v.insert("above_2m_n".into(), json!(high.len())); + v.insert( + "above_2m_mae_m".into(), + json!( + (!high.is_empty()) + .then(|| py_round(mean(high.iter().map(|(p, o)| (p - o).abs())), 3)) + ), + ); + v.insert("enough_data".into(), json!(rows.len() >= MIN_VERIFIED)); + versions.push(v); + } + // stable sort on the ISO string, as Python + versions.sort_by(|a, b| a["first_issued"].as_str().cmp(&b["first_issued"].as_str())); + + let current = versions.last().cloned(); + let previous = versions + .len() + .checked_sub(1) + .and_then(|n| { + versions[..n] + .iter() + .rev() + .find(|v| v["enough_data"] == true) + }) + .cloned(); + let trend = match (¤t, &previous) { + (Some(c), Some(p)) if c["enough_data"] == true => { + let (cm, pm) = (c["mae_m"].as_f64().unwrap(), p["mae_m"].as_f64().unwrap()); + let skill_delta = match (c["skill"].as_f64(), p["skill"].as_f64()) { + (Some(cs), Some(ps)) => json!(py_round(cs - ps, 3)), + _ => Value::Null, + }; + json!({ + "previous_version": p["model_version"], + "mae_delta_m": py_round(cm - pm, 3), + "skill_delta": skill_delta, + "better": cm < pm, + }) + } + _ => Value::Null, + }; + Ok(json!({ + "station_code": code, + "horizon_hours": horizon, + "verified_until": isoformat(verifiable_before), + "min_verified": MIN_VERIFIED, + "versions": versions, + "current": current, + "trend": trend, + })) +} + +/// GET /api/forecast/skill: is the model getting better? +pub async fn skill(State(st): State, RawQuery(q): RawQuery, h: HeaderMap) -> ApiResult { + st.metrics.count("forecast_skill"); + let a = params::parse(SKILL, ¶ms::query_map(q.as_deref()), &[], &h)?; + let (code, horizon) = (a.str("station_code").to_string(), a.int("horizon")); + let pool = st.pool.clone(); + let (body, stale) = st + .caches + .hii + .get( + format!("skill:{code}:{horizon}"), + secs(st.cfg.web.hii_cache_ttl_s.max(900)), + None, + move || { + let (pool, code) = (pool.clone(), code.clone()); + async move { + let v = compute_skill(&pool, &code, horizon, bangkok_now()).await?; + Ok((to_body(&v), true)) + } + }, + ) + .await?; + ok(body, stale) +} diff --git a/rust/collector/src/web/hii.rs b/rust/collector/src/web/hii.rs new file mode 100644 index 0000000..7a1ff9a --- /dev/null +++ b/rust/collector/src/web/hii.rs @@ -0,0 +1,427 @@ +//! HII/ThaiWater feeds: /api/hii/*, /sensors/thaiwater. + +use super::data::{hours_before, rows_json}; +use super::error::{ApiError, ApiResult, ok, to_body}; +use super::params::{self, Param}; +use super::pyfmt::{isoformat, num, numpy_mean, py_round, py_str}; +use super::{Shared, secs}; +use crate::util::{bangkok_now, floor_hour}; +use axum::extract::{RawQuery, State}; +use axum::http::HeaderMap; +use chrono::{NaiveDateTime, TimeDelta}; +use serde_json::{Map, Value, json}; +use std::collections::BTreeMap; +use std::time::Duration; + +pub const LATEST: &[Param] = &[Param::int("hours", 26).ge(1).le(168)]; + +const RAIN_SQL: &str = " + SELECT s.id AS station_id, s.oldcode, s.name_en, s.name_th, + s.latitude::float8 AS latitude, s.longitude::float8 AS longitude, s.agency, + m.timestamp, m.rain_1h::float8 AS rain_1h, m.rain_24h::float8 AS rain_24h + FROM hii_rainfall m + JOIN hii_rain_stations s ON s.id = m.station_id + JOIN (SELECT station_id, MAX(timestamp) AS latest_ts + FROM hii_rainfall WHERE timestamp >= $1 + GROUP BY station_id) latest + ON latest.station_id = m.station_id AND latest.latest_ts = m.timestamp + ORDER BY s.id"; + +const WATERLEVEL_SQL: &str = " + SELECT s.id AS station_id, s.oldcode, s.rid_code, s.name_en, s.name_th, + s.latitude::float8 AS latitude, s.longitude::float8 AS longitude, s.agency, + s.river_name, s.offset_msl::float8 AS offset_msl, + s.min_bank_msl::float8 AS min_bank_msl, s.is_key_station, + m.timestamp, m.wl_msl::float8 AS wl_msl, m.discharge::float8 AS discharge, + m.storage_percent::float8 AS storage_percent, m.situation_level, + m.diff_wl_bank::float8 AS diff_wl_bank + FROM hii_waterlevel m + JOIN hii_wl_stations s ON s.id = m.station_id + JOIN (SELECT station_id, MAX(timestamp) AS latest_ts + FROM hii_waterlevel WHERE timestamp >= $1 + GROUP BY station_id) latest + ON latest.station_id = m.station_id AND latest.latest_ts = m.timestamp + ORDER BY s.id"; + +async fn latest_feed( + st: Shared, + feed: &'static str, + sql: &'static str, + q: Option, + h: HeaderMap, +) -> ApiResult { + let a = params::parse(LATEST, ¶ms::query_map(q.as_deref()), &[], &h)?; + let hours = a.int("hours"); + let pool = st.pool.clone(); + let (body, stale) = st + .caches + .hii + .get( + format!("{feed}:{hours}"), + secs(st.cfg.web.hii_cache_ttl_s), + None, + move || { + let pool = pool.clone(); + async move { + let cutoff = hours_before(bangkok_now(), hours) + .ok_or_else(|| ApiError::new(500, "date value out of range"))?; + let rows = rows_json(&sqlx::query(sql).bind(cutoff).fetch_all(&pool).await?); + let keep = !rows.is_empty(); + Ok((to_body(&rows), keep)) + } + }, + ) + .await?; + ok(body, stale) +} + +/// GET /api/hii/rainfall/latest: latest rainfall per HII station (Ping basin). +pub async fn rainfall_latest( + State(st): State, + RawQuery(q): RawQuery, + h: HeaderMap, +) -> ApiResult { + st.metrics.count("hii_rainfall"); + latest_feed(st, "rain", RAIN_SQL, q, h).await +} + +/// GET /api/hii/waterlevel/latest: latest water level per HII station (m MSL). +pub async fn waterlevel_latest( + State(st): State, + RawQuery(q): RawQuery, + h: HeaderMap, +) -> ApiResult { + st.metrics.count("hii_waterlevel"); + latest_feed(st, "waterlevel", WATERLEVEL_SQL, q, h).await +} + +/// The upper Ping above P.1 (src/ml/hii_rain.py CATCHMENT_BOX): +/// lat_lo, lat_hi, lon_lo, lon_hi. +const CATCHMENT_BOX: [f64; 4] = [18.75, 19.60, 98.60, 99.30]; +const MIN_GAUGES_PER_HOUR: i64 = 5; +const WINDOW_H: usize = 24; + +pub const CATCHMENT: &[Param] = &[Param::int("days", 14).ge(1).le(60)]; + +/// `hii_rain.load_gauge_mean`: hourly catchment-mean rain_1h; hours with +/// fewer than MIN_GAUGES_PER_HOUR gauges are NaN. +async fn gauge_mean( + pool: &sqlx::PgPool, + start: NaiveDateTime, +) -> Option> { + let [lat_lo, lat_hi, lon_lo, lon_hi] = CATCHMENT_BOX.map(|x| x.to_string()); + let rows: Vec<(NaiveDateTime, i64, Option)> = sqlx::query_as( + "SELECT m.timestamp, COUNT(m.rain_1h) AS n, AVG(m.rain_1h)::float8 AS rain_1h + FROM hii_rainfall m JOIN hii_rain_stations s ON s.id = m.station_id + WHERE s.latitude BETWEEN $1::numeric AND $2::numeric + AND s.longitude BETWEEN $3::numeric AND $4::numeric + AND m.rain_1h IS NOT NULL AND m.timestamp >= $5 + GROUP BY m.timestamp ORDER BY m.timestamp", + ) + .bind(lat_lo) + .bind(lat_hi) + .bind(lon_lo) + .bind(lon_hi) + .bind(start) + .fetch_all(pool) + .await + .map_err(|e| tracing::warn!("HII gauge rain load failed: {e}")) + .ok()?; + if rows.is_empty() { + return None; + } + // groupby(floor('h')).agg(n=sum, rain_1h=mean): pandas' group_mean + // (Kahan-compensated sum / count) + struct Acc { + n: i64, + sum: f64, + comp: f64, + nobs: u32, + } + let mut groups: BTreeMap = BTreeMap::new(); + for (t, n, rain) in rows { + let g = groups.entry(floor_hour(t)).or_insert(Acc { + n: 0, + sum: 0.0, + comp: 0.0, + nobs: 0, + }); + g.n += n; + if let Some(val) = rain.filter(|v| !v.is_nan()) { + g.nobs += 1; + let y = val - g.comp; + let t = g.sum + y; + g.comp = t - g.sum - y; + if g.comp.is_nan() { + g.comp = 0.0; + } + g.sum = t; + } + } + Some( + groups + .into_iter() + .map(|(t, g)| { + let mean = if g.nobs == 0 { + f64::NAN + } else { + g.sum / g.nobs as f64 + }; + ( + t, + if g.n < MIN_GAUGES_PER_HOUR { + f64::NAN + } else { + mean + }, + ) + }) + .collect(), + ) +} + +/// `hii_rain.compare_with_openmeteo`: agreement of 24 h sums over the hours +/// both series have. +fn compare(gauge: &[(NaiveDateTime, f64)], openmeteo: &[(NaiveDateTime, f64)]) -> Value { + let om: std::collections::HashMap = openmeteo.iter().copied().collect(); + let joined: Vec<(f64, f64)> = gauge + .iter() + .filter(|(_, g)| !g.is_nan()) + .filter_map(|(t, g)| om.get(t).filter(|o| !o.is_nan()).map(|o| (*g, *o))) + .collect(); + if joined.is_empty() { + return json!({"overlap_hours": 0}); + } + if joined.len() < WINDOW_H { + return json!({"overlap_hours": joined.len()}); + } + let gs: Vec = joined.iter().map(|p| p.0).collect(); + let os: Vec = joined.iter().map(|p| p.1).collect(); + let g = ping_forecast::rolling::rolling_sum(&gs, WINDOW_H)[WINDOW_H - 1..].to_vec(); + let o = ping_forecast::rolling::rolling_sum(&os, WINDOW_H)[WINDOW_H - 1..].to_vec(); + let diff: Vec = o.iter().zip(&g).map(|(o, g)| o - g).collect(); + let abs: Vec = diff.iter().map(|d| d.abs()).collect(); + json!({ + "overlap_hours": joined.len(), + "window_h": WINDOW_H, + "gauge_mean_mm": num(Some(numpy_mean(&g))), + "openmeteo_mean_mm": num(Some(numpy_mean(&o))), + "bias_mm": num(Some(numpy_mean(&diff))), + "mae_mm": num(Some(numpy_mean(&abs))), + "corr": num(Some(pearson(&g, &o))), + }) +} + +/// `np.corrcoef(a, b)[0, 1]` (same formula; numpy's BLAS dot may differ in +/// the last bits). NaN for a constant series, which JSON shows as null. +fn pearson(a: &[f64], b: &[f64]) -> f64 { + let (ma, mb) = (numpy_mean(a), numpy_mean(b)); + let da: Vec = a.iter().map(|x| x - ma).collect(); + let db: Vec = b.iter().map(|x| x - mb).collect(); + let dot = |x: &[f64], y: &[f64]| x.iter().zip(y).map(|(p, q)| p * q).sum::(); + let fact = (a.len() - 1) as f64; + let (cab, caa, cbb) = ( + dot(&da, &db) / fact, + dot(&da, &da) / fact, + dot(&db, &db) / fact, + ); + (cab / caa.sqrt() / cbb.sqrt()).clamp(-1.0, 1.0) +} + +fn series_rows(s: &[(NaiveDateTime, f64)]) -> Vec { + s.iter() + .map(|(t, v)| { + json!({ + "timestamp": isoformat(*t), + "rain_mm": if v.is_nan() { Value::Null } else { json!(py_round(*v, 2)) }, + }) + }) + .collect() +} + +/// GET /api/hii/rainfall/catchment: HII gauges vs the Open-Meteo series the +/// flood model uses, plus their agreement over the window. +pub async fn rainfall_catchment( + State(st): State, + RawQuery(q): RawQuery, + h: HeaderMap, +) -> ApiResult { + st.metrics.count("hii_rain_catchment"); + let a = params::parse(CATCHMENT, ¶ms::query_map(q.as_deref()), &[], &h)?; + let days = a.int("days"); + let pool = st.pool.clone(); + let (body, stale) = st + .caches + .hii + .get( + format!("rain_catchment:{days}"), + secs(st.cfg.web.hii_cache_ttl_s), + None, + move || { + let pool = pool.clone(); + async move { + let start = bangkok_now() - TimeDelta::days(days); + let gauge = gauge_mean(&pool, start).await; + let openmeteo: Option> = + match sqlx::query_as::<_, (NaiveDateTime, Option)>( + "SELECT timestamp, catchment_mean::float8 FROM openmeteo_rain + WHERE timestamp >= $1 ORDER BY timestamp", + ) + .bind(start) + .fetch_all(&pool) + .await + { + Ok(rows) if !rows.is_empty() => Some( + rows.into_iter() + .map(|(t, v)| (t, v.unwrap_or(f64::NAN))) + .collect(), + ), + Ok(_) => None, + Err(e) => { + tracing::warn!("openmeteo_rain read failed: {e}"); + None + } + }; + let comparison = match (&gauge, &openmeteo) { + (Some(g), Some(o)) => compare(g, o), + _ => json!({"overlap_hours": 0}), + }; + let body = json!({ + "box": CATCHMENT_BOX, + "gauge": gauge.as_deref().map(series_rows).unwrap_or_default(), + "openmeteo": openmeteo.as_deref().map(series_rows).unwrap_or_default(), + "comparison_24h_sums": comparison, + }); + Ok((to_body(&body), true)) + } + }, + ) + .await?; + ok(body, stale) +} + +pub const THAIWATER_URL: &str = "https://twa-api-public.thaiwater.net/v2/waterlevel"; + +/// `ThaiWaterClient._parse_ping_features`: the Ping-basin sensors. +pub fn parse_ping_features(payload: &Value) -> Vec { + let empty = Map::new(); + let obj = |v: Option<&Value>| -> Map { + v.and_then(Value::as_object).cloned().unwrap_or_default() + }; + let mut out = vec![]; + let collections = payload + .get("data") + .and_then(Value::as_object) + .unwrap_or(&empty); + for collection in collections.values() { + let features = collection.get("features").and_then(Value::as_array); + for feature in features.into_iter().flatten() { + let props = obj(feature.get("properties")); + let basin = obj(props.get("basin")); + if basin.get("basin").and_then(Value::as_str) != Some("Ping") { + continue; + } + let geometry = obj(feature.get("geometry")); + let coords = geometry + .get("coordinates") + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + if coords.len() < 2 { + continue; + } + let station = obj(props.get("station")); + let code = station + .get("stationCode") + .and_then(Value::as_str) + .unwrap_or(""); + let get = |m: &Map, k: &str| m.get(k).cloned().unwrap_or(Value::Null); + out.push(json!({ + "id": format!("thaiwater:{}", py_str(props.get("id").unwrap_or(&Value::Null))), + "station_code": code.split_once('-').map_or(code, |(_, rest)| rest), + "station_name": get(&station, "station"), + "latitude": coords[1], + "longitude": coords[0], + "timestamp": get(&props, "waterlevelDatetime"), + "water_level_msl": get(&props, "waterlevelMsl"), + "bank_percent": get(&props, "storagePercent"), + "distance_to_bank": get(&props, "diffWlBank"), + "river_name": get(&props, "riverName"), + "agency": get(&obj(props.get("agency")), "agencyShort"), + "source": "ThaiWater", + })); + } + } + out +} + +/// GET /sensors/thaiwater: current ThaiWater water-level sensors in the Ping +/// basin, fetched live (cached for a minute; the feed updates every ~10 min). +pub async fn thaiwater_sensors(State(st): State) -> ApiResult { + st.metrics.count("thaiwater_sensors"); + let Some(key) = st.cfg.web.thaiwater_api_key.clone() else { + return Err(ApiError::new(503, "THAIWATER_API_KEY is not configured")); + }; + let (http, timeout) = (st.http.clone(), st.cfg.web.request_timeout_s); + let (body, stale) = st + .caches + .sensors + .get("ping".into(), Duration::from_secs(60), None, move || { + let (http, key) = (http.clone(), key.clone()); + async move { + let fetch = async { + http.get(THAIWATER_URL) + .header("Accept-Language", "en") + .header("x-api-key", key) + .timeout(Duration::from_secs(timeout)) + .send() + .await? + .error_for_status()? + .json::() + .await + }; + match fetch.await { + Ok(payload) => { + let sensors = parse_ping_features(&payload); + let keep = !sensors.is_empty(); + Ok((to_body(&sensors), keep)) + } + Err(e) => { + tracing::error!("Error fetching ThaiWater sensors: {e}"); + Err(ApiError::new(502, "ThaiWater API unavailable")) + } + } + } + }) + .await?; + ok(body, stale) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn thaiwater_features() { + let payload = json!({"data": {"a": {"features": [ + {"properties": {"id": 2186, "basin": {"basin": "Ping"}, + "station": {"stationCode": "FOP-MOU298", "station": "Mae Tom Bridge"}, + "waterlevelDatetime": "2026-10-18T15:10:00+07:00", "waterlevelMsl": 794.41, + "storagePercent": -10.76, "diffWlBank": 1.9, "riverName": "Nam Mae Tuen", + "agency": {"agencyShort": "FOP"}}, + "geometry": {"coordinates": [98.36419, 17.799877]}}, + {"properties": {"id": 1, "basin": {"basin": "Wang"}}, + "geometry": {"coordinates": [99.0, 18.0]}}, + {"properties": {"basin": {"basin": "Ping"}}, "geometry": {"coordinates": [99.0]}}, + {"properties": {"basin": {"basin": "Ping"}}, "geometry": {"coordinates": [99.0, 18.0]}} + ]}}}); + let s = parse_ping_features(&payload); + assert_eq!(s.len(), 2); + assert_eq!( + s[0].to_string(), + r#"{"id":"thaiwater:2186","station_code":"MOU298","station_name":"Mae Tom Bridge","latitude":17.799877,"longitude":98.36419,"timestamp":"2026-10-18T15:10:00+07:00","water_level_msl":794.41,"bank_percent":-10.76,"distance_to_bank":1.9,"river_name":"Nam Mae Tuen","agency":"FOP","source":"ThaiWater"}"# + ); + assert_eq!(s[1]["id"], "thaiwater:None"); + assert_eq!(s[1]["station_code"], ""); + } +} diff --git a/rust/collector/src/web/mod.rs b/rust/collector/src/web/mod.rs new file mode 100644 index 0000000..bbcd7d4 --- /dev/null +++ b/rust/collector/src/web/mod.rs @@ -0,0 +1,335 @@ +//! The website and read API: the port of the FastAPI app (src/web_api.py). +//! +//! Same routes, parameters, JSON fields, status codes, cache headers and +//! response caches as the Python app, checked route by route against it by +//! scripts/api_parity.py. It runs inside `ping-collector run` (WEB_LISTEN) +//! or alone (`ping-collector serve`). /docs is the same Swagger UI, fed by an +//! OpenAPI document built from the route table that also validates the +//! parameters (openapi.rs, params.rs). + +mod cache; +mod data; +mod error; +mod forecasts; +mod hii; +mod openapi; +mod params; +mod pyfmt; +mod system; +#[cfg(test)] +mod tests; + +use crate::config::Config; +use crate::stations::Stations; +use anyhow::{Context, Result}; +use axum::Router; +use axum::body::Bytes; +use axum::extract::{Request, State}; +use axum::handler::HandlerWithoutStateExt; +use axum::http::{HeaderValue, Method, StatusCode, header}; +use axum::middleware::{self, Next}; +use axum::response::{IntoResponse, Response}; +use axum::routing::{get, post}; +use cache::Swr; +use serde_json::{Map, Value, json}; +use sqlx::PgPool; +use std::sync::{Arc, Mutex, RwLock}; +use std::time::{Duration, Instant}; +use tower_http::compression::CompressionLayer; +use tower_http::compression::predicate::{NotForContentType, Predicate, SizeAbove}; +use tower_http::cors::{AllowHeaders, AllowMethods, AllowOrigin, Any, CorsLayer}; +use tower_http::services::ServeDir; + +pub struct AppState { + pub cfg: Config, + pub pool: PgPool, + pub http: reqwest::Client, + pub stations: Arc>, + /// The collector's status when it runs in this process; otherwise + /// /scraping/status reads SCRAPING_STATUS_FILE, as the Python app did. + pub status: Option>>, + /// Wakes the collection loop (POST /scrape/trigger). + pub trigger: Option>, + pub dashboard: String, + pub metrics: Metrics, + pub started: Instant, + caches: Caches, +} + +pub type Shared = Arc; + +struct Caches { + latest: Arc>, + history: Arc>, + hii: Arc>, + stats: Arc>, + health: Arc>, + sensors: Arc>, + /// body, older than FORECAST_MAX_AGE_SECONDS + forecast: Arc>, +} + +impl AppState { + pub fn new( + cfg: Config, + pool: PgPool, + http: reqwest::Client, + stations: Arc>, + status: Option>>, + trigger: Option>, + ) -> Shared { + let path = cfg.web.static_dir.join("dashboard.html"); + let dashboard = std::fs::read_to_string(&path).unwrap_or_else(|e| { + tracing::error!("could not load the dashboard {}: {e}", path.display()); + "

Northern Thailand Ping River Monitor API

See /docs.

".into() + }); + Arc::new(AppState { + cfg, + pool, + http, + stations, + status, + trigger, + dashboard, + metrics: Metrics::default(), + started: Instant::now(), + // byte budgets: ~100 MB in all, inside the unit's MemoryMax + caches: Caches { + latest: Swr::new(4 * MB), + history: Swr::new(48 * MB), + hii: Swr::new(24 * MB), + stats: Swr::new(MB), + health: Swr::new(MB), + sensors: Swr::new(4 * MB), + forecast: Swr::new(4 * MB), + }, + }) + } +} + +const MB: usize = 1 << 20; + +fn secs(s: u64) -> Duration { + Duration::from_secs(s) +} + +/// `/metrics`: request counters per endpoint, as the Python app kept them. +#[derive(Default)] +pub struct Metrics { + counters: Mutex>, +} + +impl Metrics { + pub fn count(&self, endpoint: &str) { + let key = format!("api_requests{{endpoint={endpoint}}}"); + let mut c = self.counters.lock().unwrap(); + let n = c.get(&key).and_then(Value::as_f64).unwrap_or(0.0); + c.insert(key, json!(n + 1.0)); + } + + fn counters(&self) -> Value { + Value::Object(self.counters.lock().unwrap().clone()) + } +} + +/// Cache-Control by path, as the Python app's middleware: lets browsers and +/// Cloudflare absorb repeat traffic. Max-ages mirror the data cadence. +const CACHE_CONTROL: &[(&str, &str)] = &[ + ("/static/", "public, max-age=3600"), + ("/measurements/latest", "public, max-age=30"), + ("/api/forecast/", "public, max-age=300"), + ("/api/hii/", "public, max-age=60"), + ("/measurements/history", "public, max-age=300"), + ("/forecast", "public, max-age=120"), + ("/api/stats", "public, max-age=300"), + ("/stations", "public, max-age=300"), +]; + +/// Requests reported to Umami (server-side API analytics). +const UMAMI_PREFIXES: &[&str] = &[ + "/api/", + "/measurements", + "/forecast", + "/stations", + "/sensors", +]; + +fn cache_control(path: &str) -> Option<&'static str> { + if path == "/" { + return Some("public, max-age=120"); + } + CACHE_CONTROL + .iter() + .find(|(prefix, _)| path.starts_with(prefix)) + .map(|(_, v)| *v) +} + +async fn finish(State(st): State, req: Request, next: Next) -> Response { + let path = req.uri().path().to_string(); + let method = req.method().clone(); + let (host, user_agent) = { + let header = |name: header::HeaderName| { + req.headers() + .get(name) + .and_then(|v| v.to_str().ok()) + .map(String::from) + }; + ( + header(header::HOST).unwrap_or_else(|| "water.buildfor.life".into()), + header(header::USER_AGENT).unwrap_or_default(), + ) + }; + let mut resp = next.run(req).await; + if method == Method::GET + && resp.status() == StatusCode::OK + && !resp.headers().contains_key(header::CACHE_CONTROL) + && let Some(v) = cache_control(&path) + { + resp.headers_mut() + .insert(header::CACHE_CONTROL, HeaderValue::from_static(v)); + } + let web = &st.cfg.web; + if web.umami_track_api + && !web.umami_website_id.is_empty() + && UMAMI_PREFIXES.iter().any(|p| path.starts_with(p)) + { + let event = json!({ + "type": "event", + "payload": { + "website": web.umami_website_id, + "url": path, + "hostname": host, + "name": "api-request", + "data": {"method": method.as_str(), "status": resp.status().as_u16()}, + }, + }); + let req = st + .http + .post(&web.umami_api_url) + .timeout(Duration::from_secs(3)) + .header( + header::USER_AGENT, + if user_agent.is_empty() { + "api-client".into() + } else { + user_agent + }, + ) + .json(&event); + // analytics must never affect the response + tokio::spawn(async move { + let _ = req.send().await; + }); + } + resp +} + +async fn not_found() -> Response { + error::ApiError::new(404, "Not Found").into_response() +} + +async fn method_not_allowed() -> Response { + error::ApiError::new(405, "Method Not Allowed").into_response() +} + +fn cors(cfg: &Config) -> CorsLayer { + // as the Python app: any origin without credentials unless + // CORS_ALLOW_ORIGINS lists the allowed ones + let origins = &cfg.web.cors_allow_origins; + if origins.is_empty() { + return CorsLayer::new() + .allow_origin(Any) + .allow_methods(Any) + .allow_headers(Any); + } + let list: Vec = origins.iter().filter_map(|o| o.parse().ok()).collect(); + CorsLayer::new() + .allow_origin(AllowOrigin::list(list)) + .allow_credentials(true) + .allow_methods(AllowMethods::mirror_request()) + .allow_headers(AllowHeaders::mirror_request()) +} + +pub fn router(st: Shared) -> Router { + let static_files = + ServeDir::new(st.cfg.web.static_dir.clone()).not_found_service(not_found.into_service()); + // as Starlette's GZipMiddleware(minimum_size=500) + let compress = CompressionLayer::new().compress_when( + SizeAbove::new(500) + .and(NotForContentType::GRPC) + .and(NotForContentType::IMAGES) + .and(NotForContentType::SSE), + ); + Router::new() + .route("/", get(system::root)) + .route("/robots.txt", get(system::robots)) + .route("/llms.txt", get(system::llms)) + .route("/sitemap.xml", get(system::sitemap)) + .route("/docs", get(openapi::swagger_ui)) + .route("/redoc", get(openapi::redoc)) + .route("/openapi.json", get(openapi::spec)) + .route("/health", get(system::health)) + .route("/metrics", get(system::metrics)) + .route( + "/stations", + get(system::stations).post(system::create_station), + ) + .route( + "/stations/{station_id}", + get(system::station) + .put(system::update_station) + .delete(system::delete_station), + ) + .route("/sensors/thaiwater", get(hii::thaiwater_sensors)) + .route("/api/hii/rainfall/latest", get(hii::rainfall_latest)) + .route("/api/hii/rainfall/catchment", get(hii::rainfall_catchment)) + .route("/api/hii/waterlevel/latest", get(hii::waterlevel_latest)) + .route("/measurements/history/{station_code}", get(data::history)) + .route("/forecast", get(forecasts::forecast)) + .route( + "/api/forecast/history/{station_code}", + get(forecasts::history), + ) + .route("/api/notifications", get(forecasts::notifications)) + .route("/api/forecast/skill", get(forecasts::skill)) + .route("/measurements/latest", get(data::latest)) + .route("/measurements/station/{station_code}", get(data::station)) + .route("/api/stats", get(data::stats)) + .route("/scrape/trigger", post(system::trigger)) + .route("/scraping/status", get(system::scraping_status)) + .route("/config", get(system::config)) + .nest_service("/static", static_files) + .fallback(not_found) + .method_not_allowed_fallback(method_not_allowed) + .layer(middleware::from_fn_with_state(st.clone(), finish)) + .layer(cors(&st.cfg)) + .layer(compress) + .with_state(st) +} + +/// Serve until `shutdown` resolves. While the address is taken (the Python +/// app still running during a switch-over) binding is retried every 30 s. +pub async fn serve( + st: Shared, + listen: String, + shutdown: impl std::future::Future + Send + 'static, +) -> Result<()> { + let mut shutdown = Box::pin(shutdown); + let listener = loop { + match tokio::net::TcpListener::bind(&listen).await { + Ok(l) => break l, + Err(e) => { + tracing::error!("website: cannot listen on {listen}: {e}; retrying in 30 s"); + tokio::select! { + _ = tokio::time::sleep(Duration::from_secs(30)) => {} + _ = &mut shutdown => return Ok(()), + } + } + } + }; + tracing::info!("website and API on http://{listen}"); + axum::serve(listener, router(st)) + .with_graceful_shutdown(shutdown) + .await + .context("web server") +} diff --git a/rust/collector/src/web/openapi.rs b/rust/collector/src/web/openapi.rs new file mode 100644 index 0000000..fcfb56d --- /dev/null +++ b/rust/collector/src/web/openapi.rs @@ -0,0 +1,430 @@ +//! /docs, /redoc and /openapi.json. The document is built from the route +//! table below, whose parameter lists are the same constants the handlers +//! validate with, so the documented parameters are the accepted ones. +//! Summaries, operation ids and descriptions follow FastAPI's conventions +//! (from the Python handler names and docstrings) so /docs reads as before. + +use super::params::{Param, python_title}; +use super::{data, forecasts, hii, system}; +use axum::http::header; +use axum::response::{IntoResponse, Response}; +use serde_json::{Map, Value, json}; +use std::sync::OnceLock; + +pub const TITLE: &str = "Northern Thailand Ping River Monitor API"; +const DESCRIPTION: &str = + "Real-time water level monitoring system for Northern Thailand's Ping River Basin stations"; +const VERSION: &str = "3.1.3"; + +enum Resp { + Any, + Html, + Ref(&'static str), + ArrayOf(&'static str), +} + +struct Route { + method: &'static str, + path: &'static str, + /// the Python handler's name (summary and operationId) + name: &'static str, + description: &'static str, + params: &'static [Param], + body: Option<&'static str>, + response: Resp, +} + +const NONE: &[Param] = &[]; + +const ROUTES: &[Route] = &[ + Route { + method: "get", + path: "/", + name: "root", + description: "Root endpoint with basic dashboard", + params: NONE, + body: None, + response: Resp::Html, + }, + Route { + method: "get", + path: "/health", + name: "get_health", + description: "Get system health status", + params: NONE, + body: None, + response: Resp::Ref("HealthResponse"), + }, + Route { + method: "get", + path: "/metrics", + name: "get_metrics", + description: "Get application metrics", + params: NONE, + body: None, + response: Resp::Ref("MetricsResponse"), + }, + Route { + method: "get", + path: "/stations", + name: "get_stations", + description: "Get list of all monitoring stations", + params: NONE, + body: None, + response: Resp::ArrayOf("StationResponse"), + }, + Route { + method: "post", + path: "/stations", + name: "create_station", + description: "Create a new monitoring station", + params: system::ADMIN, + body: Some("StationCreateModel"), + response: Resp::Ref("StationResponse"), + }, + Route { + method: "put", + path: "/stations/{station_id}", + name: "update_station", + description: "Update an existing monitoring station", + params: system::ADMIN_STATION_ID, + body: Some("StationUpdateModel"), + response: Resp::Ref("StationResponse"), + }, + Route { + method: "delete", + path: "/stations/{station_id}", + name: "delete_station", + description: "Delete a monitoring station", + params: system::ADMIN_STATION_ID, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/stations/{station_id}", + name: "get_station", + description: "Get details of a specific monitoring station", + params: system::STATION_ID, + body: None, + response: Resp::Ref("StationResponse"), + }, + Route { + method: "get", + path: "/sensors/thaiwater", + name: "get_thaiwater_sensors", + description: "Get current ThaiWater water-level sensors in the Ping basin.", + params: NONE, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/api/hii/rainfall/latest", + name: "get_hii_rainfall_latest", + description: "Latest rainfall reading per HII station (Ping basin, collected hourly).", + params: hii::LATEST, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/api/hii/rainfall/catchment", + name: "get_hii_rainfall_catchment", + description: "Upper-Ping catchment-mean hourly rain: HII gauges vs the Open-Meteo\nseries the flood model actually uses, plus their agreement over the window.\n\nEvidence-gathering endpoint (docs/FLOOD_FORECASTING.md, HII gauge rain):\nthe gauge table only exists since 2026-08 so it cannot be a training\nfeature yet; this makes the two sources' relationship observable meanwhile.", + params: hii::CATCHMENT, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/api/hii/waterlevel/latest", + name: "get_hii_waterlevel_latest", + description: "Latest water-level reading per HII station (Ping basin, m MSL).", + params: hii::LATEST, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/measurements/history/{station_code}", + name: "get_postgres_history", + description: "Get historical measurements for a station from the configured database.", + params: data::HISTORY, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/forecast", + name: "get_flood_forecasts", + description: "Flood-risk forecasts per station for the 6/12/24 h horizons.", + params: NONE, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/api/forecast/history/{station_code}", + name: "get_forecast_history", + description: "Issued model forecasts for a station — what the model predicted, when.\n\nPopulated by the hourly precompute; enables predicted-vs-actual charts.", + params: forecasts::HISTORY, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/api/notifications", + name: "get_notifications_config", + description: "Public ntfy settings so the dashboard can offer subscribe links.", + params: NONE, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/api/forecast/skill", + name: "get_forecast_skill", + description: "Is the model getting better? Issued forecasts verified against what the\nriver then did, per model version, with a persistence baseline.\n\nRead from forecast_history (what each deployed version predicted, hourly)\njoined to water_measurements; no retraining involved. Cached like the HII\nfeeds because the join is a few hundred correlated subqueries.", + params: forecasts::SKILL, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/measurements/latest", + name: "get_latest_measurements", + description: "Get latest measurements from all stations", + params: data::LATEST, + body: None, + response: Resp::ArrayOf("MeasurementResponse"), + }, + Route { + method: "get", + path: "/measurements/station/{station_code}", + name: "get_station_measurements", + description: "Get measurements for a specific station", + params: data::STATION, + body: None, + response: Resp::ArrayOf("MeasurementResponse"), + }, + Route { + method: "get", + path: "/api/stats", + name: "get_database_stats", + description: "Get database coverage statistics (totals, date range, hourly coverage)", + params: NONE, + body: None, + response: Resp::Any, + }, + Route { + method: "post", + path: "/scrape/trigger", + name: "trigger_scraping", + description: "Trigger manual data scraping", + params: system::ADMIN, + body: None, + response: Resp::Any, + }, + Route { + method: "get", + path: "/scraping/status", + name: "get_scraping_status", + description: "Get current scraping status", + params: NONE, + body: None, + response: Resp::Ref("ScrapingStatusResponse"), + }, + Route { + method: "get", + path: "/config", + name: "get_config", + description: "Get current configuration (sensitive data masked)", + params: system::ADMIN, + body: None, + response: Resp::Any, + }, +]; + +/// FastAPI's `generate_unique_id`: name + path with non-word characters as +/// `_`, then the method. +fn operation_id(r: &Route) -> String { + let p: String = format!("{}{}", r.name, r.path) + .chars() + .map(|c| { + if c.is_alphanumeric() || c == '_' { + c + } else { + '_' + } + }) + .collect(); + format!("{p}_{}", r.method) +} + +fn reference(name: &str) -> Value { + json!({ "$ref": format!("#/components/schemas/{name}") }) +} + +fn operation(r: &Route) -> Value { + let op_id = operation_id(r); + let ok_content = match r.response { + Resp::Html => json!({"text/html": {"schema": {"type": "string"}}}), + Resp::Any => json!({"application/json": {"schema": {}}}), + Resp::Ref(name) => json!({"application/json": {"schema": reference(name)}}), + Resp::ArrayOf(name) => json!({"application/json": {"schema": { + "type": "array", + "items": reference(name), + "title": format!("Response {}", python_title(&op_id.replace('_', " "))), + }}}), + }; + let mut responses = Map::new(); + responses.insert( + "200".into(), + json!({"description": "Successful Response", "content": ok_content}), + ); + if !r.params.is_empty() || r.body.is_some() { + responses.insert( + "422".into(), + json!({"description": "Validation Error", + "content": {"application/json": {"schema": reference("HTTPValidationError")}}}), + ); + } + let mut op = Map::new(); + op.insert( + "summary".into(), + json!(python_title(&r.name.replace('_', " "))), + ); + op.insert("description".into(), json!(r.description)); + op.insert("operationId".into(), json!(op_id)); + if !r.params.is_empty() { + op.insert( + "parameters".into(), + Value::Array(r.params.iter().map(Param::openapi).collect()), + ); + } + if let Some(body) = r.body { + op.insert( + "requestBody".into(), + json!({"required": true, "content": {"application/json": {"schema": reference(body)}}}), + ); + } + op.insert("responses".into(), Value::Object(responses)); + Value::Object(op) +} + +pub fn document() -> &'static str { + static DOC: OnceLock = OnceLock::new(); + DOC.get_or_init(|| { + let mut paths = Map::new(); + for r in ROUTES { + let entry = paths + .entry(r.path) + .or_insert_with(|| Value::Object(Map::new())); + entry + .as_object_mut() + .expect("object") + .insert(r.method.into(), operation(r)); + } + let schemas: Value = serde_json::from_str(include_str!("openapi_components.json")) + .expect("openapi_components.json is valid JSON"); + json!({ + "openapi": "3.1.0", + "info": {"title": TITLE, "description": DESCRIPTION, "version": VERSION}, + "paths": paths, + "components": {"schemas": schemas}, + }) + .to_string() + }) +} + +pub async fn spec() -> Response { + ([(header::CONTENT_TYPE, "application/json")], document()).into_response() +} + +fn html(body: String) -> Response { + ([(header::CONTENT_TYPE, "text/html; charset=utf-8")], body).into_response() +} + +/// The Swagger UI page FastAPI serves (assets from the jsDelivr CDN). +pub async fn swagger_ui() -> Response { + html(format!( + r##" + + + + +{TITLE} - Swagger UI + + +
+
+ + + + +"## + )) +} + +pub async fn redoc() -> Response { + html(format!( + r#" + + +{TITLE} - ReDoc + + + + + + + + + + +"# + )) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn fastapi_naming() { + let doc: Value = serde_json::from_str(document()).unwrap(); + let op = &doc["paths"]["/measurements/history/{station_code}"]["get"]; + assert_eq!( + op["operationId"], + "get_postgres_history_measurements_history__station_code__get" + ); + assert_eq!(op["summary"], "Get Postgres History"); + assert_eq!(op["parameters"].as_array().unwrap().len(), 5); + let latest = &doc["paths"]["/measurements/latest"]["get"]["responses"]["200"]; + assert_eq!( + latest["content"]["application/json"]["schema"]["title"], + "Response Get Latest Measurements Measurements Latest Get" + ); + assert!(doc["paths"]["/stations"]["post"]["responses"]["422"].is_object()); + assert!( + doc["paths"]["/health"]["get"]["responses"] + .get("422") + .is_none() + ); + assert!(doc["components"]["schemas"]["HTTPValidationError"].is_object()); + } +} diff --git a/rust/collector/src/web/openapi_components.json b/rust/collector/src/web/openapi_components.json new file mode 100644 index 0000000..9fccb04 --- /dev/null +++ b/rust/collector/src/web/openapi_components.json @@ -0,0 +1,478 @@ +{ + "HTTPValidationError": { + "properties": { + "detail": { + "items": { + "$ref": "#/components/schemas/ValidationError" + }, + "type": "array", + "title": "Detail" + } + }, + "type": "object", + "title": "HTTPValidationError" + }, + "HealthResponse": { + "properties": { + "overall_status": { + "type": "string", + "title": "Overall Status" + }, + "timestamp": { + "type": "string", + "title": "Timestamp" + }, + "checks": { + "additionalProperties": { + "additionalProperties": true, + "type": "object" + }, + "type": "object", + "title": "Checks" + } + }, + "type": "object", + "required": [ + "overall_status", + "timestamp", + "checks" + ], + "title": "HealthResponse" + }, + "MeasurementResponse": { + "properties": { + "timestamp": { + "type": "string", + "format": "date-time", + "title": "Timestamp" + }, + "station_code": { + "type": "string", + "title": "Station Code" + }, + "station_name_en": { + "type": "string", + "title": "Station Name En" + }, + "station_name_th": { + "type": "string", + "title": "Station Name Th" + }, + "water_level": { + "type": "number", + "title": "Water Level" + }, + "discharge": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Discharge" + }, + "discharge_percent": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Discharge Percent" + }, + "status": { + "type": "string", + "title": "Status", + "default": "active" + } + }, + "type": "object", + "required": [ + "timestamp", + "station_code", + "station_name_en", + "station_name_th", + "water_level" + ], + "title": "MeasurementResponse" + }, + "MetricsResponse": { + "properties": { + "counters": { + "additionalProperties": { + "type": "number" + }, + "type": "object", + "title": "Counters" + }, + "gauges": { + "additionalProperties": { + "type": "number" + }, + "type": "object", + "title": "Gauges" + }, + "histograms": { + "additionalProperties": { + "additionalProperties": { + "type": "number" + }, + "type": "object" + }, + "type": "object", + "title": "Histograms" + } + }, + "type": "object", + "required": [ + "counters", + "gauges", + "histograms" + ], + "title": "MetricsResponse" + }, + "ScrapingStatusResponse": { + "properties": { + "is_running": { + "type": "boolean", + "title": "Is Running" + }, + "last_run": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Last Run" + }, + "next_run": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Next Run" + }, + "total_runs": { + "type": "integer", + "title": "Total Runs", + "default": 0 + }, + "successful_runs": { + "type": "integer", + "title": "Successful Runs", + "default": 0 + }, + "failed_runs": { + "type": "integer", + "title": "Failed Runs", + "default": 0 + }, + "feed_current": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Feed Current" + }, + "latest_reading": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Latest Reading" + }, + "readings_saved": { + "type": "integer", + "title": "Readings Saved", + "default": 0 + } + }, + "type": "object", + "required": [ + "is_running" + ], + "title": "ScrapingStatusResponse" + }, + "StationCreateModel": { + "properties": { + "station_code": { + "type": "string", + "title": "Station Code", + "description": "Station code (e.g., P.1, P.20)" + }, + "thai_name": { + "type": "string", + "title": "Thai Name", + "description": "Thai name of the station" + }, + "english_name": { + "type": "string", + "title": "English Name", + "description": "English name of the station" + }, + "latitude": { + "anyOf": [ + { + "type": "number", + "maximum": 90.0, + "minimum": -90.0 + }, + { + "type": "null" + } + ], + "title": "Latitude", + "description": "Latitude coordinate" + }, + "longitude": { + "anyOf": [ + { + "type": "number", + "maximum": 180.0, + "minimum": -180.0 + }, + { + "type": "null" + } + ], + "title": "Longitude", + "description": "Longitude coordinate" + }, + "geohash": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Geohash", + "description": "Geohash for the location" + }, + "status": { + "type": "string", + "title": "Status", + "description": "Station status", + "default": "active" + } + }, + "type": "object", + "required": [ + "station_code", + "thai_name", + "english_name" + ], + "title": "StationCreateModel" + }, + "StationResponse": { + "properties": { + "station_id": { + "type": "integer", + "title": "Station Id" + }, + "station_code": { + "type": "string", + "title": "Station Code" + }, + "thai_name": { + "type": "string", + "title": "Thai Name" + }, + "english_name": { + "type": "string", + "title": "English Name" + }, + "latitude": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Latitude" + }, + "longitude": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Longitude" + }, + "geohash": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Geohash" + }, + "status": { + "type": "string", + "title": "Status", + "default": "active" + } + }, + "type": "object", + "required": [ + "station_id", + "station_code", + "thai_name", + "english_name" + ], + "title": "StationResponse" + }, + "StationUpdateModel": { + "properties": { + "thai_name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Thai Name", + "description": "Thai name of the station" + }, + "english_name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "English Name", + "description": "English name of the station" + }, + "latitude": { + "anyOf": [ + { + "type": "number", + "maximum": 90.0, + "minimum": -90.0 + }, + { + "type": "null" + } + ], + "title": "Latitude", + "description": "Latitude coordinate" + }, + "longitude": { + "anyOf": [ + { + "type": "number", + "maximum": 180.0, + "minimum": -180.0 + }, + { + "type": "null" + } + ], + "title": "Longitude", + "description": "Longitude coordinate" + }, + "geohash": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Geohash", + "description": "Geohash for the location" + }, + "status": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Status", + "description": "Station status" + } + }, + "type": "object", + "title": "StationUpdateModel" + }, + "ValidationError": { + "properties": { + "loc": { + "items": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + } + ] + }, + "type": "array", + "title": "Location" + }, + "msg": { + "type": "string", + "title": "Message" + }, + "type": { + "type": "string", + "title": "Error Type" + }, + "input": { + "title": "Input" + }, + "ctx": { + "type": "object", + "title": "Context" + } + }, + "type": "object", + "required": [ + "loc", + "msg", + "type" + ], + "title": "ValidationError" + } +} diff --git a/rust/collector/src/web/params.rs b/rust/collector/src/web/params.rs new file mode 100644 index 0000000..99fbb16 --- /dev/null +++ b/rust/collector/src/web/params.rs @@ -0,0 +1,450 @@ +//! Query/path parameters: one declaration per route drives both the +//! validation (pydantic-style 422 errors, as FastAPI) and the OpenAPI +//! document behind /docs, so the two cannot drift apart. + +use super::error::ApiError; +use chrono::{NaiveDate, NaiveDateTime}; +use serde_json::{Map, Value, json}; +use std::collections::HashMap; + +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum In { + Query, + Path, + Header, +} + +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Kind { + Int, + Str, + Date, +} + +#[derive(Clone, Copy, Debug)] +pub enum Default { + None, + Int(i64), + Str(&'static str), +} + +#[derive(Clone, Copy, Debug)] +pub struct Param { + pub name: &'static str, + pub loc: In, + pub kind: Kind, + pub required: bool, + /// `Optional[...] = None` + pub nullable: bool, + pub default: Default, + pub ge: Option, + pub le: Option, + pub description: Option<&'static str>, +} + +impl Param { + const fn new(name: &'static str, loc: In, kind: Kind) -> Self { + Param { + name, + loc, + kind, + required: matches!(loc, In::Path), + nullable: false, + default: Default::None, + ge: None, + le: None, + description: None, + } + } + pub const fn path_str(name: &'static str) -> Self { + Self::new(name, In::Path, Kind::Str) + } + pub const fn path_int(name: &'static str) -> Self { + Self::new(name, In::Path, Kind::Int) + } + pub const fn int(name: &'static str, default: i64) -> Self { + Param { + default: Default::Int(default), + ..Self::new(name, In::Query, Kind::Int) + } + } + pub const fn opt_int(name: &'static str) -> Self { + Param { + nullable: true, + ..Self::new(name, In::Query, Kind::Int) + } + } + pub const fn str(name: &'static str, default: &'static str) -> Self { + Param { + default: Default::Str(default), + ..Self::new(name, In::Query, Kind::Str) + } + } + pub const fn opt_date(name: &'static str, description: &'static str) -> Self { + Param { + nullable: true, + description: Some(description), + ..Self::new(name, In::Query, Kind::Date) + } + } + /// The optional `X-API-Key` header of the admin endpoints. + pub const fn api_key() -> Self { + Param { + nullable: true, + ..Self::new("X-API-Key", In::Header, Kind::Str) + } + } + pub const fn ge(self, v: i64) -> Self { + Param { + ge: Some(v), + ..self + } + } + pub const fn le(self, v: i64) -> Self { + Param { + le: Some(v), + ..self + } + } + + fn loc_name(&self) -> &'static str { + match self.loc { + In::Query => "query", + In::Path => "path", + In::Header => "header", + } + } + + /// pydantic's default title: `snake_case` / `X-API-Key` -> `Snake Case` / `X-Api-Key` + pub fn title(&self) -> String { + python_title(&self.name.replace('_', " ")) + } + + /// The OpenAPI parameter object, as FastAPI writes it. + pub fn openapi(&self) -> Value { + let mut base = Map::new(); + base.insert( + "type".into(), + json!(match self.kind { + Kind::Int => "integer", + Kind::Str | Kind::Date => "string", + }), + ); + if self.kind == Kind::Date { + base.insert("format".into(), json!("date")); + } + if let Some(le) = self.le { + base.insert("maximum".into(), json!(le)); + } + if let Some(ge) = self.ge { + base.insert("minimum".into(), json!(ge)); + } + let mut schema = if self.nullable { + let mut m = Map::new(); + m.insert( + "anyOf".into(), + json!([Value::Object(base), {"type": "null"}]), + ); + m + } else { + base + }; + match self.default { + Default::Int(d) => { + schema.insert("default".into(), json!(d)); + } + Default::Str(d) => { + schema.insert("default".into(), json!(d)); + } + Default::None => {} + } + if let Some(d) = self.description { + schema.insert("description".into(), json!(d)); + } + schema.insert("title".into(), json!(self.title())); + let mut p = Map::new(); + p.insert("name".into(), json!(self.name)); + p.insert("in".into(), json!(self.loc_name())); + p.insert("required".into(), json!(self.required)); + p.insert("schema".into(), Value::Object(schema)); + if let Some(d) = self.description { + p.insert("description".into(), json!(d)); + } + Value::Object(p) + } +} + +/// Python's `str.title()` for ASCII. +pub fn python_title(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + let mut prev_letter = false; + for c in s.chars() { + out.extend(if prev_letter { + c.to_lowercase().collect::>() + } else { + c.to_uppercase().collect::>() + }); + prev_letter = c.is_alphabetic(); + } + out +} + +#[derive(Clone, Debug, PartialEq)] +pub enum Val { + Int(i64), + Str(String), + Date(NaiveDate), + None, +} + +#[derive(Debug, Default)] +pub struct Args(HashMap<&'static str, Val>); + +impl Args { + pub fn int(&self, name: &str) -> i64 { + match self.0.get(name) { + Some(Val::Int(v)) => *v, + other => panic!("parameter {name}: expected an int, got {other:?}"), + } + } + pub fn opt_int(&self, name: &str) -> Option { + match self.0.get(name) { + Some(Val::Int(v)) => Some(*v), + _ => None, + } + } + pub fn str(&self, name: &str) -> &str { + match self.0.get(name) { + Some(Val::Str(v)) => v, + other => panic!("parameter {name}: expected a string, got {other:?}"), + } + } + pub fn date(&self, name: &str) -> Option { + match self.0.get(name) { + Some(Val::Date(d)) => Some(*d), + _ => None, + } + } + /// Python's `str()` of the value (cache keys built like the Python app's). + pub fn py_str(&self, name: &str) -> String { + match self.0.get(name) { + Some(Val::Int(v)) => v.to_string(), + Some(Val::Str(s)) => s.clone(), + Some(Val::Date(d)) => d.format("%Y-%m-%d").to_string(), + Some(Val::None) | None => "None".into(), + } + } +} + +/// The query string as a map; a repeated name keeps its last value (as +/// Starlette's `QueryParams.get`). +pub fn query_map(raw: Option<&str>) -> HashMap { + form_urlencoded::parse(raw.unwrap_or("").as_bytes()) + .map(|(k, v)| (k.into_owned(), v.into_owned())) + .collect() +} + +fn error(kind: &str, p: &Param, msg: String, input: Value, ctx: Option) -> Value { + let mut e = Map::new(); + e.insert("type".into(), json!(kind)); + e.insert("loc".into(), json!([p.loc_name(), p.name])); + e.insert("msg".into(), json!(msg)); + e.insert("input".into(), input); + if let Some(c) = ctx { + e.insert("ctx".into(), c); + } + Value::Object(e) +} + +fn parse_date(s: &str) -> Result { + if s.len() < 10 { + return Err("input is too short"); + } + if let Ok(d) = NaiveDate::parse_from_str(s, "%Y-%m-%d") { + return Ok(d); + } + // pydantic accepts a datetime at midnight for a date + for fmt in [ + "%Y-%m-%dT%H:%M:%S%.f", + "%Y-%m-%dT%H:%M:%S", + "%Y-%m-%d %H:%M:%S", + ] { + if let Ok(t) = NaiveDateTime::parse_from_str(s, fmt) { + if t.time() == chrono::NaiveTime::MIN { + return Ok(t.date()); + } + return Err("Datetimes provided to dates should have zero time - e.g. be exact dates"); + } + } + Err("invalid date format, expected YYYY-MM-DD") +} + +/// Validate every declared parameter; all problems are reported at once. +pub fn parse( + spec: &[Param], + query: &HashMap, + path: &[(&str, &str)], + headers: &axum::http::HeaderMap, +) -> Result { + let mut args = Args::default(); + let mut errors = vec![]; + for p in spec { + let raw: Option = match p.loc { + In::Query => query.get(p.name).cloned(), + In::Path => path + .iter() + .find(|(k, _)| *k == p.name) + .map(|(_, v)| v.to_string()), + In::Header => headers + .get(p.name) + .and_then(|v| v.to_str().ok()) + .map(String::from), + }; + let Some(raw) = raw else { + if p.required { + errors.push(error( + "missing", + p, + "Field required".into(), + Value::Null, + None, + )); + continue; + } + let v = match p.default { + Default::Int(d) => Val::Int(d), + Default::Str(d) => Val::Str(d.into()), + Default::None => Val::None, + }; + args.0.insert(p.name, v); + continue; + }; + let v = match p.kind { + Kind::Str => Val::Str(raw), + Kind::Int => match raw.trim().parse::() { + Err(_) => { + errors.push(error( + "int_parsing", + p, + "Input should be a valid integer, unable to parse string as an integer" + .into(), + json!(raw), + None, + )); + continue; + } + Ok(v) => { + if let Some(ge) = p.ge + && v < ge + { + errors.push(error( + "greater_than_equal", + p, + format!("Input should be greater than or equal to {ge}"), + json!(raw), + Some(json!({ "ge": ge })), + )); + continue; + } + if let Some(le) = p.le + && v > le + { + errors.push(error( + "less_than_equal", + p, + format!("Input should be less than or equal to {le}"), + json!(raw), + Some(json!({ "le": le })), + )); + continue; + } + Val::Int(v) + } + }, + Kind::Date => match parse_date(&raw) { + Ok(d) => Val::Date(d), + Err(why) => { + errors.push(error( + "date_from_datetime_parsing", + p, + format!("Input should be a valid date or datetime, {why}"), + json!(raw), + Some(json!({ "error": why })), + )); + continue; + } + }, + }; + args.0.insert(p.name, v); + } + if errors.is_empty() { + Ok(args) + } else { + Err(ApiError::validation(errors)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const SPEC: &[Param] = &[ + Param::path_str("station_code"), + Param::int("hours", 168).ge(1), + Param::int("limit", 50000).ge(1).le(100000), + Param::opt_date("start", "First day (overrides hours)"), + Param::opt_int("horizon").ge(1).le(48), + ]; + + fn run(q: &str) -> Result { + parse( + SPEC, + &query_map(Some(q)), + &[("station_code", "P.1")], + &axum::http::HeaderMap::new(), + ) + } + + #[test] + fn defaults_and_values() { + let a = run("hours=6&start=2026-10-15&hours=12").unwrap(); + assert_eq!(a.int("hours"), 12); // last wins + assert_eq!(a.int("limit"), 50000); + assert_eq!(a.str("station_code"), "P.1"); + assert_eq!(a.date("start"), NaiveDate::from_ymd_opt(2026, 10, 15)); + assert_eq!(a.opt_int("horizon"), None); + assert_eq!(a.py_str("horizon"), "None"); + } + + #[test] + fn errors_like_fastapi() { + // FastAPI: {"detail":[{"type":"greater_than_equal","loc":["query","hours"], + // "msg":"Input should be greater than or equal to 1","input":"0","ctx":{"ge":1}}]} + let e = run("hours=0&limit=x").unwrap_err(); + assert_eq!(e.status.as_u16(), 422); + assert_eq!( + e.detail, + json!([ + {"type": "greater_than_equal", "loc": ["query", "hours"], + "msg": "Input should be greater than or equal to 1", "input": "0", "ctx": {"ge": 1}}, + {"type": "int_parsing", "loc": ["query", "limit"], + "msg": "Input should be a valid integer, unable to parse string as an integer", + "input": "x"} + ]) + ); + } + + #[test] + fn openapi_parameter_objects() { + assert_eq!( + SPEC[2].openapi().to_string(), + r#"{"name":"limit","in":"query","required":false,"schema":{"type":"integer","maximum":100000,"minimum":1,"default":50000,"title":"Limit"}}"# + ); + assert_eq!( + SPEC[4].openapi().to_string(), + r#"{"name":"horizon","in":"query","required":false,"schema":{"anyOf":[{"type":"integer","maximum":48,"minimum":1},{"type":"null"}],"title":"Horizon"}}"# + ); + assert_eq!(Param::api_key().title(), "X-Api-Key"); + assert_eq!(python_title("station code"), "Station Code"); + } +} diff --git a/rust/collector/src/web/pyfmt.rs b/rust/collector/src/web/pyfmt.rs new file mode 100644 index 0000000..aca5f71 --- /dev/null +++ b/rust/collector/src/web/pyfmt.rs @@ -0,0 +1,158 @@ +//! Output conventions of the Python app that the API keeps: `isoformat()`, +//! `round()`, numpy's summation order and Python's `str()` of JSON scalars. + +use chrono::{NaiveDate, NaiveDateTime, Timelike}; +use serde_json::Value; + +/// `datetime.isoformat()`: microseconds only when they are not zero. +pub fn isoformat(t: NaiveDateTime) -> String { + let base = t.format("%Y-%m-%dT%H:%M:%S"); + match t.nanosecond() / 1000 { + 0 => base.to_string(), + us => format!("{base}.{us:06}"), + } +} + +/// `str(datetime)`: as `isoformat()` with a space instead of the `T`. +pub fn py_str_datetime(t: NaiveDateTime) -> String { + isoformat(t).replacen('T', " ", 1) +} + +/// `datetime.combine(d, datetime.max.time())` +pub fn end_of_day(d: NaiveDate) -> NaiveDateTime { + d.and_hms_micro_opt(23, 59, 59, 999_999) + .expect("valid time") +} + +pub fn start_of_day(d: NaiveDate) -> NaiveDateTime { + d.and_hms_opt(0, 0, 0).expect("valid time") +} + +/// `round(x, ndigits)` for floats: the correctly rounded decimal, ties to +/// even (Rust's fixed-precision formatting rounds the same way). +pub fn py_round(x: f64, ndigits: usize) -> f64 { + if !x.is_finite() { + return x; + } + format!("{x:.ndigits$}") + .parse() + .expect("formatted float parses") +} + +/// numpy's `np.add.reduce` on a contiguous float64 array (pairwise +/// summation, 8 accumulators per block of up to 128), which pandas' +/// `Series.mean()` uses without bottleneck. +pub fn numpy_sum(a: &[f64]) -> f64 { + 0.0 + pairwise(a) +} + +fn pairwise(a: &[f64]) -> f64 { + let n = a.len(); + if n < 8 { + let mut res = -0.0; + for x in a { + res += x; + } + res + } else if n <= 128 { + let mut r = [0.0; 8]; + r.copy_from_slice(&a[..8]); + let mut i = 8; + while i < n - (n % 8) { + for j in 0..8 { + r[j] += a[i + j]; + } + i += 8; + } + let mut res = ((r[0] + r[1]) + (r[2] + r[3])) + ((r[4] + r[5]) + (r[6] + r[7])); + while i < n { + res += a[i]; + i += 1; + } + res + } else { + let mut n2 = n / 2; + n2 -= n2 % 8; + pairwise(&a[..n2]) + pairwise(&a[n2..]) + } +} + +/// `Series.mean()` (NaN-free input). +pub fn numpy_mean(a: &[f64]) -> f64 { + numpy_sum(a) / a.len() as f64 +} + +/// `str(x)` of a JSON scalar as Python would print the decoded value. +pub fn py_str(v: &Value) -> String { + match v { + Value::Null => "None".into(), + Value::Bool(true) => "True".into(), + Value::Bool(false) => "False".into(), + Value::String(s) => s.clone(), + other => other.to_string(), + } +} + +/// A float as JSON (`NaN` and infinities, which Python refuses to encode, +/// become null). +pub fn num(x: Option) -> Value { + x.filter(|x| x.is_finite()).map_or(Value::Null, Value::from) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn isoformat_drops_zero_microseconds() { + let d = NaiveDate::from_ymd_opt(2026, 10, 18).unwrap(); + assert_eq!( + isoformat(d.and_hms_opt(14, 0, 0).unwrap()), + "2026-10-18T14:00:00" + ); + assert_eq!( + isoformat(d.and_hms_micro_opt(15, 30, 38, 454_399).unwrap()), + "2026-10-18T15:30:38.454399" + ); + assert_eq!(isoformat(end_of_day(d)), "2026-10-18T23:59:59.999999"); + assert_eq!( + py_str_datetime(d.and_hms_opt(21, 0, 0).unwrap()), + "2026-10-18 21:00:00" + ); + } + + #[test] + fn rounding_matches_python() { + // values from CPython 3.11: round(x, n) + for (x, n, want) in [ + (0.125, 2, 0.12), + (0.375, 2, 0.38), + (2.675, 2, 2.67), + (1.0005, 3, 1.0), + (0.0875, 3, 0.087), + (-0.0049, 2, -0.0), + (93.35, 1, 93.3), + (24.6, 2, 24.6), + (1.2345e-7, 3, 0.0), + ] { + assert_eq!(py_round(x, n), want, "round({x}, {n})"); + } + } + + #[test] + fn numpy_pairwise_sum_order() { + // np.add.reduce / pd.Series.mean (numpy 1.26), where a naive loop + // gives 20.000000000000014, 1027.6428571428573, 1481.0200000000002 + assert_eq!(numpy_sum(&[0.1; 200]), 20.0); + let series = |n: usize| -> Vec { + (0..n) + .map(|i| ((i * 7919) % 1000) as f64 / 7.0 * 0.013 + 0.1) + .collect() + }; + assert_eq!(numpy_sum(&series(1000)), 1027.642857142857); + assert_eq!(numpy_sum(&series(1441)), 1481.02); + assert_eq!(numpy_mean(&series(1441)), 1.0277723802914642); + assert_eq!(numpy_sum(&series(300)), 308.84999999999997); + assert_eq!(numpy_sum(&[]), 0.0); + } +} diff --git a/rust/collector/src/web/system.rs b/rust/collector/src/web/system.rs new file mode 100644 index 0000000..92ef7fc --- /dev/null +++ b/rust/collector/src/web/system.rs @@ -0,0 +1,635 @@ +//! The dashboard, health, metrics, collector status and the admin endpoints. + +use super::Shared; +use super::error::{ApiError, ApiResult, ok, ok_json, to_body}; +use super::params::{self, Param}; +use super::pyfmt::{isoformat, py_str_datetime}; +use crate::stations::StationInfo; +use crate::util::bangkok_now; +use axum::body::Bytes; +use axum::extract::{Path, State}; +use axum::http::{HeaderMap, StatusCode, header}; +use axum::response::{IntoResponse, Response}; +use serde_json::{Map, Value, json}; +use std::time::{Duration, Instant}; + +/// GET /: the dashboard (src/static/dashboard.html, loaded at start). +pub async fn root(State(st): State) -> Response { + ( + [(header::CONTENT_TYPE, "text/html; charset=utf-8")], + st.dashboard.clone(), + ) + .into_response() +} + +async fn static_file(st: &Shared, name: &str, content_type: &'static str) -> Response { + match tokio::fs::read(st.cfg.web.static_dir.join(name)).await { + Ok(bytes) => ([(header::CONTENT_TYPE, content_type)], bytes).into_response(), + Err(_) => ApiError::new(404, "Not Found").into_response(), + } +} + +/// Crawler/indexing files at the domain root. +pub async fn robots(State(st): State) -> Response { + static_file(&st, "robots.txt", "text/plain; charset=utf-8").await +} + +pub async fn llms(State(st): State) -> Response { + static_file(&st, "llms.txt", "text/plain; charset=utf-8").await +} + +pub async fn sitemap(State(st): State) -> Response { + static_file(&st, "sitemap.xml", "application/xml").await +} + +// ---------------------------------------------------------------- /health + +struct Check { + status: &'static str, + message: String, +} + +fn check(status: &'static str, message: impl Into) -> Check { + Check { + status, + message: message.into(), + } +} + +/// Data freshness. As the Python check it judges the newest reading of the +/// first station (`get_latest_measurements(limit=1)`: P.1) against 2 h. One +/// change, as in the notifier: while RID's report lags (2-10 h at night) but +/// the ThaiWater telemetry of the same gauges is current, the data is current. +async fn database_check(st: &Shared) -> Check { + let latest: Result, _> = sqlx::query_as( + "SELECT m.timestamp + FROM water_measurements m + JOIN stations s ON m.station_id = s.id + INNER JOIN ( + SELECT station_id, MAX(timestamp) as max_timestamp + FROM water_measurements GROUP BY station_id + ) latest ON m.station_id = latest.station_id AND m.timestamp = latest.max_timestamp + ORDER BY s.station_code LIMIT 1", + ) + .fetch_optional(&st.pool) + .await; + let latest = match latest { + Ok(l) => l.map(|(t,)| t), + Err(e) => return check("unhealthy", format!("Database check failed: {e}")), + }; + let now = bangkok_now(); + let max_age = chrono::TimeDelta::hours(2); + if let Some(ts) = latest + && now - ts > max_age + { + let codes = &ping_forecast::config::config().hii_fill_stations; + if let Ok(Some(hii)) = crate::db::latest_hii_timestamp(&st.pool, codes).await + && now - hii <= max_age + { + return check( + "healthy", + format!( + "Database connection OK; RID report lags (latest {}), ThaiWater telemetry current (latest {})", + py_str_datetime(ts), + py_str_datetime(hii) + ), + ); + } + return check( + "degraded", + format!("Latest data is old: {}", py_str_datetime(ts)), + ); + } + check("healthy", "Database connection OK") +} + +/// The RID API answers (5 s, a liveness probe must fail fast). +async fn api_check(st: &Shared) -> Check { + match st + .http + .get(&st.cfg.rid_api_url) + .timeout(Duration::from_secs(5)) + .send() + .await + { + Ok(r) if r.status() == StatusCode::OK => check("healthy", "API connection OK"), + Ok(r) => check( + "degraded", + format!("API returned status {}", r.status().as_u16()), + ), + Err(e) => check("unhealthy", format!("API check failed: {e}")), + } +} + +/// Resident memory of this process in MB (Linux). +pub fn rss_mb() -> Option { + let status = std::fs::read_to_string("/proc/self/status").ok()?; + let kb: f64 = status + .lines() + .find(|l| l.starts_with("VmRSS:"))? + .split_whitespace() + .nth(1)? + .parse() + .ok()?; + Some(kb / 1024.0) +} + +async fn memory_check() -> Check { + match rss_mb() { + Some(mb) if mb > 1000.0 => check("degraded", format!("High memory usage: {mb:.1}MB")), + Some(mb) => check("healthy", format!("Memory usage OK: {mb:.1}MB")), + None => check("healthy", "Memory check skipped (no /proc/self/status)"), + } +} + +async fn timed>( + f: F, +) -> (Check, f64, chrono::NaiveDateTime) { + let t = Instant::now(); + let c = f.await; + (c, t.elapsed().as_secs_f64() * 1000.0, bangkok_now()) +} + +async fn compute_health(st: Shared) -> super::cache::Computed { + let results = [ + ("database", timed(database_check(&st)).await), + ("api", timed(api_check(&st)).await), + ("memory", timed(memory_check()).await), + ]; + let statuses: Vec<&str> = results.iter().map(|(_, (c, _, _))| c.status).collect(); + let overall = if statuses.contains(&"unhealthy") { + "unhealthy" + } else if statuses.contains(&"degraded") { + "degraded" + } else { + "healthy" + }; + let mut checks = Map::new(); + for (name, (c, ms, at)) in results { + checks.insert( + name.into(), + json!({ + "status": c.status, + "message": c.message, + "response_time_ms": ms, + "timestamp": isoformat(at), + }), + ); + } + let body = json!({ + "overall_status": overall, + "timestamp": isoformat(bangkok_now()), + "checks": checks, + }); + Ok((to_body(&body), true)) +} + +/// GET /health: cached briefly (it probes the RID API), never older than 4 TTLs. +pub async fn health(State(st): State) -> ApiResult { + st.metrics.count("health"); + let ttl = st.cfg.web.health_cache_ttl_s; + let me = st.clone(); + let (body, _) = st + .caches + .health + .get( + "health".into(), + Duration::from_secs(ttl), + Some(Duration::from_secs(4 * ttl)), + move || compute_health(me.clone()), + ) + .await?; + ok(body, false) +} + +/// GET /metrics +pub async fn metrics(State(st): State) -> ApiResult { + st.metrics.count("metrics"); + let mut gauges = Map::new(); + gauges.insert( + "uptime_seconds".into(), + json!(st.started.elapsed().as_secs_f64()), + ); + if let Some(mb) = rss_mb() { + gauges.insert("memory_rss_mb".into(), json!(mb)); + } + gauges.insert("db_pool_connections".into(), json!(st.pool.size() as f64)); + ok_json(&json!({ + "counters": st.metrics.counters(), + "gauges": gauges, + "histograms": {}, + })) +} + +// -------------------------------------------------------------- stations + +fn station_json(id: u32, s: &StationInfo, geohash: bool, status: &str) -> Value { + json!({ + "station_id": id, + "station_code": s.code, + "thai_name": s.thai_name, + "english_name": s.english_name, + "latitude": s.latitude, + "longitude": s.longitude, + "geohash": if geohash { json!(s.geohash) } else { Value::Null }, + "status": status, + }) +} + +/// GET /stations: the RID gauges (report column -> station). +pub async fn stations(State(st): State) -> ApiResult { + st.metrics.count("stations"); + let list: Vec = st + .stations + .read() + .unwrap() + .iter() + .map(|(id, s)| station_json(*id, s, false, "active")) + .collect(); + ok_json(&list) +} + +pub const STATION_ID: &[Param] = &[Param::path_int("station_id")]; +pub const ADMIN_STATION_ID: &[Param] = &[Param::path_int("station_id"), Param::api_key()]; +pub const ADMIN: &[Param] = &[Param::api_key()]; + +fn station_id(id: &str, h: &HeaderMap) -> Result { + let a = params::parse(STATION_ID, &Default::default(), &[("station_id", id)], h)?; + // ids beyond u32 cannot exist + Ok(u32::try_from(a.int("station_id")).unwrap_or(u32::MAX)) +} + +fn not_found() -> ApiError { + ApiError::new(404, "Station not found") +} + +/// GET /stations/{station_id} +pub async fn station(State(st): State, Path(id): Path, h: HeaderMap) -> ApiResult { + st.metrics.count("get_station"); + let id = station_id(&id, &h)?; + let stations = st.stations.read().unwrap(); + let s = stations.info(id).ok_or_else(not_found)?; + ok_json(&station_json(id, s, true, "active")) +} + +/// `require_admin_key`: the X-API-Key header must match ADMIN_API_KEY; +/// with no key configured the admin endpoints are disabled. +fn require_admin(st: &Shared, h: &HeaderMap) -> Result<(), ApiError> { + let Some(want) = &st.cfg.web.admin_api_key else { + return Err(ApiError::new( + 503, + "Admin API disabled: ADMIN_API_KEY is not configured on the server", + )); + }; + let given = h.get("x-api-key").map(|v| v.as_bytes()).unwrap_or_default(); + // constant time, as secrets.compare_digest + let same = given.len() == want.len() + && given + .iter() + .zip(want.as_bytes()) + .fold(0u8, |acc, (a, b)| acc | (a ^ b)) + == 0; + if same { + Ok(()) + } else { + Err(ApiError::new(401, "Invalid or missing X-API-Key")) + } +} + +fn body_object(body: &Bytes) -> Result, ApiError> { + match serde_json::from_slice::(body) { + Ok(Value::Object(m)) => Ok(m), + Ok(other) => Err(ApiError::validation(vec![json!({ + "type": "model_attributes_type", "loc": ["body"], + "msg": "Input should be a valid dictionary or object to extract fields from", + "input": other, + })])), + Err(e) => Err(ApiError::validation(vec![json!({ + "type": "json_invalid", "loc": ["body", e.column()], + "msg": "JSON decode error", "input": {}, "ctx": {"error": e.to_string()}, + })])), + } +} + +/// Field checks of StationCreateModel / StationUpdateModel. +struct Fields<'a> { + body: &'a Map, + errors: Vec, +} + +impl<'a> Fields<'a> { + fn err(&mut self, kind: &str, field: &str, msg: String, ctx: Option) { + let mut e = json!({ + "type": kind, "loc": ["body", field], "msg": msg, + "input": self.body.get(field).cloned().unwrap_or(Value::Object(self.body.clone())), + }); + if let Some(c) = ctx { + e["ctx"] = c; + } + self.errors.push(e); + } + + fn string(&mut self, field: &str, required: bool) -> Option { + match self.body.get(field) { + None if required => { + self.err("missing", field, "Field required".into(), None); + None + } + None | Some(Value::Null) if !required => None, + Some(Value::String(s)) => Some(s.clone()), + _ => { + self.err( + "string_type", + field, + "Input should be a valid string".into(), + None, + ); + None + } + } + } + + fn coord(&mut self, field: &str, limit: f64) -> Option { + let v = match self.body.get(field) { + None | Some(Value::Null) => return None, + Some(Value::Number(n)) => n.as_f64(), + Some(Value::String(s)) => s.trim().parse().ok(), + _ => None, + }; + let Some(x) = v else { + self.err( + "float_parsing", + field, + "Input should be a valid number, unable to parse string as a number".into(), + None, + ); + return None; + }; + if x < -limit { + self.err( + "greater_than_equal", + field, + format!("Input should be greater than or equal to {}", -limit), + Some(json!({"ge": -limit})), + ); + } else if x > limit { + self.err( + "less_than_equal", + field, + format!("Input should be less than or equal to {limit}"), + Some(json!({"le": limit})), + ); + } + Some(x) + } + + fn finish(self) -> Result<(), ApiError> { + if self.errors.is_empty() { + Ok(()) + } else { + Err(ApiError::validation(self.errors)) + } + } +} + +fn persist(st: &Shared) -> bool { + let stations = st.stations.read().unwrap(); + match stations.save(&st.cfg.station_config_path) { + Ok(()) => true, + Err(e) => { + tracing::error!( + "Failed to persist station mapping to {}: {e:#}", + st.cfg.station_config_path.display() + ); + false + } + } +} + +/// POST /stations (admin) +pub async fn create_station(State(st): State, h: HeaderMap, body: Bytes) -> ApiResult { + st.metrics.count("create_station"); + require_admin(&st, &h)?; + let body = body_object(&body)?; + let mut f = Fields { + body: &body, + errors: vec![], + }; + let code = f.string("station_code", true); + let thai = f.string("thai_name", true); + let english = f.string("english_name", true); + let latitude = f.coord("latitude", 90.0); + let longitude = f.coord("longitude", 180.0); + let geohash = f.string("geohash", false); + let status = f.string("status", false).unwrap_or_else(|| "active".into()); + f.finish()?; + let info = StationInfo { + code: code.unwrap_or_default(), + thai_name: thai.unwrap_or_default(), + english_name: english.unwrap_or_default(), + latitude, + longitude, + geohash, + }; + let id = { + let mut stations = st.stations.write().unwrap(); + let id = stations.next_id(); + stations.insert(id, info.clone()); + id + }; + if !persist(&st) { + st.stations.write().unwrap().remove(id); + return Err(ApiError::new(500, "Failed to persist new station")); + } + tracing::info!("Created new station: {} ({})", info.code, info.english_name); + ok_json(&station_json(id, &info, true, &status)) +} + +/// PUT /stations/{station_id} (admin) +pub async fn update_station( + State(st): State, + Path(id): Path, + h: HeaderMap, + body: Bytes, +) -> ApiResult { + st.metrics.count("update_station"); + require_admin(&st, &h)?; + let id = station_id(&id, &h)?; + let body = body_object(&body)?; + let mut f = Fields { + body: &body, + errors: vec![], + }; + let thai = f.string("thai_name", false); + let english = f.string("english_name", false); + let latitude = f.coord("latitude", 90.0); + let longitude = f.coord("longitude", 180.0); + let geohash = f.string("geohash", false); + let status = f.string("status", false); + f.finish()?; + let (original, updated) = { + let mut stations = st.stations.write().unwrap(); + let s = stations.info_mut(id).ok_or_else(not_found)?; + let original = s.clone(); + if let Some(v) = thai { + s.thai_name = v; + } + if let Some(v) = english { + s.english_name = v; + } + if latitude.is_some() { + s.latitude = latitude; + } + if longitude.is_some() { + s.longitude = longitude; + } + if geohash.is_some() { + s.geohash = geohash; + } + (original, s.clone()) + }; + if !persist(&st) { + st.stations.write().unwrap().insert(id, original); + return Err(ApiError::new(500, "Failed to persist station update")); + } + tracing::info!("Updated station {id}: {}", updated.code); + ok_json(&station_json( + id, + &updated, + true, + status + .as_deref() + .filter(|s| !s.is_empty()) + .unwrap_or("active"), + )) +} + +/// DELETE /stations/{station_id} (admin) +pub async fn delete_station( + State(st): State, + Path(id): Path, + h: HeaderMap, +) -> ApiResult { + st.metrics.count("delete_station"); + require_admin(&st, &h)?; + let id = station_id(&id, &h)?; + let removed = st + .stations + .write() + .unwrap() + .remove(id) + .ok_or_else(not_found)?; + if !persist(&st) { + st.stations.write().unwrap().insert(id, removed); + return Err(ApiError::new(500, "Failed to persist station deletion")); + } + tracing::info!("Deleted station {id}: {}", removed.code); + ok_json(&json!({"message": format!("Station {} deleted successfully", removed.code)})) +} + +// ------------------------------------------------------------ collection + +/// GET /scraping/status: the collector's last cycle (in this process, or +/// SCRAPING_STATUS_FILE when serving alone). +pub async fn scraping_status(State(st): State) -> ApiResult { + st.metrics.count("scraping_status"); + let status: Value = match &st.status { + Some(s) => s.read().unwrap().clone(), + None => st + .cfg + .status_file + .as_ref() + .and_then(|p| std::fs::read_to_string(p).ok()) + .and_then(|t| serde_json::from_str(&t).ok()) + .unwrap_or_else(|| json!({"is_running": false})), + }; + // ScrapingStatusResponse's fields and defaults first, then what the + // collector adds (version, forecast run) + let defaults = [ + ("is_running", json!(false)), + ("last_run", Value::Null), + ("next_run", Value::Null), + ("total_runs", json!(0)), + ("successful_runs", json!(0)), + ("failed_runs", json!(0)), + ("feed_current", Value::Null), + ("latest_reading", Value::Null), + ("readings_saved", json!(0)), + ]; + let mut out = Map::new(); + for (k, d) in defaults { + out.insert(k.into(), status.get(k).cloned().unwrap_or(d)); + } + for (k, v) in status.as_object().into_iter().flatten() { + if !out.contains_key(k) { + out.insert(k.clone(), v.clone()); + } + } + ok_json(&Value::Object(out)) +} + +/// POST /scrape/trigger (admin): poll every source now. +pub async fn trigger(State(st): State, h: HeaderMap) -> ApiResult { + st.metrics.count("scrape_trigger"); + require_admin(&st, &h)?; + let Some(trigger) = &st.trigger else { + return Err(ApiError::new( + 409, + "Collection does not run in this process (ping-collector serve)", + )); + }; + let running = st + .status + .as_ref() + .is_some_and(|s| s.read().unwrap()["is_running"] == true); + if running { + return Err(ApiError::new(409, "Scraping already in progress")); + } + trigger.notify_one(); + ok_json(&json!({"message": "Scraping triggered", "status": "started"})) +} + +/// GET /config (admin): the settings, secrets masked. +pub async fn config(State(st): State, h: HeaderMap) -> ApiResult { + st.metrics.count("config"); + require_admin(&st, &h)?; + let c = &st.cfg; + let w = &c.web; + let mut out = json!({ + "DB_TYPE": "postgresql", + "DATABASE_URL": c.redacted_database_url(), + "API_URL": c.rid_api_url, + "REQUEST_TIMEOUT": w.request_timeout_s, + "RID_POLL_OFFSET_MINUTES": c.poll_offset_minutes, + "RID_RETRY_MINUTES": c.retry_minutes, + "GAP_FILL_DAYS_ON_START": c.gap_fill_days_on_start, + "ENABLE_HII_COLLECTION": c.hii_enabled, + "ENABLE_RESERVOIR_COLLECTION": c.dams_enabled, + "ENABLE_RAIN_COLLECTION": c.rain_enabled, + "FORECAST_IN_COLLECTOR": c.forecast_enabled, + "MODELS_DIR": c.models_dir.display().to_string(), + "STATION_CONFIG_PATH": c.station_config_path.display().to_string(), + "SCRAPING_STATUS_FILE": c.status_file.as_ref().map(|p| p.display().to_string()), + "NTFY_SERVER": c.ntfy_server, + "NTFY_TOPIC_PREFIX": c.ntfy_topic_prefix, + "MATRIX_ENABLED": c.matrix.is_some(), + "WEB_LISTEN": w.listen, + "STATIC_DIR": w.static_dir.display().to_string(), + "CORS_ALLOW_ORIGINS": w.cors_allow_origins, + "HII_CACHE_TTL_SECONDS": w.hii_cache_ttl_s, + "LATEST_CACHE_TTL_SECONDS": w.latest_cache_ttl_s, + "HEALTH_CACHE_TTL_SECONDS": w.health_cache_ttl_s, + "FORECAST_MAX_AGE_SECONDS": w.forecast_max_age_s, + "UMAMI_TRACK_API": w.umami_track_api, + "collector": concat!("ping-collector ", env!("CARGO_PKG_VERSION")), + }); + if let Some(m) = out.as_object_mut() { + for (k, v) in m.iter_mut() { + let k = k.to_lowercase(); + if (k.contains("password") || k.contains("secret")) && !v.is_null() { + *v = json!("********"); + } + } + } + ok_json(&out) +} diff --git a/rust/collector/src/web/tests.rs b/rust/collector/src/web/tests.rs new file mode 100644 index 0000000..d635ac8 --- /dev/null +++ b/rust/collector/src/web/tests.rs @@ -0,0 +1,293 @@ +//! Route tests: the router as the server runs it, against a throwaway +//! schema (TEST_DATABASE_URL; see db_tests.rs) or, for the routes that do +//! not touch the database, a pool that never connects. Output formats are +//! the FastAPI app's (scripts/api_parity.py compares the two live). + +use super::{AppState, router}; +use crate::config::Config; +use crate::db_tests::TestDb; +use crate::stations::Stations; +use axum::Router; +use axum::body::Body; +use axum::http::{HeaderMap, Request, StatusCode}; +use serde_json::{Value, json}; +use std::collections::HashMap; +use std::sync::{Arc, RwLock}; +use tower::ServiceExt; + +fn config(extra: &[(&str, &str)]) -> Config { + let static_dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../../src/static"); + let mut vars: HashMap = [ + ( + "POSTGRES_CONNECTION_STRING", + "postgres://test@localhost:1/none", + ), + ("UMAMI_TRACK_API", "false"), + ("STATIC_DIR", static_dir), + ("NTFY_SERVER", "https://ntfy.example"), + ("STATION_CONFIG_PATH", "/nonexistent/stations.json"), + ] + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + for (k, v) in extra { + vars.insert(k.to_string(), v.to_string()); + } + Config::from_vars(&|k| vars.get(k).cloned()).unwrap() +} + +fn app(cfg: Config, pool: sqlx::PgPool) -> Router { + let state = AppState::new( + cfg, + pool, + reqwest::Client::new(), + Arc::new(RwLock::new(Stations::bundled())), + None, + None, + ); + router(state) +} + +async fn call( + app: &Router, + method: &str, + uri: &str, + headers: &[(&str, &str)], +) -> (StatusCode, HeaderMap, Vec) { + let mut req = Request::builder().method(method).uri(uri); + for (k, v) in headers { + req = req.header(*k, *v); + } + let resp = app + .clone() + .oneshot(req.body(Body::empty()).unwrap()) + .await + .unwrap(); + let (parts, body) = resp.into_parts(); + let bytes = axum::body::to_bytes(body, usize::MAX).await.unwrap(); + (parts.status, parts.headers, bytes.to_vec()) +} + +async fn get_json(app: &Router, uri: &str) -> (StatusCode, HeaderMap, Value) { + let (s, h, b) = call(app, "GET", uri, &[]).await; + let v = serde_json::from_slice(&b) + .unwrap_or_else(|e| panic!("{uri}: {e}: {}", String::from_utf8_lossy(&b))); + (s, h, v) +} + +fn header<'a>(h: &'a HeaderMap, name: &str) -> Option<&'a str> { + h.get(name).and_then(|v| v.to_str().ok()) +} + +#[tokio::test] +async fn routes_without_database() { + let pool = sqlx::PgPool::connect_lazy("postgres://test@localhost:1/none").unwrap(); + let app = app(config(&[]), pool); + + let (s, h, stations) = get_json(&app, "/stations").await; + assert_eq!(s, StatusCode::OK); + assert_eq!(header(&h, "cache-control"), Some("public, max-age=300")); + assert_eq!(header(&h, "content-type"), Some("application/json")); + assert_eq!(stations.as_array().unwrap().len(), 16); + // the list leaves geohash out, the single station has it (as FastAPI) + assert_eq!(stations[7]["station_code"], "P.1"); + assert_eq!(stations[7]["geohash"], Value::Null); + let (_, _, p1) = get_json(&app, "/stations/8").await; + assert_eq!( + p1.as_object().unwrap().keys().collect::>(), + [ + "station_id", + "station_code", + "thai_name", + "english_name", + "latitude", + "longitude", + "geohash", + "status" + ] + ); + + let (s, _, e) = get_json(&app, "/stations/abc").await; + assert_eq!(s, StatusCode::UNPROCESSABLE_ENTITY); + assert_eq!(e["detail"][0]["type"], "int_parsing"); + assert_eq!(e["detail"][0]["loc"], json!(["path", "station_id"])); + assert_eq!( + get_json(&app, "/stations/99").await.2, + json!({"detail": "Station not found"}) + ); + assert_eq!( + get_json(&app, "/nope").await.2, + json!({"detail": "Not Found"}) + ); + let (s, _, b) = call(&app, "DELETE", "/forecast", &[]).await; + assert_eq!(s, StatusCode::METHOD_NOT_ALLOWED); + assert_eq!(b, br#"{"detail":"Method Not Allowed"}"#); + + let (s, _, e) = get_json(&app, "/config").await; + assert_eq!(s, StatusCode::SERVICE_UNAVAILABLE); + assert_eq!( + e["detail"], + "Admin API disabled: ADMIN_API_KEY is not configured on the server" + ); + let (s, _, e) = get_json(&app, "/measurements/history/P.1?hours=0").await; + assert_eq!(s, StatusCode::UNPROCESSABLE_ENTITY); + assert_eq!(e["detail"][0]["ctx"], json!({"ge": 1})); + + let (_, _, n) = get_json(&app, "/api/notifications").await; + assert_eq!(n["topics"]["warning"], "ping-warning"); + assert_eq!(n["server"], "https://ntfy.example"); + + let (s, h, spec) = get_json(&app, "/openapi.json").await; + assert_eq!(s, StatusCode::OK); + assert_eq!(header(&h, "content-encoding"), None); + assert!(spec["paths"]["/forecast"]["get"].is_object()); + + let (s, h, html) = call(&app, "GET", "/", &[("accept-encoding", "gzip")]).await; + assert_eq!(s, StatusCode::OK); + assert_eq!(header(&h, "cache-control"), Some("public, max-age=120")); + assert_eq!(header(&h, "content-encoding"), Some("gzip")); + assert!(html.len() > 100); + let (s, h, _) = call(&app, "GET", "/docs", &[]).await; + assert_eq!(s, StatusCode::OK); + assert_eq!(header(&h, "content-type"), Some("text/html; charset=utf-8")); + let (s, h, _) = call(&app, "GET", "/static/custom-markers.json", &[]).await; + assert_eq!(s, StatusCode::OK); + assert_eq!(header(&h, "cache-control"), Some("public, max-age=3600")); + let (s, h, _) = call(&app, "GET", "/robots.txt", &[]).await; + assert_eq!(s, StatusCode::OK); + assert_eq!( + header(&h, "content-type"), + Some("text/plain; charset=utf-8") + ); + assert_eq!( + call(&app, "GET", "/static/nope.json", &[]).await.0, + StatusCode::NOT_FOUND + ); + + // admin key + let app = super::router(AppState::new( + config(&[("ADMIN_API_KEY", "k3y")]), + sqlx::PgPool::connect_lazy("postgres://test@localhost:1/none").unwrap(), + reqwest::Client::new(), + Arc::new(RwLock::new(Stations::bundled())), + None, + None, + )); + assert_eq!( + call(&app, "GET", "/config", &[("x-api-key", "nope")]) + .await + .0, + StatusCode::UNAUTHORIZED + ); + let (s, _, b) = call(&app, "GET", "/config", &[("x-api-key", "k3y")]).await; + assert_eq!(s, StatusCode::OK, "{}", String::from_utf8_lossy(&b)); + assert!(String::from_utf8_lossy(&b).contains("postgres://test:***@localhost:1/none")); + let (s, _, b) = call(&app, "POST", "/scrape/trigger", &[("x-api-key", "k3y")]).await; + assert_eq!(s, StatusCode::CONFLICT, "{}", String::from_utf8_lossy(&b)); +} + +async fn seed(pool: &sqlx::PgPool) { + for sql in [ + "INSERT INTO stations (id, station_code, thai_name, english_name) VALUES + (1, 'P.20', 'บ้านเชียงดาว', 'Ban Chiang Dao'), (8, 'P.1', 'สะพานนวรัฐ', 'Nawarat Bridge')", + // P.1 hourly 2026-10-01 00:00 .. 10-03 23:00, one missing discharge, a hole at 10-02 05:00 + "INSERT INTO water_measurements (timestamp, station_id, water_level, discharge, discharge_percent, status) + SELECT t, 8, round((1.2 + extract(hour from t) / 100.0)::numeric, 3), + CASE WHEN t = timestamp '2026-10-02 03:00' THEN NULL ELSE 55.5 END, 12.94, 'active' + FROM generate_series(timestamp '2026-10-01 00:00', timestamp '2026-10-03 23:00', interval '1 hour') t + WHERE t <> timestamp '2026-10-02 05:00'", + "INSERT INTO water_measurements (timestamp, station_id, water_level, discharge, discharge_percent, status) + VALUES (timestamp '2026-10-03 22:00', 1, 0.95, NULL, NULL, 'active')", + "INSERT INTO forecast_history (as_of, station_code, horizon_hours, predicted_max_level, p_warning, + p_danger, current_level, model_version, source) + SELECT t, 'P.1', 6, 1.5, 0.01, 0.0, 1.2, CASE WHEN t < timestamp '2026-10-02 00:00' THEN 'v1' ELSE 'v2' END, 'model' + FROM generate_series(timestamp '2026-10-01 00:00', timestamp '2026-10-03 12:00', interval '1 hour') t", + "INSERT INTO forecast_current (id, computed_at, model_versions, payload) + VALUES ('all', timestamp '2026-10-03 23:10', 'v2', '[{\"station_code\":\"P.1\",\"horizon_hours\":6}]')", + ] { + sqlx::query(sql).execute(pool).await.unwrap(); + } +} + +#[tokio::test] +async fn routes_against_the_database() { + let Some(db) = TestDb::new().await else { + return; + }; + seed(&db.pool).await; + let app = app( + config(&[("FORECAST_MAX_AGE_SECONDS", "60")]), + db.pool.clone(), + ); + + let (s, h, latest) = get_json(&app, "/measurements/latest?limit=5").await; + assert_eq!(s, StatusCode::OK); + assert_eq!(header(&h, "cache-control"), Some("public, max-age=30")); + // ORDER BY station_code: P.1 before P.20 + assert_eq!( + latest, + json!([ + {"timestamp": "2026-10-03T23:00:00", "station_code": "P.1", "station_name_en": "Nawarat Bridge", + "station_name_th": "สะพานนวรัฐ", "water_level": 1.43, "discharge": 55.5, + "discharge_percent": 12.94, "status": "active"}, + {"timestamp": "2026-10-03T22:00:00", "station_code": "P.20", "station_name_en": "Ban Chiang Dao", + "station_name_th": "บ้านเชียงดาว", "water_level": 0.95, "discharge": null, + "discharge_percent": null, "status": "active"} + ]) + ); + + let (_, _, h) = get_json( + &app, + "/measurements/history/P.1?start=2026-10-02&end=2026-10-02&limit=6", + ) + .await; + let rows = h.as_array().unwrap(); + assert_eq!(rows.len(), 6); + assert_eq!(rows[0]["timestamp"], "2026-10-02T00:00:00"); + // discharge missing in the database: 20 * H, rounded + assert_eq!(rows[3]["discharge"], json!(24.6)); + assert_eq!(rows[5]["timestamp"], "2026-10-02T06:00:00"); // the hole at 05:00 + let (s, _, e) = get_json( + &app, + "/measurements/history/P.1?start=2026-10-03&end=2026-10-02", + ) + .await; + assert_eq!(s, StatusCode::BAD_GATEWAY); + assert_eq!(e["detail"], "Measurement history unavailable"); + + let (_, _, stats) = get_json(&app, "/api/stats").await; + assert_eq!(stats["rid_measurements"], 72); + assert_eq!(stats["rid_station_count"], 2); + assert_eq!(stats["first_timestamp"], "2026-10-01T00:00:00"); + assert_eq!(stats["days_spanned"], 3); + assert_eq!(stats["coverage_percent"], json!(98.6)); // 71 of 72 hours + + let (_, _, fh) = get_json(&app, "/api/forecast/history/P.1?start=2026-10-03&horizon=6").await; + assert_eq!(fh.as_array().unwrap().len(), 13); + assert_eq!(fh[0]["predicted_max_level"], json!(1.5)); + + let (s, h, f) = get_json(&app, "/forecast").await; + assert_eq!(s, StatusCode::OK); + assert_eq!(f[0]["station_code"], "P.1"); + // computed long ago: served, marked + assert_eq!(header(&h, "x-data-stale"), Some("true")); + + let (_, _, sk) = get_json(&app, "/api/forecast/skill?station_code=P.1&horizon=6").await; + let versions = sk["versions"].as_array().unwrap(); + assert_eq!(versions.len(), 2); + assert_eq!(versions[0]["model_version"], "v1"); + assert_eq!(versions[0]["n"], 24); + assert_eq!(versions[0]["enough_data"], true); + assert_eq!(sk["current"]["model_version"], "v2"); + assert_eq!(sk["trend"]["previous_version"], "v1"); + + let (_, _, health) = get_json(&app, "/health").await; + assert_eq!(health["checks"]["database"]["status"], "degraded"); // data from 2026-10-03 + assert!( + health["checks"]["database"]["message"] + .as_str() + .unwrap() + .starts_with("Latest data is old: 2026-10-03 23:00:00") + ); + db.drop().await; +} diff --git a/scripts/api_parity.py b/scripts/api_parity.py new file mode 100644 index 0000000..85692b6 --- /dev/null +++ b/scripts/api_parity.py @@ -0,0 +1,389 @@ +#!/usr/bin/env python3 +"""Compare the Rust website/API with the FastAPI app, route by route. + +Phase 3 of docs/RUST_REWRITE.md: before the Rust service takes over port +8000, both answer the same requests against the same database and every +response is compared: status, content type, Cache-Control and the JSON body +(floats exactly; lists of database rows without ORDER BY as sets). Static +files and the dashboard byte for byte. + + # the FastAPI app on :8000, `ping-collector serve --listen 127.0.0.1:8001` + uv run python scripts/api_parity.py \\ + --python http://127.0.0.1:8000 --rust http://127.0.0.1:8001 + +Query parameters vary per run (limits, hours) so the FastAPI app's response +caches (up to 15 min) answer with fresh data, not an older copy. Data the +collector writes during the run can still make a route differ once: a +mismatch is retried after a pause before it counts. + +Intended differences (the Rust side may answer differently): + /health a lagging RID report with current ThaiWater telemetry for + the same gauges is healthy (as the notifier, phase 2) + /scraping/status extra fields (collector version, forecast run) + /metrics process counters: compared by shape only + /sensors/thaiwater live upstream: compared, a difference only warns + /api/hii/rainfall/catchment correlation within 1e-9 (numpy uses BLAS) + /api/stats newer counts on the Rust side are accepted (the FastAPI app + caches this one for 5 minutes) + *.geojson content type application/geo+json (FastAPI: octet-stream) + +Exit status 0 when every hard check passed. +""" + +import argparse +import random +import sys +import time +from typing import Any, Callable, List, Optional, Tuple + +import requests + +SOFT = "soft" # a difference warns, does not fail + + +def get(base: str, path: str, method: str = "GET", **kw) -> requests.Response: + last = None + for _ in range(3): + try: + return requests.request(method, base + path, timeout=120, **kw) + except requests.RequestException as e: # connection reset etc. + last = e + time.sleep(1) + raise last + + +def first_diff(a: Any, b: Any, path: str = "$", rel: float = 0.0) -> Optional[str]: + """Where two decoded JSON values differ (None if equal).""" + if isinstance(a, bool) or isinstance(b, bool): + return None if a is b else f"{path}: {a!r} != {b!r}" + if isinstance(a, (int, float)) and isinstance(b, (int, float)): + if a == b: + return None + if rel and abs(a - b) <= rel * max(abs(a), abs(b)): + return None + return f"{path}: {a!r} != {b!r}" + if type(a) is not type(b): + return f"{path}: {type(a).__name__} {a!r} != {type(b).__name__} {b!r}" + if isinstance(a, dict): + if list(a.keys()) != list(b.keys()): + if set(a.keys()) != set(b.keys()): + return f"{path}: keys {sorted(set(a) ^ set(b))} only on one side" + for k in a: + d = first_diff(a[k], b[k], f"{path}.{k}", rel) + if d: + return d + return None + if isinstance(a, list): + if len(a) != len(b): + return f"{path}: {len(a)} items != {len(b)} items" + for i, (x, y) in enumerate(zip(a, b)): + d = first_diff(x, y, f"{path}[{i}]", rel) + if d: + return d + return None + return None if a == b else f"{path}: {a!r} != {b!r}" + + +def media_type(r: requests.Response) -> str: + return r.headers.get("content-type", "").split(";")[0].strip() + + +class Case: + def __init__( + self, + path: str, + method: str = "GET", + compare: Optional[Callable] = None, + sort_key: Optional[str] = None, + soft: bool = False, + body: Any = None, + headers: Optional[dict] = None, + ): + self.path, self.method, self.compare = path, method, compare + self.sort_key, self.soft, self.body = sort_key, soft, body + self.headers = headers or {} + + def run(self, py: str, rs: str) -> Tuple[Optional[str], Optional[str]]: + """(hard difference, warning)""" + kw = {"headers": self.headers} + if self.body is not None: + kw["json"] = self.body + a = get(py, self.path, self.method, **kw) + b = get(rs, self.path, self.method, **kw) + if a.status_code != b.status_code: + return self._fail( + f"status {a.status_code} != {b.status_code} " + f"({a.text[:120]!r} / {b.text[:120]!r})" + ) + if media_type(a) != media_type(b): + warn = f"content-type {media_type(a)!r} != {media_type(b)!r}" + if media_type(a).startswith("application/json") or media_type(b).startswith( + "application/json" + ): + return self._fail(warn) + return None, warn + if self.method == "GET" and a.status_code == 200: + ca, cb = a.headers.get("cache-control"), b.headers.get("cache-control") + if ca != cb: + return self._fail(f"Cache-Control {ca!r} != {cb!r}") + if self.compare is not None: + d = self.compare(a, b) + elif media_type(a) == "application/json": + ja, jb = a.json(), b.json() + if a.status_code == 422: + # pydantic's messages are approximated; type and location must agree + ja = [(e["type"], e["loc"]) for e in ja["detail"]] + jb = [(e["type"], e["loc"]) for e in jb["detail"]] + if any(t.startswith("date") for t, _ in ja): + ja, jb = [loc for _, loc in ja], [loc for _, loc in jb] + elif self.sort_key and isinstance(ja, list): + ja = sorted(ja, key=lambda r: str(r.get(self.sort_key))) + jb = sorted(jb, key=lambda r: str(r.get(self.sort_key))) + d = first_diff(ja, jb) + else: + d = ( + None + if a.content == b.content + else f"bodies differ ({len(a.content)} / {len(b.content)} bytes)" + ) + return self._fail(d) if d else (None, None) + + def _fail(self, d: str) -> Tuple[Optional[str], Optional[str]]: + return (None, d) if self.soft else (d, None) + + +def compare_health(a, b) -> Optional[str]: + ja, jb = a.json(), b.json() + if set(ja["checks"]) != set(jb["checks"]): + return f"checks {sorted(ja['checks'])} != {sorted(jb['checks'])}" + for name, ca in ja["checks"].items(): + cb = jb["checks"][name] + if ca["status"] == cb["status"]: + continue + # the intended change: RID lagging, ThaiWater current + if ( + name == "database" + and ca["status"] == "degraded" + and cb["status"] == "healthy" + and "ThaiWater telemetry current" in cb["message"] + ): + continue + if name in ("api", "memory"): + continue # probes of the RID API / own process, compared loosely + return f"{name}: {ca} != {cb}" + return None + + +def compare_metrics(a, b) -> Optional[str]: + ja, jb = a.json(), b.json() + return None if set(ja) == set(jb) else f"keys {sorted(ja)} != {sorted(jb)}" + + +def compare_status(a, b) -> Optional[str]: + ja, jb = a.json(), b.json() + for k, v in ja.items(): + if jb.get(k) != v: + return f"{k}: {v!r} != {jb.get(k)!r}" + return None + + +def compare_skill(a, b) -> Optional[str]: + ja, jb = a.json(), b.json() + va, vb = ja.pop("verified_until"), jb.pop("verified_until") + gap = abs( + time.mktime(time.strptime(va[:19], "%Y-%m-%dT%H:%M:%S")) + - time.mktime(time.strptime(vb[:19], "%Y-%m-%dT%H:%M:%S")) + ) + if gap > 300: + return f"verified_until {va} != {vb}" + return first_diff(ja, jb) + + +def compare_stats(a, b) -> Optional[str]: + """Exact, except that the FastAPI app may answer from its 5-minute cache + while the collector has written newer rows since.""" + ja, jb = a.json(), b.json() + if ja == jb: + return None + for k, v in ja.items(): + w = jb.get(k) + if k.endswith("_measurements") or k == "days_spanned": + if not isinstance(w, int) or w < v: + return f"{k}: {v!r} > {w!r}" + elif k == "last_timestamp": + if not isinstance(w, str) or w < v: + return f"{k}: {v!r} > {w!r}" + elif k == "coverage_percent": + if not isinstance(w, (int, float)) or abs(w - v) > 0.2: + return f"{k}: {v!r} != {w!r}" + elif v != w: + return f"{k}: {v!r} != {w!r}" + return None + + +def compare_catchment(a, b) -> Optional[str]: + return first_diff(a.json(), b.json(), rel=1e-9) + + +def compare_openapi(a, b) -> Optional[str]: + ja, jb = a.json(), b.json() + if ja["info"]["title"] != jb["info"]["title"]: + return "title differs" + ops = lambda spec: { # noqa: E731 + (m, p): op for p, item in spec["paths"].items() for m, op in item.items() + } + oa, ob = ops(ja), ops(jb) + if set(oa) != set(ob): + return f"operations only on one side: {sorted(set(oa) ^ set(ob))}" + for k, op in oa.items(): + other = ob[k] + for field in ("operationId", "summary", "parameters", "requestBody"): + d = first_diff(op.get(field), other.get(field), f"{k}.{field}") + if d: + return d + if set(op["responses"]) != set(other["responses"]): + return f"{k}: responses {sorted(op['responses'])} != {sorted(other['responses'])}" + for name in ja["components"]["schemas"]: + if name not in jb["components"]["schemas"]: + return f"schema {name} missing" + return None + + +def compare_html_docs(a, b) -> Optional[str]: + return ( + None + if "swagger-ui" in b.text and "/openapi.json" in b.text + else "no Swagger UI" + ) + + +def cases(admin_key: Optional[str]) -> List[Case]: + r = random.Random(time.time()) + hours = r.randint(150, 190) # fresh keys for the Python app's caches + limit = r.randint(40000, 49999) + feed_hours = r.randint(20, 30) + skill_station = r.choice( + ["P.67", "P.103", "P.20", "P.75", "P.92", "P.21", "P.5", "P.82"] + ) + day = time.strftime("%Y-%m-%d", time.localtime(time.time() - 86400 * 3)) + day2 = time.strftime("%Y-%m-%d", time.localtime(time.time() - 86400)) + out = [ + Case("/"), + Case("/robots.txt"), + Case("/llms.txt"), + Case("/sitemap.xml"), + Case("/static/ping-river-network.geojson", soft=True), + Case("/static/custom-markers.json"), + Case("/static/flood-zones-p1.geojson", soft=True), + Case("/static/flood-2024-all.json"), + Case("/static/nope.json"), + Case("/openapi.json", compare=compare_openapi), + Case("/docs", compare=compare_html_docs), + Case("/health", compare=compare_health), + Case("/metrics", compare=compare_metrics), + Case("/stations"), + Case("/stations/8"), + Case("/stations/1"), + Case("/stations/999"), + Case("/stations/abc"), + Case(f"/measurements/latest?limit={r.randint(200, 499)}"), + Case(f"/measurements/latest?limit={r.randint(1, 15)}"), + Case(f"/measurements/history/P.1?hours={hours}&limit={limit}"), + Case(f"/measurements/history/P.67?hours={hours + 1}"), + Case(f"/measurements/history/P.1?start={day}&end={day2}&limit={limit - 1}"), + Case(f"/measurements/history/P.1?start={day}&limit={r.randint(5, 50)}"), + Case(f"/measurements/history/P.1?start={day2}&end={day}"), + Case(f"/measurements/history/NOPE?hours={hours}"), + Case("/measurements/history/P.1?hours=0"), + Case("/measurements/history/P.1?limit=100001"), + Case("/measurements/history/P.1?start=yesterday"), + Case(f"/measurements/station/P.1?hours={r.randint(20, 48)}"), + Case("/measurements/station/P.103?hours=6&limit=2"), + Case(f"/api/hii/rainfall/latest?hours={feed_hours}", sort_key="station_id"), + Case( + f"/api/hii/waterlevel/latest?hours={feed_hours + 1}", sort_key="station_id" + ), + Case("/api/hii/waterlevel/latest?hours=500"), + Case( + f"/api/hii/rainfall/catchment?days={r.randint(15, 30)}", + compare=compare_catchment, + ), + Case("/sensors/thaiwater", sort_key="id", soft=True), + Case("/forecast"), + Case(f"/api/forecast/history/P.1?hours={hours}&horizon=24"), + Case(f"/api/forecast/history/P.103?hours={feed_hours}"), + Case(f"/api/forecast/history/P.1?start={day}&end={day2}&horizon=6"), + Case("/api/forecast/history/P.1?horizon=99"), + Case("/api/notifications"), + # a station/horizon pair the dashboard does not ask for: not in the + # FastAPI app's 15-minute cache; the dashboard's own one may be + Case( + f"/api/forecast/skill?station_code={skill_station}" + f"&horizon={r.randint(1, 48)}", + compare=compare_skill, + ), + Case( + "/api/forecast/skill?station_code=P.1&horizon=24", + compare=compare_skill, + soft=True, + ), + Case("/api/forecast/skill?horizon=0"), + Case("/api/stats", compare=compare_stats), + Case("/scraping/status", compare=compare_status), + Case("/nope"), + Case("/forecast", method="DELETE"), + Case("/config"), + Case("/scrape/trigger", method="POST"), + Case("/stations", method="POST", body={"station_code": "X"}), + Case("/stations/8", method="DELETE"), + ] + if admin_key: + key = {"X-API-Key": admin_key} + out += [ + Case("/config", headers={"X-API-Key": "wrong"}), + Case("/stations", method="POST", body={"thai_name": 1}, headers=key), + Case("/stations/999", method="PUT", body={}, headers=key), + ] + return out + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--python", default="http://127.0.0.1:8000") + ap.add_argument("--rust", default="http://127.0.0.1:8001") + ap.add_argument( + "--admin-key", help="ADMIN_API_KEY, to compare the admin errors too" + ) + ap.add_argument("--retries", type=int, default=2) + ap.add_argument("--pause", type=float, default=30.0, help="seconds before a retry") + args = ap.parse_args() + + failed, warned, passed = [], [], 0 + for case in cases(args.admin_key): + label = f"{case.method} {case.path}" + for attempt in range(args.retries + 1): + try: + hard, warn = case.run(args.python, args.rust) + except Exception as e: # noqa: BLE001 - report and go on + hard, warn = f"{type(e).__name__}: {e}", None + if not hard or attempt == args.retries: + break + print(f" retry {label}: {hard}", flush=True) + time.sleep(args.pause) + if hard: + failed.append((label, hard)) + print(f"FAIL {label}: {hard}", flush=True) + elif warn: + warned.append((label, warn)) + print(f"warn {label}: {warn}", flush=True) + else: + passed += 1 + print( + f"{passed} identical, {len(warned)} warnings, {len(failed)} different " + f"({args.python} vs {args.rust})" + ) + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/install_collector.sh b/scripts/install_collector.sh index 0c7bcf4..677f3f2 100755 --- a/scripts/install_collector.sh +++ b/scripts/install_collector.sh @@ -1,18 +1,29 @@ #!/usr/bin/env bash -# Install the Rust collector (rust/collector, `ping-collector`) and move work +# Install the Rust service (rust/collector, `ping-collector`) and move work # from the Python app to it, or back. # # sudo bash scripts/install_collector.sh # install binary + unit; checks only # sudo bash scripts/install_collector.sh --cutover # Rust collects (COLLECTION_ENABLED=false) -# sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and sends the -# # ntfy notifications (FORECAST_IN_COLLECTOR=true) +# sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies: +# # ntfy + Matrix (FORECAST_IN_COLLECTOR=true) +# sudo bash scripts/install_collector.sh --check-api # compare the Rust website/API with the +# # Python app's, change nothing +# sudo bash scripts/install_collector.sh --cutover-api # Rust serves the website and API on +# # port 8000 (WEB_LISTEN); the Python app stops +# sudo bash scripts/install_collector.sh --rollback-api # the Python app serves the website again # sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + notifications back to Python # sudo bash scripts/install_collector.sh --rollback # Python does everything again # # Every install also exports the trained models for the Rust forecaster and # compares its forecast with Python's on the live data # (scripts/forecast_parity.py); --cutover-forecast refuses to switch when they -# differ (FORCE=1 overrides). +# differ. --cutover-api first runs `ping-collector serve` next to the Python +# app and compares every route (scripts/api_parity.py); it refuses to switch +# when they differ. FORCE=1 overrides either check. +# +# Once ping-collector sends the Matrix messages (FORECAST_IN_COLLECTOR=true +# and MATRIX_* set), the old `run.py --alert-check` cron job is commented out +# (backup next to the app); it repeated an alert on every run. # # The binary comes from BIN=/path/to/ping-collector (e.g. the "ping-collector- # linux-x86_64" CI artifact) or from rust/target/release/ping-collector, built @@ -28,6 +39,8 @@ ENV_FILE="${APP_DIR}/.env" STATUS_FILE="${APP_DIR}/collector-status.json" PYTHON="${APP_DIR}/.venv/bin/python" REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +WEB_ADDR="${WEB_ADDR:-0.0.0.0:8000}" # where the Python app listens today +CHECK_PORT="${CHECK_PORT:-8001}" # the Rust side during --check-api MODE="${1:-install}" log() { printf '\033[1;32m==>\033[0m %s\n' "$*"; } @@ -64,24 +77,116 @@ show() { journalctl -u ping-collector -n 25 --no-pager || true } +env_is() { grep -q "^$1=$2\s*$" "$ENV_FILE"; } + +wait_http() { # url seconds + local i + for i in $(seq "$2"); do + curl -fsS -o /dev/null --max-time 5 "$1" && return 0 + sleep 1 + done + return 1 +} + +# The Matrix messages come from ping-collector once it notifies; the old +# cron job (`run.py --alert-check`, every 15 min, repeating while a level +# stays high) is commented out, with a backup of the crontab. +retire_matrix_cron() { + env_is FORECAST_IN_COLLECTOR true || return 0 + grep -q '^MATRIX_ACCESS_TOKEN=..*' "$ENV_FILE" || return 0 + local user backup + for user in root "$SERVICE_USER"; do + crontab -l -u "$user" 2>/dev/null | grep -v '^\s*#' | grep -q -- '--alert-check' || continue + backup="${APP_DIR}/crontab-${user}.$(date +%Y%m%d%H%M%S).bak" + crontab -l -u "$user" >"$backup" + sed '/--alert-check/ s|^\([^#]\)|# retired: Matrix is sent by ping-collector # \1|' "$backup" | + crontab -u "$user" - + log "Matrix: commented out the old --alert-check cron job of $user (backup: $backup)" + done + if grep -rlqs -- '--alert-check' /etc/cron.d /etc/crontab 2>/dev/null; then + warn "an --alert-check job is still in $(grep -rls -- '--alert-check' /etc/cron.d /etc/crontab | tr '\n' ' '): remove it, ping-collector sends Matrix now" + fi + if systemctl list-timers --all --no-pager 2>/dev/null | grep -qi alert; then + warn "a systemd timer mentions 'alert' (systemctl list-timers --all): if it runs --alert-check, disable it" + fi +} + +# ping-collector serve next to the Python app; every route compared. +api_check() { + local unit=ping-api-check ok=0 + systemctl stop "$unit" 2>/dev/null || true + systemd-run --quiet --unit="$unit" --collect \ + -p User="$SERVICE_USER" -p Group="$(id -gn "$SERVICE_USER")" \ + -p WorkingDirectory="$APP_DIR" -p EnvironmentFile="$ENV_FILE" \ + "${APP_DIR}/bin/ping-collector" serve --listen "127.0.0.1:${CHECK_PORT}" --no-analytics + if wait_http "http://127.0.0.1:${CHECK_PORT}/api/notifications" 30; then + as_service "$PYTHON" scripts/api_parity.py \ + --python "http://127.0.0.1:${WEB_ADDR##*:}" --rust "http://127.0.0.1:${CHECK_PORT}" && ok=1 + else + warn "ping-collector serve did not answer on port ${CHECK_PORT}:" + journalctl -u "$unit" -n 20 --no-pager || true + fi + systemctl stop "$unit" 2>/dev/null || true + [ "$ok" = 1 ] +} + +rollback_api() { + if env_is WEB_LISTEN '..*'; then + log "The Python app serves the website and API again" + set_env WEB_LISTEN "" + systemctl restart ping-collector.service + fi + systemctl enable water-monitor.service + systemctl restart water-monitor.service +} + +matrix_backups() { + ls "${APP_DIR}"/crontab-*.bak 2>/dev/null | sed 's/^/ /' || true +} + case "$MODE" in --rollback) - log "Stopping ping-collector; the Python app collects, forecasts and notifies again" + log "Stopping ping-collector; the Python app collects, serves, forecasts and notifies again" systemctl disable --now ping-collector.service 2>/dev/null || true + set_env WEB_LISTEN "" set_env FORECAST_IN_COLLECTOR false set_env COLLECTION_ENABLED true + systemctl enable water-monitor.service systemctl restart water-monitor.service + if [ -n "$(matrix_backups)" ]; then + warn "Matrix alerts came from ping-collector; to restore the old cron job: crontab -u USER BACKUP" + matrix_backups + fi log "Done. Check: journalctl -u water-monitor -n 50" exit 0 ;; + --rollback-api) + rollback_api + log "Done. Check: journalctl -u water-monitor -n 50; curl -s http://127.0.0.1:${WEB_ADDR##*:}/health" + exit 0 + ;; --rollback-forecast) + # the Python app forecasts in its own process: it must serve again + rollback_api log "Forecasts and notifications back to the Python app; ping-collector keeps collecting" set_env FORECAST_IN_COLLECTOR false systemctl restart ping-collector.service water-monitor.service + if [ -n "$(matrix_backups)" ]; then + warn "Matrix alerts stop with this (the Python app sends ntfy only); old cron job: crontab -u USER BACKUP" + matrix_backups + fi log "Done. Check: journalctl -u water-monitor -n 50" exit 0 ;; - install | --cutover | --cutover-forecast) ;; + --check-api) + [ -x "${APP_DIR}/bin/ping-collector" ] || die "install first" + if env_is WEB_LISTEN '..*'; then + die "ping-collector already serves the website (WEB_LISTEN is set)" + fi + api_check && log "The Rust API answers as the Python app" || die "the API check found differences" + exit 0 + ;; + install | --cutover | --cutover-forecast | --cutover-api) ;; *) die "unknown option: $MODE" ;; esac @@ -130,12 +235,47 @@ case "$MODE" in fi log "Cutover: ping-collector also forecasts and notifies; the Python app serves its forecast" set_env FORECAST_IN_COLLECTOR true + retire_matrix_cron systemctl enable ping-collector.service systemctl restart ping-collector.service water-monitor.service show 30 - log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback-forecast" + log "Next step: --cutover-api. Rollback: sudo bash scripts/install_collector.sh --rollback-forecast" + ;; + --cutover-api) + env_is FORECAST_IN_COLLECTOR true || die "ping-collector must forecast first: run --cutover-forecast" + if env_is WEB_LISTEN '..*'; then + log "ping-collector already serves the website; restarting it on the new binary" + retire_matrix_cron + systemctl restart ping-collector.service + show 10 + exit 0 + fi + api_ok=1 + if [ "$(systemctl is-active water-monitor.service 2>/dev/null)" = active ]; then + log "API check: ping-collector serve on :${CHECK_PORT} next to the Python app, every route compared" + api_check || api_ok=0 + else + warn "the Python app is not running: nothing to compare the API with" + api_ok=0 + fi + if [ "$api_ok" != 1 ] && [ "${FORCE:-0}" != 1 ]; then + die "the API check did not pass; not switching (FORCE=1 overrides)" + fi + log "Cutover: ping-collector serves the website and API on ${WEB_ADDR}; the Python app stops" + set_env WEB_LISTEN "$WEB_ADDR" + retire_matrix_cron + systemctl disable --now water-monitor.service + systemctl restart ping-collector.service + if wait_http "http://127.0.0.1:${WEB_ADDR##*:}/health" 60; then + log "The website answers from ping-collector: $(curl -s --max-time 5 "http://127.0.0.1:${WEB_ADDR##*:}/scraping/status" | grep -o '"collector":"[^"]*"')" + else + warn "no answer on port ${WEB_ADDR##*:} yet; check journalctl -u ping-collector" + fi + show 5 + log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback-api" ;; *) + retire_matrix_cron if [ "$(systemctl is-active ping-collector.service 2>/dev/null)" = active ]; then systemctl restart ping-collector.service log "Installed; ping-collector restarted on the new binary" diff --git a/scripts/ping-collector.service b/scripts/ping-collector.service index fe70eeb..97948eb 100644 --- a/scripts/ping-collector.service +++ b/scripts/ping-collector.service @@ -1,5 +1,5 @@ [Unit] -Description=Ping River Monitor collector (Rust: RID, HII, dams, Open-Meteo; forecast + ntfy) +Description=Ping River Monitor (Rust: collection, forecast, ntfy/Matrix, website/API) Documentation=https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/src/branch/master/rust After=network-online.target Wants=network-online.target @@ -27,7 +27,8 @@ ProtectSystem=strict ProtectHome=true ReadWritePaths=/opt/thailand-water-monitor CapabilityBoundingSet= -MemoryMax=256M +# collection + 15 models ~100 MB, response caches up to ~100 MB more +MemoryMax=512M StandardOutput=journal StandardError=journal