B4L/Northern-Thailand-Ping-River-Monitor
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions
CI / Format & lint (push) Successful in 26s
Details
CI / Test suite (push) Successful in 1m9s
Details
Docs / Validate documentation (push) Successful in 25s
Details
Rust / fmt, clippy, test, build (push) Successful in 11m46s
Details

Browse Source 
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.

Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).

scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.

Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).

install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
This commit is contained in:
grabowski committed 2026-10-18 16:08:06 +07:00
1 parent 1d1c000922
commit 779728e38b
32 files changed
+5692 -160
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

rust/Cargo.lock
Generated
+167 -1
View File
@@ -88,6 +88,18 @@ version = "1.0.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]]
name = "async-compression"
version = "0.4.51"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9d5755640120b24961f8231e05892e4a4f7e0d04e8aeca161508c9c397946970"
dependencies = [
"compression-codecs",
"compression-core",
"pin-project-lite",
"tokio",
]
[[package]]
name = "atoi"
version = "2.0.0"
@@ -109,6 +121,56 @@ version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "axum"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90"
dependencies = [
"axum-core",
"bytes",
"form_urlencoded",
"futures-util",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"itoa",
"matchit",
"memchr",
"mime",
"percent-encoding",
"pin-project-lite",
"serde_core",
"serde_json",
"serde_path_to_error",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tower",
"tower-layer",
"tower-service",
]
[[package]]
name = "axum-core"
version = "0.5.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1"
dependencies = [
"bytes",
"futures-core",
"http",
"http-body",
"http-body-util",
"mime",
"pin-project-lite",
"sync_wrapper",
"tower-layer",
"tower-service",
]
[[package]]
name = "base64"
version = "0.22.1"
@@ -237,6 +299,23 @@ version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "compression-codecs"
version = "0.4.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cae36683101b2899f48bd31f8db9839437356188e3b1ff23096e73dab09ba8fe"
dependencies = [
"compression-core",
"flate2",
"memchr",
]
[[package]]
name = "compression-core"
version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ebdda89edc534bf8d5dd2857562633b6495557e5846552ade994b99ed13c36a"
[[package]]
name = "const-oid"
version = "0.9.6"
@@ -443,6 +522,12 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "foldhash"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
[[package]]
name = "foreign-types"
version = "0.3.2"
@@ -578,7 +663,7 @@ checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
"foldhash 0.1.5",
]
[[package]]
@@ -668,12 +753,24 @@ dependencies = [
"pin-project-lite",
]
[[package]]
name = "http-range-header"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9171a2ea8a68358193d15dd5d70c1c10a2afc3e7e4c5bc92bc9f025cebd7359c"
[[package]]
name = "httparse"
version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
[[package]]
name = "httpdate"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
[[package]]
name = "hyper"
version = "1.12.0"
@@ -686,6 +783,7 @@ dependencies = [
"http",
"http-body",
"httparse",
"httpdate",
"itoa",
"pin-project-lite",
"smallvec",
@@ -979,6 +1077,12 @@ dependencies = [
"regex-automata",
]
[[package]]
name = "matchit"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
[[package]]
name = "md-5"
version = "0.10.6"
@@ -995,6 +1099,22 @@ version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "mime"
version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "mime_guess"
version = "2.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e"
dependencies = [
"mime",
"unicase",
]
[[package]]
name = "miniz_oxide"
version = "0.9.1"
@@ -1197,9 +1317,11 @@ name = "ping-collector"
version = "0.1.0"
dependencies = [
"anyhow",
"axum",
"chrono",
"clap",
"flate2",
"form_urlencoded",
"log",
"ping-forecast",
"reqwest",
@@ -1207,6 +1329,8 @@ dependencies = [
"serde_json",
"sqlx",
"tokio",
"tower",
"tower-http",
"tracing",
"tracing-subscriber",
]
@@ -1566,6 +1690,8 @@ version = "1.0.154"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
dependencies = [
"foldhash 0.2.0",
"indexmap",
"itoa",
"memchr",
"serde",
@@ -1573,6 +1699,17 @@ dependencies = [
"zmij",
]
[[package]]
name = "serde_path_to_error"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457"
dependencies = [
"itoa",
"serde",
"serde_core",
]
[[package]]
name = "serde_urlencoded"
version = "0.7.1"
@@ -2063,6 +2200,19 @@ dependencies = [
"tokio",
]
[[package]]
name = "tokio-util"
version = "0.7.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b"
dependencies = [
"bytes",
"futures-core",
"futures-sink",
"pin-project-lite",
"tokio",
]
[[package]]
name = "tower"
version = "0.5.3"
@@ -2084,12 +2234,22 @@ version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
dependencies = [
"async-compression",
"bitflags",
"bytes",
"futures-core",
"futures-util",
"http",
"http-body",
"http-body-util",
"http-range-header",
"httpdate",
"mime",
"mime_guess",
"percent-encoding",
"pin-project-lite",
"tokio",
"tokio-util",
"tower",
"tower-layer",
"tower-service",
@@ -2182,6 +2342,12 @@ version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "unicase"
version = "2.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "357cc3acc6a036009fd6c973ed009037c732d60d0b4f6c673e9041497482a28f"
[[package]]
name = "unicode-bidi"
version = "0.3.18"
rust/Cargo.toml
+8 -2
View File
@@ -13,17 +13,23 @@ repository = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor"
[workspace.dependencies]
anyhow = "1"
axum = { version = "0.8", default-features = false, features = ["http1", "tokio", "query", "json", "matched-path", "original-uri"] }
chrono = { version = "0.4", default-features = false, features = ["std", "clock", "serde"] }
clap = { version = "4", features = ["derive"] }
flate2 = "1"
form_urlencoded = "1"
log = "0.4"
# native-tls (OpenSSL on Linux, SChannel on Windows), not rustls: app.rid.go.th (dams)
# offers only TLS 1.2 CBC suites (ECDHE-RSA-AES256-SHA384), which rustls refuses.
reqwest = { version = "0.12", default-features = false, features = ["native-tls", "json"] }
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1", features = ["float_roundtrip"] }
# preserve_order: JSON objects keep the order they were built in (API
# responses list their fields in the same order as the FastAPI app did)
serde_json = { version = "1", features = ["float_roundtrip", "preserve_order"] }
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "chrono", "macros"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "signal"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "signal", "net", "sync", "fs"] }
tower = { version = "0.5", default-features = false, features = ["util"] }
tower-http = { version = "0.6", features = ["compression-gzip", "cors", "fs"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
rust/README.md
+40 -2
View File
@@ -7,6 +7,7 @@ costs and the parity rules are in [docs/RUST_REWRITE.md](../docs/RUST_REWRITE.md
| --- | --- | --- |
| `collector` (`ping-collector`) | RID, HII/ThaiWater, RID dams and Open-Meteo collection in the Python web app | phase 1: in production since 2026-10-17 |
| `forecast` (`ping-forecast`, library) | the flood forecast (`src/ml` serving path); run by `ping-collector` with ntfy | phase 2: in production since 2026-10-18 |
| `collector`: `src/web` | the FastAPI app (`src/web_api.py`): dashboard, read API, `/docs` | phase 3: built, cutover pending |
## ping-collector
@@ -19,7 +20,10 @@ the database advances.
```text
ping-collector run # the service: startup gap fill, RID at :05 (+10 min retries
# while RID is behind), HII/dams/rain once per clock hour;
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy
# with FORECAST_IN_COLLECTOR=true also forecast + ntfy/Matrix;
# with WEB_LISTEN the website and API
ping-collector serve [--listen A] [--no-analytics]
# the website and API only (default 127.0.0.1:8001)
ping-collector once [--dry-run] # one cycle of every source; --dry-run writes nothing
# and compares the RID fetch with the database
ping-collector fill-gaps <days|all> # re-fetch RID reports with holes
@@ -39,7 +43,7 @@ only its line number, never its value. Variables already set (e.g. by systemd's
`GAP_FILL_DAYS_ON_START` (14), `ENABLE_HII_COLLECTION`,
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION`, `HII_BASIN_CODE` (6),
`STATION_CONFIG_PATH` (falls back to the bundled `src/data/stations.json`),
`SCRAPING_STATUS_FILE` (status JSON served by the API's `/scraping/status`),
`SCRAPING_STATUS_FILE` (status JSON for `/scraping/status` when the website runs elsewhere),
`LOG_LEVEL` / `RUST_LOG`; for phase 2 `FORECAST_IN_COLLECTOR` (false),
`MODELS_DIR` (`models`), `NTFY_SERVER` / `NTFY_PUBLISH_URL` / `NTFY_TOPIC_PREFIX`
/ `NTFY_TOKEN` / `PUBLIC_URL` (as the Python app), `RID_STALE_ALERT_HOURS` (18,
@@ -105,6 +109,37 @@ Verified:
`src/notify.py` (thresholds, capacity guard, outlook, feed gaps, ntfy
outages); the Rust notifier sends the same 751 messages word for word.
## Website and API (phase 3)
`src/web` serves what the FastAPI app served: the dashboard
(`STATIC_DIR`/dashboard.html, default `src/static`, loaded at start), the
crawler files, `/static`, `/docs` + `/redoc` + `/openapi.json`, and every API
route with the same parameters, JSON (field order included), status codes,
`{"detail": ...}` errors, Cache-Control, gzip, CORS (`CORS_ALLOW_ORIGINS`),
Umami API tracking (`UMAMI_*`) and response caches (`*_CACHE_TTL_SECONDS`).
Admin routes need `ADMIN_API_KEY`. With `WEB_LISTEN=0.0.0.0:8000` it runs
inside `ping-collector run` (one process collects, forecasts, notifies and
serves; `/scraping/status` and `POST /scrape/trigger` then talk to the
collection loop directly); `serve` runs it alone.
- Parameters are declared once per route (`web/params.rs`): the same list
validates the request (pydantic-style 422s) and builds the OpenAPI
document (`web/openapi.rs`), so `/docs` cannot drift from what is accepted.
The component schemas are FastAPI's (`web/openapi_components.json`).
- Python's output conventions are kept where they show: `isoformat()`
timestamps, `round()`, numpy's pairwise sums for the catchment means
(`web/pyfmt.rs`), the skill metrics' summation order.
- `scripts/api_parity.py --python URL --rust URL` compares the two live on
one database; `install_collector.sh --check-api` / `--cutover-api` run it
on the server against `ping-collector serve` on port 8001.
Matrix (`MATRIX_HOMESERVER`, `MATRIX_ACCESS_TOKEN`, `MATRIX_ROOM_ID`): the
notifier sends the room the same transitions as ntfy (per station, outlook,
feed; not ntfy's per-topic summaries), each once, with its own `matrix:` keys
in `notification_state` (seeded from the ntfy ones on first use, so nothing
already announced is repeated). It replaces the `run.py --alert-check` cron
job, which the install script comments out.
### Develop
```bash
@@ -142,6 +177,9 @@ cargo build --release --locked --manifest-path rust/Cargo.toml # or BIN=<CI ar
sudo bash scripts/install_collector.sh # install + checks, nothing switched
sudo bash scripts/install_collector.sh --cutover # Rust collects from now on
sudo bash scripts/install_collector.sh --cutover-forecast # Rust also forecasts and notifies
sudo bash scripts/install_collector.sh --check-api # compare the Rust API with FastAPI
sudo bash scripts/install_collector.sh --cutover-api # Rust serves the website on :8000
sudo bash scripts/install_collector.sh --rollback-api # FastAPI serves the website again
sudo bash scripts/install_collector.sh --rollback-forecast # forecasts + ntfy back to Python
sudo bash scripts/install_collector.sh --rollback # Python does everything again
journalctl -u ping-collector -f
rust/collector/Cargo.toml
+5 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "ping-collector"
version = "0.1.0"
description = "Ping River Monitor data collector: RID gauges, HII/ThaiWater, RID dams, Open-Meteo rain -> PostgreSQL"
description = "Ping River Monitor service: collection (RID, HII/ThaiWater, RID dams, Open-Meteo), forecast, notifications and the website/API"
edition.workspace = true
rust-version.workspace = true
license.workspace = true
@@ -10,8 +10,10 @@ repository.workspace = true
[dependencies]
anyhow.workspace = true
axum.workspace = true
chrono.workspace = true
clap.workspace = true
form_urlencoded.workspace = true
log.workspace = true
ping-forecast = { path = "../forecast" }
reqwest.workspace = true
@@ -19,6 +21,8 @@ serde.workspace = true
serde_json.workspace = true
sqlx.workspace = true
tokio.workspace = true
tower.workspace = true
tower-http.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
rust/collector/src/config.rs
+144 -44
View File
@@ -48,28 +48,74 @@ pub struct Config {
pub public_url: String,
/// Report RID lagging this long while ThaiWater is current (0 = never).
pub rid_stale_alert_hours: f64,
/// Public ntfy URL (what subscribers use), shown by /api/notifications.
pub ntfy_server: Option<String>,
/// MATRIX_*: the notifications also go to this room (off when unset).
pub matrix: Option<MatrixConfig>,
pub web: WebConfig,
}
fn var(name: &str) -> Option<String> {
env::var(name)
.ok()
.map(|v| v.trim().to_string())
.filter(|v| !v.is_empty())
#[derive(Clone, Debug)]
pub struct MatrixConfig {
pub homeserver: String,
pub access_token: String,
pub room_id: String,
}
fn flag(name: &str, default: bool) -> bool {
var(name).map_or(default, |v| {
matches!(v.to_lowercase().as_str(), "1" | "true" | "yes")
})
/// The website and read API (src/web_api.py's settings, same names).
#[derive(Clone, Debug)]
pub struct WebConfig {
/// WEB_LISTEN: `run` also serves the website on this address
/// (e.g. 0.0.0.0:8000); unset = collection only.
pub listen: Option<String>,
/// Dashboard, robots/llms/sitemap and /static files.
pub static_dir: PathBuf,
/// X-API-Key for the station editing, /config and /scrape/trigger;
/// unset = those endpoints are disabled.
pub admin_api_key: Option<String>,
pub cors_allow_origins: Vec<String>,
pub umami_api_url: String,
/// Empty = no server-side analytics.
pub umami_website_id: String,
pub umami_track_api: bool,
pub hii_cache_ttl_s: u64,
pub latest_cache_ttl_s: u64,
pub health_cache_ttl_s: u64,
/// A stored forecast older than this is served marked `X-Data-Stale`.
pub forecast_max_age_s: u64,
pub thaiwater_api_key: Option<String>,
pub request_timeout_s: u64,
}
fn number<T: std::str::FromStr>(name: &str, default: T) -> Result<T>
where
T::Err: std::fmt::Display,
{
match var(name) {
None => Ok(default),
Some(v) => v.parse().map_err(|e| anyhow::anyhow!("{name}={v:?}: {e}")),
/// Variable lookup (the process environment, or a map in tests).
struct Vars<'a>(&'a dyn Fn(&str) -> Option<String>);
impl Vars<'_> {
fn var(&self, name: &str) -> Option<String> {
(self.0)(name)
.map(|v| v.trim().to_string())
.filter(|v| !v.is_empty())
}
/// `os.getenv(name, default)`: a variable set to "" stays "".
fn raw_var(&self, name: &str, default: &str) -> String {
(self.0)(name).map_or_else(|| default.to_string(), |v| v.trim().to_string())
}
fn flag(&self, name: &str, default: bool) -> bool {
self.var(name).map_or(default, |v| {
matches!(v.to_lowercase().as_str(), "1" | "true" | "yes")
})
}
fn number<T: std::str::FromStr>(&self, name: &str, default: T) -> Result<T>
where
T::Err: std::fmt::Display,
{
match self.var(name) {
None => Ok(default),
Some(v) => v.parse().map_err(|e| anyhow::anyhow!("{name}={v:?}: {e}")),
}
}
}
@@ -101,52 +147,106 @@ pub fn normalize_database_url(url: &str) -> String {
impl Config {
pub fn from_env() -> Result<Self> {
let database_url = match var("POSTGRES_CONNECTION_STRING") {
Self::from_vars(&|name| env::var(name).ok())
}
pub fn from_vars(get: &dyn Fn(&str) -> Option<String>) -> Result<Self> {
let v = Vars(get);
let database_url = match v.var("POSTGRES_CONNECTION_STRING") {
Some(url) => normalize_database_url(&url),
None => {
let password = var("POSTGRES_PASSWORD")
let password = v
.var("POSTGRES_PASSWORD")
.context("POSTGRES_CONNECTION_STRING or POSTGRES_PASSWORD must be set")?;
format!(
"postgres://{}:{}@{}:{}/{}",
encode_userinfo(&var("POSTGRES_USER").unwrap_or_else(|| "postgres".into())),
encode_userinfo(&v.var("POSTGRES_USER").unwrap_or_else(|| "postgres".into())),
encode_userinfo(&password),
var("POSTGRES_HOST").unwrap_or_else(|| "localhost".into()),
var("POSTGRES_PORT").unwrap_or_else(|| "5432".into()),
var("POSTGRES_DB").unwrap_or_else(|| "water_monitoring".into()),
v.var("POSTGRES_HOST").unwrap_or_else(|| "localhost".into()),
v.var("POSTGRES_PORT").unwrap_or_else(|| "5432".into()),
v.var("POSTGRES_DB")
.unwrap_or_else(|| "water_monitoring".into()),
)
}
};
if let Some(db_type) = var("DB_TYPE")
if let Some(db_type) = v.var("DB_TYPE")
&& db_type != "postgresql"
{
bail!("ping-collector supports PostgreSQL only (DB_TYPE={db_type})");
}
let cfg = Config {
database_url,
rid_api_url: var("RID_API_URL").unwrap_or_else(|| RID_API_URL.into()),
hii_api_base: var("HII_API_BASE").unwrap_or_else(|| HII_API_BASE.into()),
dams_url: var("RID_DAMS_URL").unwrap_or_else(|| RID_DAMS_URL.into()),
openmeteo_url: var("OPENMETEO_URL").unwrap_or_else(|| OPENMETEO_URL.into()),
hii_enabled: flag("ENABLE_HII_COLLECTION", true),
dams_enabled: flag("ENABLE_RESERVOIR_COLLECTION", true),
rain_enabled: flag("ENABLE_RAIN_COLLECTION", true),
hii_basin_code: number("HII_BASIN_CODE", 6)?,
poll_offset_minutes: number("RID_POLL_OFFSET_MINUTES", 5)?,
retry_minutes: number("RID_RETRY_MINUTES", 10)?,
gap_fill_days_on_start: number("GAP_FILL_DAYS_ON_START", 14)?,
station_config_path: var("STATION_CONFIG_PATH")
rid_api_url: v.var("RID_API_URL").unwrap_or_else(|| RID_API_URL.into()),
hii_api_base: v.var("HII_API_BASE").unwrap_or_else(|| HII_API_BASE.into()),
dams_url: v.var("RID_DAMS_URL").unwrap_or_else(|| RID_DAMS_URL.into()),
openmeteo_url: v
.var("OPENMETEO_URL")
.unwrap_or_else(|| OPENMETEO_URL.into()),
hii_enabled: v.flag("ENABLE_HII_COLLECTION", true),
dams_enabled: v.flag("ENABLE_RESERVOIR_COLLECTION", true),
rain_enabled: v.flag("ENABLE_RAIN_COLLECTION", true),
hii_basin_code: v.number("HII_BASIN_CODE", 6)?,
poll_offset_minutes: v.number("RID_POLL_OFFSET_MINUTES", 5)?,
retry_minutes: v.number("RID_RETRY_MINUTES", 10)?,
gap_fill_days_on_start: v.number("GAP_FILL_DAYS_ON_START", 14)?,
station_config_path: v
.var("STATION_CONFIG_PATH")
.unwrap_or_else(|| "stations.json".into())
.into(),
status_file: var("SCRAPING_STATUS_FILE").map(PathBuf::from),
forecast_enabled: flag("FORECAST_IN_COLLECTOR", false),
models_dir: var("MODELS_DIR").unwrap_or_else(|| "models".into()).into(),
status_file: v.var("SCRAPING_STATUS_FILE").map(PathBuf::from),
forecast_enabled: v.flag("FORECAST_IN_COLLECTOR", false),
models_dir: v
.var("MODELS_DIR")
.unwrap_or_else(|| "models".into())
.into(),
// as src/config.py: off unless NTFY_SERVER is set
ntfy_publish_url: var("NTFY_SERVER")
.map(|server| var("NTFY_PUBLISH_URL").unwrap_or(server)),
ntfy_token: var("NTFY_TOKEN"),
ntfy_topic_prefix: var("NTFY_TOPIC_PREFIX").unwrap_or_else(|| "ping".into()),
public_url: var("PUBLIC_URL").unwrap_or_else(|| "https://water.buildfor.life/".into()),
rid_stale_alert_hours: number("RID_STALE_ALERT_HOURS", 18.0)?,
ntfy_publish_url: v
.var("NTFY_SERVER")
.map(|server| v.var("NTFY_PUBLISH_URL").unwrap_or(server)),
ntfy_token: v.var("NTFY_TOKEN"),
ntfy_topic_prefix: v.var("NTFY_TOPIC_PREFIX").unwrap_or_else(|| "ping".into()),
public_url: v
.var("PUBLIC_URL")
.unwrap_or_else(|| "https://water.buildfor.life/".into()),
rid_stale_alert_hours: v.number("RID_STALE_ALERT_HOURS", 18.0)?,
ntfy_server: v.var("NTFY_SERVER"),
matrix: match (v.var("MATRIX_ACCESS_TOKEN"), v.var("MATRIX_ROOM_ID")) {
(Some(access_token), Some(room_id)) => Some(MatrixConfig {
homeserver: v
.var("MATRIX_HOMESERVER")
.unwrap_or_else(|| "https://matrix.org".into()),
access_token,
room_id,
}),
_ => None,
},
web: WebConfig {
listen: v.var("WEB_LISTEN"),
static_dir: v
.var("STATIC_DIR")
.unwrap_or_else(|| "src/static".into())
.into(),
admin_api_key: v.var("ADMIN_API_KEY"),
cors_allow_origins: v
.raw_var("CORS_ALLOW_ORIGINS", "")
.split(',')
.map(str::trim)
.filter(|o| !o.is_empty())
.map(String::from)
.collect(),
umami_api_url: v
.var("UMAMI_API_URL")
.unwrap_or_else(|| "https://stats.buildfor.life/api/send".into()),
umami_website_id: v
.raw_var("UMAMI_WEBSITE_ID", "00b2be73-8f5f-4400-9029-3be852eb08f7"),
umami_track_api: v.flag("UMAMI_TRACK_API", true),
hii_cache_ttl_s: v.number("HII_CACHE_TTL_SECONDS", 120)?,
latest_cache_ttl_s: v.number("LATEST_CACHE_TTL_SECONDS", 45)?,
health_cache_ttl_s: v.number("HEALTH_CACHE_TTL_SECONDS", 30)?,
forecast_max_age_s: v.number("FORECAST_MAX_AGE_SECONDS", 7200)?,
thaiwater_api_key: v.var("THAIWATER_API_KEY"),
request_timeout_s: v.number("REQUEST_TIMEOUT", 30)?,
},
};
if cfg.poll_offset_minutes >= 60 {
bail!("RID_POLL_OFFSET_MINUTES must be between 0 and 59");
rust/collector/src/db.rs
+2 -2
View File
@@ -14,7 +14,7 @@ use sqlx::PgPool;
use sqlx::postgres::PgPoolOptions;
use std::collections::{BTreeMap, HashMap, HashSet};
pub async fn connect(url: &str) -> Result<PgPool> {
pub async fn connect(url: &str, max_connections: u32) -> Result<PgPool> {
use sqlx::ConnectOptions;
use std::str::FromStr;
// sqlx warns about any statement over 1 s; the whole-history gap scan
@@ -23,7 +23,7 @@ pub async fn connect(url: &str) -> Result<PgPool> {
.context("invalid database URL")?
.log_slow_statements(log::LevelFilter::Warn, std::time::Duration::from_secs(30));
PgPoolOptions::new()
.max_connections(4)
.max_connections(max_connections)
.acquire_timeout(std::time::Duration::from_secs(30))
.connect_with(options)
.await
rust/collector/src/db_tests.rs
+4 -4
View File
@@ -17,14 +17,14 @@ use std::collections::HashMap;
use std::str::FromStr;
use std::sync::Mutex;
struct TestDb {
pool: PgPool,
pub(crate) struct TestDb {
pub(crate) pool: PgPool,
admin: PgPool,
schema: String,
}
impl TestDb {
async fn new() -> Option<Self> {
pub(crate) async fn new() -> Option<Self> {
let Ok(url) = std::env::var("TEST_DATABASE_URL") else {
eprintln!("TEST_DATABASE_URL not set: skipping database test");
return None;
@@ -58,7 +58,7 @@ impl TestDb {
})
}
async fn drop(self) {
pub(crate) async fn drop(self) {
self.pool.close().await;
sqlx::query(&format!("DROP SCHEMA {} CASCADE", self.schema))
.execute(&self.admin)
rust/collector/src/forecasting.rs
+32 -8
View File
@@ -130,12 +130,23 @@ pub fn summarize(s: &Snapshot) -> String {
)
}
/// ntfy (NTFY_SERVER) and/or Matrix (MATRIX_ACCESS_TOKEN + MATRIX_ROOM_ID);
/// `None` when neither is configured.
pub fn notifier(cfg: &Config, http: reqwest::Client) -> Option<Notifier> {
let url = cfg.ntfy_publish_url.clone()?;
let ntfy = cfg
.ntfy_publish_url
.clone()
.map(|publish_url| notify::Ntfy {
publish_url,
token: cfg.ntfy_token.clone(),
});
if ntfy.is_none() && cfg.matrix.is_none() {
return None;
}
Some(Notifier {
http,
publish_url: url,
token: cfg.ntfy_token.clone(),
ntfy,
matrix: cfg.matrix.clone(),
dashboard_url: cfg.public_url.clone(),
settings: Settings {
prefix: cfg.ntfy_topic_prefix.clone(),
@@ -164,14 +175,27 @@ pub struct Forecaster {
impl Forecaster {
pub fn new(cfg: &Config, http: reqwest::Client) -> Self {
let notifier = notifier(cfg, http);
match &notifier {
Some(n) => tracing::info!(
match notifier
.as_ref()
.and_then(|n| n.ntfy.as_ref().map(|t| (n, t)))
{
Some((n, t)) => tracing::info!(
"ntfy notifications: publish to {}, topics {}-*",
n.publish_url,
t.publish_url,
n.settings.prefix
),
None => tracing::info!("ntfy notifications off (NTFY_SERVER not set)"),
}
match &cfg.matrix {
Some(m) => tracing::info!(
"Matrix notifications: room {} on {}",
m.room_id,
m.homeserver
),
None => tracing::info!(
"Matrix notifications off (MATRIX_ACCESS_TOKEN/MATRIX_ROOM_ID not set)"
),
}
Self {
store: ModelStore::new(&cfg.models_dir),
notifier,
@@ -236,7 +260,7 @@ impl Forecaster {
notified = sent.len();
if !sent.is_empty() {
tracing::info!(
"ntfy: published {}",
"notifications: published {}",
sent.iter()
.map(|s| format!("{}: {}", s.topic, s.title))
.collect::<Vec<_>>()
@@ -244,7 +268,7 @@ impl Forecaster {
);
}
}
Err(e) => tracing::warn!("ntfy notify cycle failed: {e:#}"),
Err(e) => tracing::warn!("notify cycle failed: {e:#}"),
}
}
Ok(RunInfo {
rust/collector/src/main.rs
+35 -6
View File
@@ -3,8 +3,9 @@
//! Replaces the collection half of the Python web app (RID hourly gauges,
//! HII/ThaiWater rain + water level, RID dams, Open-Meteo rain) and, with
//! FORECAST_IN_COLLECTOR=true, the flood forecast (rust/forecast) and the
//! ntfy notifications. Writes the same PostgreSQL tables as the Python app,
//! which then runs with `COLLECTION_ENABLED=false`. See docs/RUST_REWRITE.md.
//! ntfy/Matrix notifications, and with WEB_LISTEN the website and read API
//! (the port of the FastAPI app). Writes the same PostgreSQL tables as the
//! Python app did. See docs/RUST_REWRITE.md.
mod config;
mod dams;
@@ -20,6 +21,7 @@ mod rid;
mod service;
mod stations;
mod util;
mod web;
use anyhow::{Result, bail};
use clap::{Parser, Subcommand};
@@ -38,8 +40,20 @@ struct Cli {
#[derive(Subcommand)]
enum Command {
/// Run the collection service (what systemd starts)
/// Run the service (what systemd starts): collection, with
/// FORECAST_IN_COLLECTOR=true forecasts and notifications, with WEB_LISTEN
/// the website and API
Run,
/// Serve only the website and API (no collection), e.g. next to the
/// running service for a comparison
Serve {
/// Address to listen on [default: WEB_LISTEN, else 127.0.0.1:8001]
#[arg(long)]
listen: Option<String>,
/// Do not report requests to Umami (comparison runs)
#[arg(long)]
no_analytics: bool,
},
/// One cycle of every source, then exit
Once {
/// Fetch and compare with the database, write nothing
@@ -132,18 +146,33 @@ fn main() -> Result<()> {
}
async fn run(cli: Cli) -> Result<()> {
let cfg = config::Config::from_env()?;
let mut cfg = config::Config::from_env()?;
if let Command::Serve {
no_analytics: true, ..
} = cli.command
{
cfg.web.umami_track_api = false;
}
tracing::info!(
"ping-collector {} -> {}",
env!("CARGO_PKG_VERSION"),
cfg.redacted_database_url()
);
let pool = db::connect(&cfg.database_url).await?;
// the website answers concurrent requests; collection alone needs few
let serving = matches!(cli.command, Command::Serve { .. })
|| (matches!(cli.command, Command::Run) && cfg.web.listen.is_some());
let pool = db::connect(&cfg.database_url, if serving { 12 } else { 4 }).await?;
let stations = stations::Stations::load(&cfg.station_config_path)?;
let collector = service::Collector::new(cfg, pool, stations)?;
match cli.command {
Command::Run => collector.run().await,
Command::Serve { listen, .. } => {
let listen = listen
.or_else(|| collector.cfg.web.listen.clone())
.unwrap_or_else(|| "127.0.0.1:8001".into());
collector.serve(listen).await
}
Command::Once { dry_run } => collector.once(dry_run).await,
Command::FillGaps { days } => {
let days = match days.as_str() {
@@ -157,7 +186,7 @@ async fn run(cli: Cli) -> Result<()> {
let filled = rid::fill_gaps(
&collector.rid,
&collector.pool,
&collector.stations,
&collector.stations(),
days,
util::bangkok_now(),
)
rust/collector/src/notify.rs
+241 -15
View File
@@ -1,5 +1,8 @@
//! Public flood notifications over ntfy: the port of src/notify.py, run by
//! the collector when FORECAST_IN_COLLECTOR=true.
//! Public flood notifications over ntfy, and the same transitions to a
//! Matrix room: the port of src/notify.py, run by the collector when
//! FORECAST_IN_COLLECTOR=true. (Matrix replaces the old `run.py
//! --alert-check` cron job, which repeated an alert on every run while a
//! level stayed high.)
//!
//! Same topics, texts, hysteresis, capacity guard, outlook and persisted
//! state keys (`notification_state`) as Python, so a cutover neither repeats
@@ -9,6 +12,7 @@
//! "stale" only when neither source has a reading for 3 h; RID lagging alone
//! is reported separately after RID_STALE_ALERT_HOURS.
use crate::config::MatrixConfig;
use crate::db::{self, LatestReading};
use anyhow::Result;
use chrono::{Duration, NaiveDateTime};
@@ -425,22 +429,105 @@ pub fn evaluate(
plan
}
/// Posts to ntfy and records delivered transitions.
pub struct Notifier {
pub http: reqwest::Client,
/// ntfy publishing target.
pub struct Ntfy {
pub publish_url: String,
pub token: Option<String>,
}
/// State keys of the Matrix channel: the same transitions as ntfy, tracked
/// separately so a failed delivery on one channel is retried on that one
/// only.
const MATRIX_PREFIX: &str = "matrix:";
/// Posts to ntfy and Matrix and records delivered transitions.
pub struct Notifier {
pub http: reqwest::Client,
pub ntfy: Option<Ntfy>,
pub matrix: Option<MatrixConfig>,
pub dashboard_url: String,
pub settings: Settings,
}
/// ntfy tag shortcodes the notifications use, as emoji for Matrix.
fn emoji(tags: &[String]) -> &'static str {
match tags.first().map(String::as_str) {
Some("rotating_light") => "🚨",
Some("warning") => "⚠️",
Some("arrow_down") => "⬇️",
Some("white_check_mark") => "✅",
Some("crystal_ball") => "🔮",
Some("hourglass") => "⏳",
_ => "📊",
}
}
/// The Matrix message: the ntfy title and text, in the small Markdown
/// subset `markdown_to_html` renders.
pub fn matrix_text(n: &Notification, dashboard_url: &str) -> String {
format!(
"{} **{}**\n{}\n\n📈 Dashboard: {dashboard_url}",
emoji(&n.tags),
n.title,
n.message
)
}
/// `**bold**` removed, for Matrix's plain-text `body`.
pub fn strip_markdown(text: &str) -> String {
text.replace("**", "")
}
/// `alerting.markdown_to_matrix_html`: escape, `**bold**`, bare URLs as
/// links, newlines as `<br/>`.
pub fn markdown_to_html(text: &str) -> String {
let escaped = text
.replace('&', "&amp;")
.replace('<', "&lt;")
.replace('>', "&gt;");
let mut bold = String::new();
let mut parts = escaped.split("**").peekable();
let mut open = false;
while let Some(part) = parts.next() {
bold.push_str(part);
if parts.peek().is_some() {
bold.push_str(if open { "</strong>" } else { "<strong>" });
open = !open;
}
}
if open {
// an unpaired ** stays literal
if let Some(i) = bold.rfind("<strong>") {
bold.replace_range(i..i + "<strong>".len(), "**");
}
}
let mut out = String::new();
let mut rest = bold.as_str();
while let Some(i) = rest
.find("http://")
.into_iter()
.chain(rest.find("https://"))
.min()
{
out.push_str(&rest[..i]);
let url_len = rest[i..]
.find(|c: char| c.is_whitespace() || c == '<')
.unwrap_or(rest.len() - i);
let url = &rest[i..i + url_len];
out.push_str(&format!("<a href=\"{url}\">{url}</a>"));
rest = &rest[i + url_len..];
}
out.push_str(rest);
out.replace('\n', "<br/>")
}
impl Notifier {
pub async fn publish(&self, n: &Notification) -> bool {
pub async fn publish(&self, ntfy: &Ntfy, n: &Notification) -> bool {
let mut req = self
.http
.post(format!(
"{}/{}",
self.publish_url.trim_end_matches('/'),
ntfy.publish_url.trim_end_matches('/'),
n.topic
))
.timeout(std::time::Duration::from_secs(10))
@@ -455,7 +542,7 @@ impl Notifier {
if !n.tags.is_empty() {
req = req.header("Tags", n.tags.join(","));
}
if let Some(t) = &self.token {
if let Some(t) = &ntfy.token {
req = req.bearer_auth(t);
}
match req.send().await {
@@ -477,7 +564,106 @@ impl Notifier {
}
}
/// Evaluate and publish; returns the notifications delivered.
/// One Matrix room message (`MatrixNotifier.send_message`).
pub async fn send_matrix(&self, m: &MatrixConfig, n: &Notification) -> bool {
static TXN: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let text = matrix_text(n, &self.dashboard_url);
let txn = format!(
"ping-{}-{}",
chrono::Utc::now().timestamp_micros(),
TXN.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
);
let url = format!(
"{}/_matrix/client/v3/rooms/{}/send/m.room.message/{txn}",
m.homeserver.trim_end_matches('/'),
m.room_id
);
let body = serde_json::json!({
"msgtype": "m.text",
"body": strip_markdown(&text),
"format": "org.matrix.custom.html",
"formatted_body": markdown_to_html(&text),
});
match self
.http
.put(url)
.bearer_auth(&m.access_token)
.timeout(std::time::Duration::from_secs(10))
.json(&body)
.send()
.await
{
Ok(r) if r.status().is_success() => true,
Ok(r) => {
let status = r.status();
let body = r.text().await.unwrap_or_default();
tracing::warn!(
"Matrix: HTTP {status} {}",
body.chars().take(120).collect::<String>()
);
false
}
Err(e) => {
tracing::warn!("Matrix: {e}");
false
}
}
}
/// The Matrix channel's states (keys without the prefix). On first use
/// they start as a copy of the ntfy states, so what ntfy already
/// announced is not sent again.
pub async fn matrix_states(
&self,
pool: &PgPool,
states: &HashMap<String, String>,
now: NaiveDateTime,
seed: bool,
) -> Result<HashMap<String, String>> {
let own: HashMap<String, String> = states
.iter()
.filter_map(|(k, v)| {
k.strip_prefix(MATRIX_PREFIX)
.map(|k| (k.to_string(), v.clone()))
})
.collect();
if !own.is_empty() {
return Ok(own);
}
let copy: HashMap<String, String> = states
.iter()
.filter(|(k, _)| !k.starts_with(MATRIX_PREFIX))
.map(|(k, v)| (k.clone(), v.clone()))
.collect();
if seed {
for (k, v) in &copy {
db::set_notification_state(pool, &format!("{MATRIX_PREFIX}{k}"), v, None, now)
.await?;
}
db::set_notification_state(pool, "matrix:seeded", "yes", None, now).await?;
}
Ok(copy)
}
/// What the Matrix channel would send: the transitions that carry a
/// state (per station, outlook, feed). The per-topic summaries ntfy also
/// gets ("Ping River: warning level reached") repeat those, so the room
/// does not get them.
pub fn matrix_plan(
&self,
readings: &[Reading],
forecasts: &[Forecast],
states: &HashMap<String, String>,
now: NaiveDateTime,
) -> Vec<Planned> {
evaluate(readings, forecasts, states, &self.settings, now)
.into_iter()
.filter(|p| p.state.is_some())
.collect()
}
/// Evaluate and publish; returns the notifications delivered (Matrix
/// messages with topic "matrix").
pub async fn run(
&self,
pool: &PgPool,
@@ -487,14 +673,32 @@ impl Notifier {
) -> Result<Vec<Notification>> {
let states = db::notification_states(pool).await?;
let mut sent = vec![];
for p in evaluate(readings, forecasts, &states, &self.settings, now) {
if !self.publish(&p.notification).await {
continue;
if let Some(ntfy) = &self.ntfy {
for p in evaluate(readings, forecasts, &states, &self.settings, now) {
if !self.publish(ntfy, &p.notification).await {
continue;
}
if let Some((key, state, value)) = &p.state {
db::set_notification_state(pool, key, state, *value, now).await?;
}
sent.push(p.notification);
}
if let Some((key, state, value)) = &p.state {
db::set_notification_state(pool, key, state, *value, now).await?;
}
if let Some(m) = &self.matrix {
let mstates = self.matrix_states(pool, &states, now, true).await?;
for p in self.matrix_plan(readings, forecasts, &mstates, now) {
if !self.send_matrix(m, &p.notification).await {
continue;
}
if let Some((key, state, value)) = &p.state {
let key = format!("{MATRIX_PREFIX}{key}");
db::set_notification_state(pool, &key, state, *value, now).await?;
}
sent.push(Notification {
topic: "matrix".into(),
..p.notification
});
}
sent.push(p.notification);
}
Ok(sent)
}
@@ -505,6 +709,28 @@ mod tests {
use super::*;
use chrono::NaiveDate;
#[test]
fn matrix_message_rendering() {
let n = Notification {
topic: "ping-p1-warning".into(),
title: "Warning level at P.1".into(),
message:
"Nawarat Bridge: 3.75 m at 18 Oct 14:00, above the warning level of 3.70 m <x> & y."
.into(),
priority: 4,
tags: vec!["warning".into(), "P.1".into()],
};
let text = matrix_text(&n, "https://water.buildfor.life/");
assert_eq!(
markdown_to_html(&text),
"⚠️ <strong>Warning level at P.1</strong><br/>Nawarat Bridge: 3.75 m at 18 Oct 14:00, \
above the warning level of 3.70 m &lt;x&gt; &amp; y.<br/><br/>📈 Dashboard: \
<a href=\"https://water.buildfor.life/\">https://water.buildfor.life/</a>"
);
assert!(strip_markdown(&text).starts_with("⚠️ Warning level at P.1\nNawarat"));
assert_eq!(markdown_to_html("a ** b"), "a ** b");
}
fn now() -> NaiveDateTime {
NaiveDate::from_ymd_opt(2026, 10, 1)
.unwrap()
rust/collector/src/service.rs
+95 -16
View File
@@ -16,11 +16,17 @@ use ping_forecast::ModelStore;
use ping_forecast::rain::RainSeries;
use serde_json::json;
use sqlx::PgPool;
use std::sync::{Arc, RwLock};
pub struct Collector {
pub cfg: Config,
pub pool: PgPool,
pub stations: Stations,
/// Shared with the website (the station CRUD API edits it).
pub stations: Arc<RwLock<Stations>>,
/// The last status written, shared with /scraping/status.
pub status: Arc<RwLock<serde_json::Value>>,
/// POST /scrape/trigger: poll now.
pub trigger: Arc<tokio::sync::Notify>,
pub rid: RidClient,
hii: HiiClient,
dams: DamsClient,
@@ -61,10 +67,36 @@ impl Collector {
http,
cfg,
pool,
stations,
stations: Arc::new(RwLock::new(stations)),
status: Arc::new(RwLock::new(json!({"is_running": false}))),
trigger: Arc::new(tokio::sync::Notify::new()),
})
}
/// The station mapping as of now (the API may edit it between cycles).
pub fn stations(&self) -> Stations {
self.stations.read().unwrap().clone()
}
/// The website/API state over this collector's pool, stations and status.
pub fn web_state(&self, in_process: bool) -> crate::web::Shared {
crate::web::AppState::new(
self.cfg.clone(),
self.pool.clone(),
self.http.clone(),
self.stations.clone(),
in_process.then(|| self.status.clone()),
in_process.then(|| self.trigger.clone()),
)
}
/// `ping-collector serve`: the website and API only; /scraping/status
/// reads the status file the collector service writes.
pub async fn serve(&self, listen: String) -> Result<()> {
db::ensure_schema(&self.pool).await?;
crate::web::serve(self.web_state(false), listen, shutdown_signal()).await
}
/// HII rain + water level, RID dams (today and yesterday), Open-Meteo.
/// Each source is failure-isolated.
pub async fn collect_hourly(&self, now: NaiveDateTime) {
@@ -107,9 +139,6 @@ impl Collector {
}
fn write_status(&self, stats: &Stats, running: bool) {
let Some(path) = &self.cfg.status_file else {
return;
};
let body = json!({
"is_running": running,
"total_runs": stats.total_runs,
@@ -129,6 +158,10 @@ impl Collector {
"notifications_sent": f.notified,
})),
});
*self.status.write().unwrap() = body.clone();
let Some(path) = &self.cfg.status_file else {
return;
};
let tmp = path.with_extension("json.tmp");
let result =
std::fs::write(&tmp, body.to_string()).and_then(|_| std::fs::rename(&tmp, path));
@@ -144,14 +177,40 @@ impl Collector {
db::ensure_schema(&self.pool).await?;
let shutdown = shutdown_signal();
tokio::pin!(shutdown);
// the website, when WEB_LISTEN is set: stopped with the collector
let (stop_web, web_stopped) = tokio::sync::watch::channel(false);
let web = self.cfg.web.listen.clone().map(|listen| {
let state = self.web_state(true);
let mut rx = web_stopped.clone();
tokio::spawn(async move {
let stop = async move {
let _ = rx.wait_for(|stop| *stop).await;
};
if let Err(e) = crate::web::serve(state, listen, stop).await {
tracing::error!("website stopped: {e:#}");
}
})
});
let result = self.collect_loop(&mut shutdown).await;
let _ = stop_web.send(true);
if let Some(handle) = web {
let _ = tokio::time::timeout(std::time::Duration::from_secs(10), handle).await;
}
result
}
async fn collect_loop(
&self,
shutdown: &mut std::pin::Pin<&mut impl std::future::Future<Output = ()>>,
) -> Result<()> {
if self.cfg.gap_fill_days_on_start > 0 {
let days = Some(self.cfg.gap_fill_days_on_start);
let stations = self.stations();
tokio::select! {
r = rid::fill_gaps(&self.rid, &self.pool, &self.stations, days, bangkok_now()) => {
r = rid::fill_gaps(&self.rid, &self.pool, &stations, days, bangkok_now()) => {
if let Err(e) = r { tracing::error!("startup gap fill failed: {e:#}") }
}
_ = &mut shutdown => return Ok(()),
_ = &mut *shutdown => return Ok(()),
}
}
@@ -168,7 +227,7 @@ impl Collector {
let start = bangkok_now();
self.write_status(&stats, true);
let outcome: CycleOutcome =
rid::run_cycle(&self.rid, &self.pool, &self.stations, start).await;
rid::run_cycle(&self.rid, &self.pool, &self.stations(), start).await;
stats.total_runs += 1;
stats.last_run = Some(start);
if outcome.current {
@@ -217,7 +276,8 @@ impl Collector {
.max(std::time::Duration::from_secs(30));
tokio::select! {
_ = tokio::time::sleep(wait) => {}
_ = &mut shutdown => {
_ = self.trigger.notified() => tracing::info!("collection triggered via the API"),
_ = &mut *shutdown => {
tracing::info!("shutdown signal received");
return Ok(());
}
@@ -231,7 +291,7 @@ impl Collector {
let now = bangkok_now();
if !dry_run {
db::ensure_schema(&self.pool).await?;
let o = rid::run_cycle(&self.rid, &self.pool, &self.stations, now).await;
let o = rid::run_cycle(&self.rid, &self.pool, &self.stations(), now).await;
println!(
"RID: reports {:?}, {} fetched, {} saved, latest {:?}, current hour in: {}{}",
o.dates,
@@ -348,7 +408,9 @@ impl Collector {
);
}
let Some(notifier) = forecasting::notifier(&self.cfg, self.http.clone()) else {
bail!("ntfy is off: NTFY_SERVER is not set");
bail!(
"notifications are off: neither NTFY_SERVER nor MATRIX_ACCESS_TOKEN/MATRIX_ROOM_ID is set"
);
};
let mut store = ModelStore::new(&self.cfg.models_dir);
let models = store.refresh()?;
@@ -370,15 +432,32 @@ impl Collector {
db::ensure_schema(&self.pool).await?;
if dry_run {
let states = db::notification_states(&self.pool).await?;
let plan =
notify::evaluate(&readings, &snap.forecasts, &states, &notifier.settings, now);
let mut plan = vec![];
if notifier.ntfy.is_some() {
plan.extend(
notify::evaluate(&readings, &snap.forecasts, &states, &notifier.settings, now)
.into_iter()
.map(|p| (p.notification.topic.clone(), p)),
);
}
if notifier.matrix.is_some() {
let mstates = notifier
.matrix_states(&self.pool, &states, now, false)
.await?;
plan.extend(
notifier
.matrix_plan(&readings, &snap.forecasts, &mstates, now)
.into_iter()
.map(|p| ("matrix".to_string(), p)),
);
}
if plan.is_empty() {
println!("no transitions: nothing would be published");
}
for p in plan {
for (to, p) in plan {
println!(
"would publish {}: {} -- {}",
p.notification.topic, p.notification.title, p.notification.message
"would publish {to}: {} -- {}",
p.notification.title, p.notification.message
);
}
} else {
rust/collector/src/stations.rs
+66 -2
View File
@@ -4,13 +4,13 @@
//! else the defaults bundled in src/data/stations.json.
use anyhow::{Context, Result};
use serde::Deserialize;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
use std::path::Path;
const BUNDLED: &str = include_str!("../../../src/data/stations.json");
#[derive(Clone, Debug, Deserialize)]
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct StationInfo {
pub code: String,
pub thai_name: String,
@@ -55,6 +55,46 @@ impl Stations {
Ok(Stations(map))
}
pub fn iter(&self) -> impl Iterator<Item = (&u32, &StationInfo)> {
self.0.iter()
}
pub fn info(&self, n: u32) -> Option<&StationInfo> {
self.0.get(&n)
}
pub fn info_mut(&mut self, n: u32) -> Option<&mut StationInfo> {
self.0.get_mut(&n)
}
/// The id a new station gets (the station CRUD API): one past the highest.
pub fn next_id(&self) -> u32 {
self.0.keys().max().map_or(1, |m| m + 1)
}
pub fn insert(&mut self, n: u32, info: StationInfo) {
self.0.insert(n, info);
}
pub fn remove(&mut self, n: u32) -> Option<StationInfo> {
self.0.remove(&n)
}
/// Persist to the runtime file as the Python scraper's `save_stations`
/// (indent 2, UTF-8, trailing newline), atomically.
pub fn save(&self, path: &Path) -> Result<()> {
// numeric key order ("10" after "9"), as the Python mapping kept it
let mut obj = serde_json::Map::new();
for (k, v) in &self.0 {
obj.insert(k.to_string(), serde_json::to_value(v)?);
}
let text = serde_json::to_string_pretty(&serde_json::Value::Object(obj))? + "\n";
let tmp = path.with_extension("json.tmp");
std::fs::write(&tmp, text).with_context(|| format!("writing {}", tmp.display()))?;
std::fs::rename(&tmp, path).with_context(|| format!("replacing {}", path.display()))?;
Ok(())
}
/// Metadata for report column `n`, with the Python fallback for unknown
/// columns (`P.{19+n}` / `Station n`).
pub fn get(&self, n: u32) -> StationInfo {
@@ -81,4 +121,28 @@ mod tests {
assert_eq!(s.get(8).code, "P.1");
assert_eq!(s.get(99).code, "P.118"); // Python fallback
}
#[test]
fn save_round_trips() {
let mut s = Stations::bundled();
let id = s.next_id();
assert_eq!(id, 17);
let mut extra = s.get(8);
extra.code = "P.999".into();
s.insert(id, extra);
let dir = std::env::temp_dir().join(format!("ping-stations-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("stations.json");
s.save(&path).unwrap();
let text = std::fs::read_to_string(&path).unwrap();
assert!(
text.starts_with("{\n \"1\": {\n \"code\": \"P.20\","),
"{text}"
);
assert!(text.contains("\"thai_name\": \"บ้านเชียงดาว\""));
let back = Stations::load(&path).unwrap();
assert_eq!(back.info(17).unwrap().code, "P.999");
assert_eq!(back.iter().count(), 17);
std::fs::remove_dir_all(&dir).unwrap();
}
}
rust/collector/src/web/cache.rs
+244
View File
@@ -0,0 +1,244 @@
//! Stale-while-revalidate response cache, as `_cached_swr` in src/web_api.py:
//!
//! - fresh entry: served;
//! - expired entry (no older than `max_stale`): served at once, marked stale,
//! and ONE background refresh is started;
//! - no entry: computed inline, single-flight per key (concurrent misses
//! wait for the one computation instead of piling queries onto Postgres);
//! - a failed computation serves the last good value, whatever its age;
//! - empty results are not cached, so recovery is immediate;
//! - a byte budget per cache: the oldest entries go first (query parameters
//! are client-controlled, so keys are unbounded, and one history response
//! can be 10 MB).
use super::error::ApiError;
use std::collections::{HashMap, HashSet};
use std::future::Future;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
/// The size of a cached value, for the budget.
pub trait Weigh {
fn weight(&self) -> usize;
}
impl Weigh for axum::body::Bytes {
fn weight(&self) -> usize {
self.len()
}
}
impl Weigh for (axum::body::Bytes, bool) {
fn weight(&self) -> usize {
self.0.len()
}
}
#[cfg(test)]
impl Weigh for u32 {
fn weight(&self) -> usize {
4
}
}
pub struct Swr<T> {
budget: usize,
entries: Mutex<HashMap<String, (Instant, T)>>,
flights: Mutex<HashMap<String, Arc<tokio::sync::Mutex<()>>>>,
refreshing: Mutex<HashSet<String>>,
}
/// What a computation returns: the value and whether to keep it.
pub type Computed<T> = Result<(T, bool), ApiError>;
impl<T: Clone + Send + Sync + Weigh + 'static> Swr<T> {
/// A cache holding at most `budget` bytes.
pub fn new(budget: usize) -> Arc<Self> {
Arc::new(Self {
budget,
entries: Mutex::new(HashMap::new()),
flights: Mutex::new(HashMap::new()),
refreshing: Mutex::new(HashSet::new()),
})
}
fn entry(&self, key: &str) -> Option<(Instant, T)> {
self.entries.lock().unwrap().get(key).cloned()
}
/// `(value, served_stale)`
pub async fn get<F, Fut>(
self: &Arc<Self>,
key: String,
ttl: Duration,
max_stale: Option<Duration>,
compute: F,
) -> Result<(T, bool), ApiError>
where
F: Fn() -> Fut + Send + Sync + 'static,
Fut: Future<Output = Computed<T>> + Send + 'static,
{
if let Some((at, v)) = self.entry(&key) {
let age = at.elapsed();
if age < ttl {
return Ok((v, false));
}
if max_stale.is_none_or(|m| age <= m) {
let start = self.refreshing.lock().unwrap().insert(key.clone());
if start {
let me = self.clone();
tokio::spawn(async move {
if let Err(e) = me.compute(&key, ttl, &compute).await {
tracing::warn!("{key}: background refresh failed: {}", e.detail);
}
me.refreshing.lock().unwrap().remove(&key);
});
}
return Ok((v, true));
}
}
self.compute(&key, ttl, &compute).await
}
async fn compute<F, Fut>(
&self,
key: &str,
ttl: Duration,
compute: &F,
) -> Result<(T, bool), ApiError>
where
F: Fn() -> Fut,
Fut: Future<Output = Computed<T>>,
{
let flight = self
.flights
.lock()
.unwrap()
.entry(key.to_string())
.or_default()
.clone();
let _guard = flight.lock().await;
let previous = self.entry(key);
if let Some((at, v)) = &previous
&& at.elapsed() < ttl
{
return Ok((v.clone(), false));
}
match compute().await {
Ok((v, keep)) => {
if keep && v.weight() <= self.budget {
let mut entries = self.entries.lock().unwrap();
entries.insert(key.to_string(), (Instant::now(), v.clone()));
let mut total: usize = entries.values().map(|(_, v)| v.weight()).sum();
if total > self.budget {
let mut ages: Vec<(Instant, String)> = entries
.iter()
.map(|(k, (at, _))| (*at, k.clone()))
.collect();
ages.sort();
for (_, k) in ages {
if total <= self.budget {
break;
}
if let Some((_, old)) = entries.remove(&k) {
total -= old.weight();
}
}
}
}
Ok((v, false))
}
Err(e) => match previous {
Some((at, v)) => {
tracing::warn!(
"{key}: recompute failed ({}); serving {}s-stale copy",
e.detail,
at.elapsed().as_secs()
);
Ok((v, true))
}
None => Err(e),
},
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicU32, Ordering};
#[tokio::test]
async fn fresh_stale_and_error_fallback() {
let cache: Arc<Swr<u32>> = Swr::new(1024);
let calls = Arc::new(AtomicU32::new(0));
let fail = Arc::new(std::sync::atomic::AtomicBool::new(false));
let compute = {
let (calls, fail) = (calls.clone(), fail.clone());
move || {
let (calls, fail) = (calls.clone(), fail.clone());
async move {
let n = calls.fetch_add(1, Ordering::SeqCst) + 1;
if fail.load(Ordering::SeqCst) {
Err(ApiError::new(500, "down"))
} else {
Ok((n, true))
}
}
}
};
let ttl = Duration::from_millis(40);
let key = || "k".to_string();
assert_eq!(
cache.get(key(), ttl, None, compute.clone()).await.unwrap(),
(1, false)
);
assert_eq!(
cache.get(key(), ttl, None, compute.clone()).await.unwrap(),
(1, false)
);
tokio::time::sleep(Duration::from_millis(60)).await;
// expired: the old value at once, refreshed in the background
assert_eq!(
cache.get(key(), ttl, None, compute.clone()).await.unwrap(),
(1, true)
);
tokio::time::sleep(Duration::from_millis(10)).await;
assert_eq!(
cache.get(key(), ttl, None, compute.clone()).await.unwrap(),
(2, false)
);
// older than max_stale and the database is down: last good value
tokio::time::sleep(Duration::from_millis(60)).await;
fail.store(true, Ordering::SeqCst);
let max = Some(Duration::from_millis(1));
assert_eq!(
cache.get(key(), ttl, max, compute.clone()).await.unwrap(),
(2, true)
);
// nothing cached yet: the error
assert!(cache.get("other".into(), ttl, max, compute).await.is_err());
}
#[tokio::test]
async fn budget_evicts_the_oldest() {
let cache: Arc<Swr<u32>> = Swr::new(8); // two u32 entries
let ttl = Duration::from_secs(60);
for (i, k) in ["a", "b", "c"].iter().enumerate() {
let v = i as u32;
cache
.get(
k.to_string(),
ttl,
None,
move || async move { Ok((v, true)) },
)
.await
.unwrap();
tokio::time::sleep(Duration::from_millis(2)).await;
}
let entries = cache.entries.lock().unwrap();
assert_eq!(entries.len(), 2);
assert!(!entries.contains_key("a"));
}
}
rust/collector/src/web/data.rs
+357
View File
@@ -0,0 +1,357 @@
//! RID gauge readings: /measurements/*, /api/stats.
use super::error::{ApiError, ApiResult, ok, to_body};
use super::params::{self, Param};
use super::pyfmt::{end_of_day, isoformat, py_round, start_of_day};
use super::{Shared, secs};
use crate::util::bangkok_now;
use axum::extract::{Path, RawQuery, State};
use axum::http::HeaderMap;
use chrono::{NaiveDateTime, TimeDelta};
use serde_json::{Map, Value, json};
use sqlx::postgres::PgRow;
use sqlx::{Column, Row, TypeInfo};
use std::time::Duration;
/// A result row as a JSON object: column names as keys (SQL aliases), in
/// select order. NUMERIC columns must be cast to float8 in the query (as
/// Python's `float(Decimal)`), timestamps become `isoformat()` strings.
pub fn row_json(row: &PgRow) -> Map<String, Value> {
let mut m = Map::new();
for (i, col) in row.columns().iter().enumerate() {
let v = match col.type_info().name() {
"INT2" => json!(row.try_get::<Option<i16>, _>(i).ok().flatten()),
"INT4" => json!(row.try_get::<Option<i32>, _>(i).ok().flatten()),
"INT8" => json!(row.try_get::<Option<i64>, _>(i).ok().flatten()),
"FLOAT4" => super::pyfmt::num(
row.try_get::<Option<f32>, _>(i)
.ok()
.flatten()
.map(f64::from),
),
"FLOAT8" => super::pyfmt::num(row.try_get::<Option<f64>, _>(i).ok().flatten()),
"BOOL" => json!(row.try_get::<Option<bool>, _>(i).ok().flatten()),
"TIMESTAMP" => json!(
row.try_get::<Option<NaiveDateTime>, _>(i)
.ok()
.flatten()
.map(isoformat)
),
"DATE" => json!(
row.try_get::<Option<chrono::NaiveDate>, _>(i)
.ok()
.flatten()
.map(|d| d.format("%Y-%m-%d").to_string())
),
_ => json!(row.try_get::<Option<String>, _>(i).ok().flatten()),
};
m.insert(col.name().to_string(), v);
}
m
}
pub fn rows_json(rows: &[PgRow]) -> Vec<Value> {
rows.iter().map(|r| Value::Object(row_json(r))).collect()
}
fn query(q: &Option<String>) -> std::collections::HashMap<String, String> {
params::query_map(q.as_deref())
}
/// `datetime.now() - timedelta(hours=h)`; `None` where Python overflows.
pub fn hours_before(t: NaiveDateTime, hours: i64) -> Option<NaiveDateTime> {
t.checked_sub_signed(TimeDelta::try_hours(hours)?)
}
const MEASUREMENT_COLUMNS: &str = "m.timestamp, s.station_code, s.english_name AS station_name_en,
s.thai_name AS station_name_th, m.water_level::float8 AS water_level,
m.discharge::float8 AS discharge, m.discharge_percent::float8 AS discharge_percent, m.status";
pub const LATEST: &[Param] = &[Param::int("limit", 100)];
/// `SQLAdapter.get_latest_measurements`: the newest reading per station.
pub async fn latest_rows(pool: &sqlx::PgPool, limit: i64) -> Result<Vec<PgRow>, sqlx::Error> {
if limit < 0 {
// Postgres refuses a negative LIMIT; the Python adapter answered []
return Ok(vec![]);
}
sqlx::query(&format!(
"SELECT {MEASUREMENT_COLUMNS}
FROM water_measurements m
JOIN stations s ON m.station_id = s.id
INNER JOIN (
SELECT station_id, MAX(timestamp) as max_timestamp
FROM water_measurements
GROUP BY station_id
) latest ON m.station_id = latest.station_id AND m.timestamp = latest.max_timestamp
ORDER BY s.station_code
LIMIT $1"
))
.bind(limit)
.fetch_all(pool)
.await
}
/// GET /measurements/latest: the most frequently hit endpoint (every
/// dashboard poll), stale-while-revalidate cached.
pub async fn latest(State(st): State<Shared>, RawQuery(q): RawQuery, h: HeaderMap) -> ApiResult {
st.metrics.count("measurements_latest");
let a = params::parse(LATEST, &query(&q), &[], &h)?;
let limit = a.int("limit");
let pool = st.pool.clone();
let (body, stale) = st
.caches
.latest
.get(
format!("latest:{limit}"),
secs(st.cfg.web.latest_cache_ttl_s),
None,
move || {
let pool = pool.clone();
async move {
let rows = rows_json(&latest_rows(&pool, limit).await?);
let keep = !rows.is_empty();
Ok((to_body(&rows), keep))
}
},
)
.await?;
ok(body, stale)
}
type HistoryRow = (NaiveDateTime, String, Option<f64>, Option<f64>, Option<f64>);
pub const HISTORY: &[Param] = &[
Param::path_str("station_code"),
Param::int("hours", 168).ge(1),
Param::int("limit", 50000).ge(1).le(100000),
Param::opt_date("start", "First day (overrides hours)"),
Param::opt_date("end", "Last day, inclusive"),
];
/// GET /measurements/history/{station_code} (`PostgresHistory.station_history`)
pub async fn history(
State(st): State<Shared>,
Path(code): Path<String>,
RawQuery(q): RawQuery,
h: HeaderMap,
) -> ApiResult {
let a = params::parse(HISTORY, &query(&q), &[("station_code", &code)], &h)?;
let key = format!(
"{code}:{}:{}:{}:{}",
a.py_str("hours"),
a.py_str("limit"),
a.py_str("start"),
a.py_str("end")
);
let (hours, limit, start, end) = (
a.int("hours"),
a.int("limit"),
a.date("start"),
a.date("end"),
);
let pool = st.pool.clone();
let unavailable = || ApiError::new(502, "Measurement history unavailable");
let (body, stale) = st
.caches
.history
.get(
key,
Duration::from_secs(300),
Some(Duration::ZERO),
move || {
let (pool, code) = (pool.clone(), code.clone());
async move {
let end_time = end.map_or_else(bangkok_now, end_of_day);
let start_time = match start {
Some(d) => start_of_day(d),
None => hours_before(end_time, hours).ok_or_else(unavailable)?,
};
if start_time >= end_time {
return Err(unavailable());
}
let rows: Vec<HistoryRow> = sqlx::query_as(
"SELECT m.timestamp, s.station_code, m.water_level::float8,
m.discharge::float8, m.discharge_percent::float8
FROM water_measurements m
JOIN stations s ON m.station_id = s.id
WHERE s.station_code = $1
AND m.timestamp >= $2
AND m.timestamp <= $3
ORDER BY m.timestamp ASC
LIMIT $4",
)
.bind(&code)
.bind(start_time)
.bind(end_time)
.bind(limit)
.fetch_all(&pool)
.await
.map_err(|e| {
tracing::error!("Error fetching measurement history: {e}");
unavailable()
})?;
let out: Vec<Value> = rows
.into_iter()
.map(|(t, code, level, discharge, pct)| {
// no rating curves are defined: Q = 20 * H, as postgres_history.py
let discharge = discharge.or(level.map(|h| py_round(20.0 * h, 2)));
json!({
"timestamp": isoformat(t),
"station_code": code,
"water_level": level,
"discharge": discharge,
"discharge_percent": pct,
})
})
.collect();
Ok((to_body(&out), true))
}
},
)
.await?;
ok(body, stale)
}
pub const STATION: &[Param] = &[
Param::path_str("station_code"),
Param::int("hours", 24),
Param::int("limit", 1000),
];
/// GET /measurements/station/{station_code}: newest first.
pub async fn station(
State(st): State<Shared>,
Path(code): Path<String>,
RawQuery(q): RawQuery,
h: HeaderMap,
) -> ApiResult {
st.metrics.count("measurements_station");
let a = params::parse(STATION, &query(&q), &[("station_code", &code)], &h)?;
let end = bangkok_now();
let start = hours_before(end, a.int("hours"))
.ok_or_else(|| ApiError::new(500, "date value out of range"))?;
let rows = sqlx::query(&format!(
"SELECT {MEASUREMENT_COLUMNS}
FROM water_measurements m
JOIN stations s ON m.station_id = s.id
WHERE m.timestamp BETWEEN $1 AND $2 AND s.station_code IN ($3)
ORDER BY m.timestamp DESC, s.station_code"
))
.bind(start)
.bind(end)
.bind(&code)
.fetch_all(&st.pool)
.await?;
let mut out = rows_json(&rows);
// measurements[:limit], Python slice semantics
let limit = a.int("limit");
let keep = if limit >= 0 {
(limit as usize).min(out.len())
} else {
out.len().saturating_sub(limit.unsigned_abs() as usize)
};
out.truncate(keep);
ok(to_body(&out), false)
}
async fn count(pool: &sqlx::PgPool, table: &str) -> i64 {
sqlx::query_scalar::<_, i64>(&format!("SELECT COUNT(*) FROM {table}"))
.fetch_one(pool)
.await
.unwrap_or(0)
}
type HiiTotals = (
i64,
i64,
i64,
i64,
Option<NaiveDateTime>,
Option<NaiveDateTime>,
Option<NaiveDateTime>,
Option<NaiveDateTime>,
);
async fn compute_stats(pool: sqlx::PgPool) -> super::cache::Computed<axum::body::Bytes> {
// Heavy: full-table counts and coverage; runs at most once per TTL
type Stats = (i64, i64, Option<NaiveDateTime>, Option<NaiveDateTime>, i64);
let (total, stations, first, last, recorded): Stats = sqlx::query_as(
"SELECT COUNT(*), COUNT(DISTINCT station_id), MIN(timestamp), MAX(timestamp),
COUNT(DISTINCT TO_CHAR(timestamp, 'YYYY-MM-DD HH24'))
FROM water_measurements",
)
.fetch_one(&pool)
.await?;
let (Some(mut first_ts), Some(mut last_ts)) = (first, last) else {
return Err(ApiError::new(503, "Database statistics unavailable"));
};
if total == 0 {
return Err(ApiError::new(503, "Database statistics unavailable"));
}
let slot = crate::util::floor_hour;
let expected = (slot(last_ts) - slot(first_ts)).num_hours() + 1;
let coverage = py_round(100.0 * recorded as f64 / expected as f64, 1);
let hii: Option<HiiTotals> = sqlx::query_as(
"SELECT (SELECT COUNT(*) FROM hii_rainfall),
(SELECT COUNT(*) FROM hii_waterlevel),
(SELECT COUNT(*) FROM hii_rain_stations),
(SELECT COUNT(*) FROM hii_wl_stations),
(SELECT MIN(timestamp) FROM hii_rainfall),
(SELECT MAX(timestamp) FROM hii_rainfall),
(SELECT MIN(timestamp) FROM hii_waterlevel),
(SELECT MAX(timestamp) FROM hii_waterlevel)",
)
.fetch_one(&pool)
.await
.map_err(|e| tracing::warn!("HII stats unavailable: {e}"))
.ok();
let openmeteo_n = count(&pool, "openmeteo_rain").await;
let reservoir_n = count(&pool, "rid_reservoir_daily").await;
let (mut rain_n, mut wl_n, mut hii_stations) = (0, 0, 0);
if let Some((rn, wn, rs, ws, rain_lo, rain_hi, wl_lo, wl_hi)) = hii {
(rain_n, wl_n, hii_stations) = (rn, wn, rs + ws);
for lo in [rain_lo, wl_lo].into_iter().flatten() {
if lo < first_ts {
first_ts = lo;
}
}
for hi in [rain_hi, wl_hi].into_iter().flatten() {
if hi > last_ts {
last_ts = hi;
}
}
}
let body = json!({
// whole-DB totals (RID + HII + Open-Meteo + dams), breakdown alongside
"total_measurements": total + rain_n + wl_n + openmeteo_n + reservoir_n,
"rid_measurements": total,
"hii_rainfall_measurements": rain_n,
"hii_waterlevel_measurements": wl_n,
"openmeteo_rain_measurements": openmeteo_n,
"reservoir_measurements": reservoir_n,
"station_count": stations + hii_stations,
"rid_station_count": stations,
"hii_station_count": hii_stations,
"first_timestamp": isoformat(first_ts),
"last_timestamp": isoformat(last_ts),
"days_spanned": (last_ts.date() - first_ts.date()).num_days() + 1,
"coverage_percent": coverage,
});
Ok((to_body(&body), true))
}
/// GET /api/stats: database coverage (totals, date range, hourly coverage).
pub async fn stats(State(st): State<Shared>) -> ApiResult {
st.metrics.count("api_stats");
let pool = st.pool.clone();
let (body, _) = st
.caches
.stats
.get("all".into(), Duration::from_secs(300), None, move || {
compute_stats(pool.clone())
})
.await?;
ok(body, false)
}
rust/collector/src/web/error.rs
+88
View File
@@ -0,0 +1,88 @@
//! FastAPI-shaped errors (`{"detail": ...}`) and JSON responses.
use axum::body::Bytes;
use axum::http::{HeaderValue, StatusCode, header};
use axum::response::{IntoResponse, Response};
use serde::Serialize;
use serde_json::{Value, json};
#[derive(Debug, Clone)]
pub struct ApiError {
pub status: StatusCode,
pub detail: Value,
}
impl ApiError {
pub fn new(status: u16, detail: impl Into<String>) -> Self {
Self {
status: StatusCode::from_u16(status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR),
detail: Value::String(detail.into()),
}
}
/// 422 with pydantic's error list.
pub fn validation(errors: Vec<Value>) -> Self {
Self {
status: StatusCode::UNPROCESSABLE_ENTITY,
detail: Value::Array(errors),
}
}
/// An unexpected failure (FastAPI handlers answer `500 str(e)`).
pub fn internal(e: impl std::fmt::Display) -> Self {
Self::new(500, e.to_string())
}
}
impl From<sqlx::Error> for ApiError {
fn from(e: sqlx::Error) -> Self {
tracing::error!("database: {e}");
Self::internal(e)
}
}
impl From<anyhow::Error> for ApiError {
fn from(e: anyhow::Error) -> Self {
tracing::error!("{e:#}");
Self::new(500, format!("{e:#}"))
}
}
impl IntoResponse for ApiError {
fn into_response(self) -> Response {
json_response(
self.status,
to_body(&json!({ "detail": self.detail })),
false,
)
}
}
pub type ApiResult = Result<Response, ApiError>;
pub fn to_body(v: &impl Serialize) -> Bytes {
Bytes::from(serde_json::to_vec(v).expect("JSON-serializable value"))
}
/// A JSON body; `stale` adds `X-Data-Stale: true` (served from the cache
/// after a failed refresh, or an old stored forecast).
pub fn json_response(status: StatusCode, body: Bytes, stale: bool) -> Response {
let mut r = (status, body).into_response();
let h = r.headers_mut();
h.insert(
header::CONTENT_TYPE,
HeaderValue::from_static("application/json"),
);
if stale {
h.insert("x-data-stale", HeaderValue::from_static("true"));
}
r
}
pub fn ok(body: Bytes, stale: bool) -> ApiResult {
Ok(json_response(StatusCode::OK, body, stale))
}
pub fn ok_json(v: &impl Serialize) -> ApiResult {
ok(to_body(v), false)
}
rust/collector/src/web/forecasts.rs
+333
View File
@@ -0,0 +1,333 @@
//! Flood forecasts: /forecast, /api/forecast/*, /api/notifications.
use super::data::{hours_before, rows_json};
use super::error::{ApiError, ApiResult, ok, ok_json, to_body};
use super::params::{self, Param};
use super::pyfmt::{end_of_day, isoformat, py_round, start_of_day};
use super::{Shared, secs};
use crate::util::bangkok_now;
use axum::extract::{Path, RawQuery, State};
use axum::http::HeaderMap;
use chrono::NaiveDateTime;
use serde_json::{Map, Value, json};
use std::time::Duration;
/// GET /forecast: flood-risk forecasts per station for the 6/12/24 h
/// horizons, as the forecaster last stored them (forecast_current).
///
/// The Python app recomputed in-process when the stored forecast was older
/// than FORECAST_MAX_AGE_SECONDS; here the forecaster runs in the same
/// process, so an old forecast is served marked `X-Data-Stale: true`.
pub async fn forecast(State(st): State<Shared>) -> ApiResult {
st.metrics.count("forecast");
let pool = st.pool.clone();
let max_age = st.cfg.web.forecast_max_age_s as i64;
let ((body, old), _) = st
.caches
.forecast
.get(
"stored".into(),
Duration::from_secs(60),
Some(Duration::ZERO),
move || {
let pool = pool.clone();
async move {
let row: Option<(NaiveDateTime, String)> = sqlx::query_as(
"SELECT computed_at, payload FROM forecast_current WHERE id = 'all'",
)
.fetch_optional(&pool)
.await?;
let Some((computed_at, payload)) = row else {
return Err(ApiError::new(
503,
"No forecast stored yet (the forecaster has not run)",
));
};
let age = (bangkok_now() - computed_at).num_seconds();
if age > max_age {
tracing::warn!("stored forecast is {:.1} h old", age as f64 / 3600.0);
}
Ok(((payload.into(), age > max_age), true))
}
},
)
.await?;
ok(body, old)
}
pub const HISTORY: &[Param] = &[
Param::path_str("station_code"),
Param::int("hours", 168).ge(1),
Param::opt_date("start", "First day (overrides hours)"),
Param::opt_date("end", "Last day, inclusive"),
Param::opt_int("horizon").ge(1).le(48),
];
/// GET /api/forecast/history/{station_code}: issued model forecasts, what
/// the model predicted when (`ForecastHistoryStore.fetch`).
pub async fn history(
State(st): State<Shared>,
Path(code): Path<String>,
RawQuery(q): RawQuery,
h: HeaderMap,
) -> ApiResult {
st.metrics.count("forecast_history");
let a = params::parse(
HISTORY,
&params::query_map(q.as_deref()),
&[("station_code", &code)],
&h,
)?;
let end = a.date("end").map_or_else(bangkok_now, end_of_day);
let start = match a.date("start") {
Some(d) => start_of_day(d),
None => hours_before(end, a.int("hours"))
.ok_or_else(|| ApiError::new(500, "date value out of range"))?,
};
let horizon = a.opt_int("horizon");
let rows = sqlx::query(
"SELECT as_of, station_code, horizon_hours,
predicted_max_level::float8 AS predicted_max_level,
p_warning::float8 AS p_warning, p_danger::float8 AS p_danger,
current_level::float8 AS current_level, model_version, source
FROM forecast_history
WHERE station_code = $1 AND as_of >= $2 AND as_of <= $3
AND ($4::int IS NULL OR horizon_hours = $4)
ORDER BY as_of ASC, horizon_hours ASC LIMIT 5000",
)
.bind(&code)
.bind(start)
.bind(end)
.bind(horizon.map(|v| v as i32))
.fetch_all(&st.pool)
.await;
match rows {
Ok(rows) => ok_json(&rows_json(&rows)),
Err(e) => {
// as the Python store: an empty history rather than an error
tracing::error!("forecast history fetch failed: {e}");
ok_json(&Vec::<Value>::new())
}
}
}
/// GET /api/notifications: public ntfy settings for the dashboard's
/// subscribe links.
pub async fn notifications(State(st): State<Shared>) -> ApiResult {
let Some(server) = &st.cfg.ntfy_server else {
return ok_json(&json!({"enabled": false}));
};
let p = &st.cfg.ntfy_topic_prefix;
ok_json(&json!({
"enabled": true,
"server": server,
"prefix": p,
"topics": {
"warning": format!("{p}-warning"),
"danger": format!("{p}-danger"),
"p1_outlook": format!("{p}-p1-outlook"),
"status": format!("{p}-status"),
"station_pattern": format!("{p}-<station>-warning | {p}-<station>-danger (station code lowercase, no dot: p1, p103)"),
},
"semantics": "transitions only: one message on crossing up, one all-clear on the way down (0.10 m hysteresis)",
}))
}
pub const SKILL: &[Param] = &[
Param::str("station_code", "P.1"),
Param::int("horizon", 24).ge(1).le(48),
];
const MIN_VERIFIED: usize = 24; // fewer than a day of verified hours is not a number
struct Verified {
as_of: NaiveDateTime,
predicted: f64,
current: Option<f64>,
observed: f64,
}
/// Python's `sum()` (3.11: left to right) / len
fn mean(v: impl Iterator<Item = f64>) -> f64 {
let (mut s, mut n) = (0.0, 0usize);
for x in v {
s += x;
n += 1;
}
s / n as f64
}
/// `src/ml/skill.compute_skill`: issued forecasts verified against what the
/// river did, per model version, with a persistence baseline.
pub async fn compute_skill(
pool: &sqlx::PgPool,
code: &str,
horizon: i64,
now: NaiveDateTime,
) -> Result<Value, sqlx::Error> {
let verifiable_before = now - chrono::TimeDelta::hours(horizon);
type Row = (
NaiveDateTime,
Option<String>,
f64,
Option<f64>,
Option<f64>,
Option<i64>,
);
let rows: Vec<Row> = sqlx::query_as(
"SELECT f.as_of, f.model_version, f.predicted_max_level::float8, f.current_level::float8,
(SELECT MAX(m.water_level) FROM water_measurements m
JOIN stations s ON s.id = m.station_id
WHERE s.station_code = f.station_code
AND m.timestamp > f.as_of
AND m.timestamp <= f.as_of + (f.horizon_hours || ' hours')::interval)::float8,
(SELECT COUNT(m.water_level) FROM water_measurements m
JOIN stations s ON s.id = m.station_id
WHERE s.station_code = f.station_code
AND m.timestamp > f.as_of
AND m.timestamp <= f.as_of + (f.horizon_hours || ' hours')::interval)
FROM forecast_history f
WHERE f.station_code = $1 AND f.horizon_hours = $2
AND f.source = 'model' AND f.predicted_max_level IS NOT NULL
AND f.as_of <= $3
ORDER BY f.as_of",
)
.bind(code)
.bind(horizon as i32)
.bind(verifiable_before)
.fetch_all(pool)
.await?;
// versions in order of first appearance, as Python's dict
let mut by_version: Vec<(String, Vec<Verified>)> = vec![];
for (as_of, version, predicted, current, observed, n) in rows {
// most of the window must be observed, or the "max" is not the peak
let Some(observed) = observed else { continue };
if (n.unwrap_or(0) as f64) < horizon as f64 * 0.75 {
continue;
}
let version = version.unwrap_or_else(|| "unknown".into());
let row = Verified {
as_of,
predicted,
current,
observed,
};
match by_version.iter_mut().find(|(v, _)| *v == version) {
Some((_, list)) => list.push(row),
None => by_version.push((version, vec![row])),
}
}
let mut versions: Vec<Map<String, Value>> = vec![];
for (version, rows) in &by_version {
let err: Vec<f64> = rows.iter().map(|r| r.predicted - r.observed).collect();
let mae = mean(err.iter().map(|e| e.abs()));
let bias = mean(err.iter().copied());
let pers: Vec<(f64, f64)> = rows
.iter()
.filter_map(|r| r.current.map(|c| (c, r.observed)))
.collect();
let persistence = (!pers.is_empty()).then(|| mean(pers.iter().map(|(c, o)| (c - o).abs())));
let high: Vec<(f64, f64)> = rows
.iter()
.filter(|r| r.observed >= 2.0)
.map(|r| (r.predicted, r.observed))
.collect();
let first = rows.iter().map(|r| r.as_of).min().expect("non-empty");
let last = rows.iter().map(|r| r.as_of).max().expect("non-empty");
let mut v = Map::new();
v.insert("model_version".into(), json!(version));
v.insert("first_issued".into(), json!(isoformat(first)));
v.insert("last_issued".into(), json!(isoformat(last)));
v.insert("n".into(), json!(rows.len()));
v.insert("mae_m".into(), json!(py_round(mae, 3)));
v.insert("bias_m".into(), json!(py_round(bias, 3)));
v.insert(
"persistence_mae_m".into(),
json!(persistence.map(|p| py_round(p, 3))),
);
v.insert(
"skill".into(),
json!(
persistence
.filter(|p| *p != 0.0)
.map(|p| py_round(1.0 - mae / p, 3))
),
);
v.insert("above_2m_n".into(), json!(high.len()));
v.insert(
"above_2m_mae_m".into(),
json!(
(!high.is_empty())
.then(|| py_round(mean(high.iter().map(|(p, o)| (p - o).abs())), 3))
),
);
v.insert("enough_data".into(), json!(rows.len() >= MIN_VERIFIED));
versions.push(v);
}
// stable sort on the ISO string, as Python
versions.sort_by(|a, b| a["first_issued"].as_str().cmp(&b["first_issued"].as_str()));
let current = versions.last().cloned();
let previous = versions
.len()
.checked_sub(1)
.and_then(|n| {
versions[..n]
.iter()
.rev()
.find(|v| v["enough_data"] == true)
})
.cloned();
let trend = match (&current, &previous) {
(Some(c), Some(p)) if c["enough_data"] == true => {
let (cm, pm) = (c["mae_m"].as_f64().unwrap(), p["mae_m"].as_f64().unwrap());
let skill_delta = match (c["skill"].as_f64(), p["skill"].as_f64()) {
(Some(cs), Some(ps)) => json!(py_round(cs - ps, 3)),
_ => Value::Null,
};
json!({
"previous_version": p["model_version"],
"mae_delta_m": py_round(cm - pm, 3),
"skill_delta": skill_delta,
"better": cm < pm,
})
}
_ => Value::Null,
};
Ok(json!({
"station_code": code,
"horizon_hours": horizon,
"verified_until": isoformat(verifiable_before),
"min_verified": MIN_VERIFIED,
"versions": versions,
"current": current,
"trend": trend,
}))
}
/// GET /api/forecast/skill: is the model getting better?
pub async fn skill(State(st): State<Shared>, RawQuery(q): RawQuery, h: HeaderMap) -> ApiResult {
st.metrics.count("forecast_skill");
let a = params::parse(SKILL, &params::query_map(q.as_deref()), &[], &h)?;
let (code, horizon) = (a.str("station_code").to_string(), a.int("horizon"));
let pool = st.pool.clone();
let (body, stale) = st
.caches
.hii
.get(
format!("skill:{code}:{horizon}"),
secs(st.cfg.web.hii_cache_ttl_s.max(900)),
None,
move || {
let (pool, code) = (pool.clone(), code.clone());
async move {
let v = compute_skill(&pool, &code, horizon, bangkok_now()).await?;
Ok((to_body(&v), true))
}
},
)
.await?;
ok(body, stale)
}
rust/collector/src/web/hii.rs
+427
View File
@@ -0,0 +1,427 @@
//! HII/ThaiWater feeds: /api/hii/*, /sensors/thaiwater.
use super::data::{hours_before, rows_json};
use super::error::{ApiError, ApiResult, ok, to_body};
use super::params::{self, Param};
use super::pyfmt::{isoformat, num, numpy_mean, py_round, py_str};
use super::{Shared, secs};
use crate::util::{bangkok_now, floor_hour};
use axum::extract::{RawQuery, State};
use axum::http::HeaderMap;
use chrono::{NaiveDateTime, TimeDelta};
use serde_json::{Map, Value, json};
use std::collections::BTreeMap;
use std::time::Duration;
pub const LATEST: &[Param] = &[Param::int("hours", 26).ge(1).le(168)];
const RAIN_SQL: &str = "
SELECT s.id AS station_id, s.oldcode, s.name_en, s.name_th,
s.latitude::float8 AS latitude, s.longitude::float8 AS longitude, s.agency,
m.timestamp, m.rain_1h::float8 AS rain_1h, m.rain_24h::float8 AS rain_24h
FROM hii_rainfall m
JOIN hii_rain_stations s ON s.id = m.station_id
JOIN (SELECT station_id, MAX(timestamp) AS latest_ts
FROM hii_rainfall WHERE timestamp >= $1
GROUP BY station_id) latest
ON latest.station_id = m.station_id AND latest.latest_ts = m.timestamp
ORDER BY s.id";
const WATERLEVEL_SQL: &str = "
SELECT s.id AS station_id, s.oldcode, s.rid_code, s.name_en, s.name_th,
s.latitude::float8 AS latitude, s.longitude::float8 AS longitude, s.agency,
s.river_name, s.offset_msl::float8 AS offset_msl,
s.min_bank_msl::float8 AS min_bank_msl, s.is_key_station,
m.timestamp, m.wl_msl::float8 AS wl_msl, m.discharge::float8 AS discharge,
m.storage_percent::float8 AS storage_percent, m.situation_level,
m.diff_wl_bank::float8 AS diff_wl_bank
FROM hii_waterlevel m
JOIN hii_wl_stations s ON s.id = m.station_id
JOIN (SELECT station_id, MAX(timestamp) AS latest_ts
FROM hii_waterlevel WHERE timestamp >= $1
GROUP BY station_id) latest
ON latest.station_id = m.station_id AND latest.latest_ts = m.timestamp
ORDER BY s.id";
async fn latest_feed(
st: Shared,
feed: &'static str,
sql: &'static str,
q: Option<String>,
h: HeaderMap,
) -> ApiResult {
let a = params::parse(LATEST, &params::query_map(q.as_deref()), &[], &h)?;
let hours = a.int("hours");
let pool = st.pool.clone();
let (body, stale) = st
.caches
.hii
.get(
format!("{feed}:{hours}"),
secs(st.cfg.web.hii_cache_ttl_s),
None,
move || {
let pool = pool.clone();
async move {
let cutoff = hours_before(bangkok_now(), hours)
.ok_or_else(|| ApiError::new(500, "date value out of range"))?;
let rows = rows_json(&sqlx::query(sql).bind(cutoff).fetch_all(&pool).await?);
let keep = !rows.is_empty();
Ok((to_body(&rows), keep))
}
},
)
.await?;
ok(body, stale)
}
/// GET /api/hii/rainfall/latest: latest rainfall per HII station (Ping basin).
pub async fn rainfall_latest(
State(st): State<Shared>,
RawQuery(q): RawQuery,
h: HeaderMap,
) -> ApiResult {
st.metrics.count("hii_rainfall");
latest_feed(st, "rain", RAIN_SQL, q, h).await
}
/// GET /api/hii/waterlevel/latest: latest water level per HII station (m MSL).
pub async fn waterlevel_latest(
State(st): State<Shared>,
RawQuery(q): RawQuery,
h: HeaderMap,
) -> ApiResult {
st.metrics.count("hii_waterlevel");
latest_feed(st, "waterlevel", WATERLEVEL_SQL, q, h).await
}
/// The upper Ping above P.1 (src/ml/hii_rain.py CATCHMENT_BOX):
/// lat_lo, lat_hi, lon_lo, lon_hi.
const CATCHMENT_BOX: [f64; 4] = [18.75, 19.60, 98.60, 99.30];
const MIN_GAUGES_PER_HOUR: i64 = 5;
const WINDOW_H: usize = 24;
pub const CATCHMENT: &[Param] = &[Param::int("days", 14).ge(1).le(60)];
/// `hii_rain.load_gauge_mean`: hourly catchment-mean rain_1h; hours with
/// fewer than MIN_GAUGES_PER_HOUR gauges are NaN.
async fn gauge_mean(
pool: &sqlx::PgPool,
start: NaiveDateTime,
) -> Option<Vec<(NaiveDateTime, f64)>> {
let [lat_lo, lat_hi, lon_lo, lon_hi] = CATCHMENT_BOX.map(|x| x.to_string());
let rows: Vec<(NaiveDateTime, i64, Option<f64>)> = sqlx::query_as(
"SELECT m.timestamp, COUNT(m.rain_1h) AS n, AVG(m.rain_1h)::float8 AS rain_1h
FROM hii_rainfall m JOIN hii_rain_stations s ON s.id = m.station_id
WHERE s.latitude BETWEEN $1::numeric AND $2::numeric
AND s.longitude BETWEEN $3::numeric AND $4::numeric
AND m.rain_1h IS NOT NULL AND m.timestamp >= $5
GROUP BY m.timestamp ORDER BY m.timestamp",
)
.bind(lat_lo)
.bind(lat_hi)
.bind(lon_lo)
.bind(lon_hi)
.bind(start)
.fetch_all(pool)
.await
.map_err(|e| tracing::warn!("HII gauge rain load failed: {e}"))
.ok()?;
if rows.is_empty() {
return None;
}
// groupby(floor('h')).agg(n=sum, rain_1h=mean): pandas' group_mean
// (Kahan-compensated sum / count)
struct Acc {
n: i64,
sum: f64,
comp: f64,
nobs: u32,
}
let mut groups: BTreeMap<NaiveDateTime, Acc> = BTreeMap::new();
for (t, n, rain) in rows {
let g = groups.entry(floor_hour(t)).or_insert(Acc {
n: 0,
sum: 0.0,
comp: 0.0,
nobs: 0,
});
g.n += n;
if let Some(val) = rain.filter(|v| !v.is_nan()) {
g.nobs += 1;
let y = val - g.comp;
let t = g.sum + y;
g.comp = t - g.sum - y;
if g.comp.is_nan() {
g.comp = 0.0;
}
g.sum = t;
}
}
Some(
groups
.into_iter()
.map(|(t, g)| {
let mean = if g.nobs == 0 {
f64::NAN
} else {
g.sum / g.nobs as f64
};
(
t,
if g.n < MIN_GAUGES_PER_HOUR {
f64::NAN
} else {
mean
},
)
})
.collect(),
)
}
/// `hii_rain.compare_with_openmeteo`: agreement of 24 h sums over the hours
/// both series have.
fn compare(gauge: &[(NaiveDateTime, f64)], openmeteo: &[(NaiveDateTime, f64)]) -> Value {
let om: std::collections::HashMap<NaiveDateTime, f64> = openmeteo.iter().copied().collect();
let joined: Vec<(f64, f64)> = gauge
.iter()
.filter(|(_, g)| !g.is_nan())
.filter_map(|(t, g)| om.get(t).filter(|o| !o.is_nan()).map(|o| (*g, *o)))
.collect();
if joined.is_empty() {
return json!({"overlap_hours": 0});
}
if joined.len() < WINDOW_H {
return json!({"overlap_hours": joined.len()});
}
let gs: Vec<f64> = joined.iter().map(|p| p.0).collect();
let os: Vec<f64> = joined.iter().map(|p| p.1).collect();
let g = ping_forecast::rolling::rolling_sum(&gs, WINDOW_H)[WINDOW_H - 1..].to_vec();
let o = ping_forecast::rolling::rolling_sum(&os, WINDOW_H)[WINDOW_H - 1..].to_vec();
let diff: Vec<f64> = o.iter().zip(&g).map(|(o, g)| o - g).collect();
let abs: Vec<f64> = diff.iter().map(|d| d.abs()).collect();
json!({
"overlap_hours": joined.len(),
"window_h": WINDOW_H,
"gauge_mean_mm": num(Some(numpy_mean(&g))),
"openmeteo_mean_mm": num(Some(numpy_mean(&o))),
"bias_mm": num(Some(numpy_mean(&diff))),
"mae_mm": num(Some(numpy_mean(&abs))),
"corr": num(Some(pearson(&g, &o))),
})
}
/// `np.corrcoef(a, b)[0, 1]` (same formula; numpy's BLAS dot may differ in
/// the last bits). NaN for a constant series, which JSON shows as null.
fn pearson(a: &[f64], b: &[f64]) -> f64 {
let (ma, mb) = (numpy_mean(a), numpy_mean(b));
let da: Vec<f64> = a.iter().map(|x| x - ma).collect();
let db: Vec<f64> = b.iter().map(|x| x - mb).collect();
let dot = |x: &[f64], y: &[f64]| x.iter().zip(y).map(|(p, q)| p * q).sum::<f64>();
let fact = (a.len() - 1) as f64;
let (cab, caa, cbb) = (
dot(&da, &db) / fact,
dot(&da, &da) / fact,
dot(&db, &db) / fact,
);
(cab / caa.sqrt() / cbb.sqrt()).clamp(-1.0, 1.0)
}
fn series_rows(s: &[(NaiveDateTime, f64)]) -> Vec<Value> {
s.iter()
.map(|(t, v)| {
json!({
"timestamp": isoformat(*t),
"rain_mm": if v.is_nan() { Value::Null } else { json!(py_round(*v, 2)) },
})
})
.collect()
}
/// GET /api/hii/rainfall/catchment: HII gauges vs the Open-Meteo series the
/// flood model uses, plus their agreement over the window.
pub async fn rainfall_catchment(
State(st): State<Shared>,
RawQuery(q): RawQuery,
h: HeaderMap,
) -> ApiResult {
st.metrics.count("hii_rain_catchment");
let a = params::parse(CATCHMENT, &params::query_map(q.as_deref()), &[], &h)?;
let days = a.int("days");
let pool = st.pool.clone();
let (body, stale) = st
.caches
.hii
.get(
format!("rain_catchment:{days}"),
secs(st.cfg.web.hii_cache_ttl_s),
None,
move || {
let pool = pool.clone();
async move {
let start = bangkok_now() - TimeDelta::days(days);
let gauge = gauge_mean(&pool, start).await;
let openmeteo: Option<Vec<(NaiveDateTime, f64)>> =
match sqlx::query_as::<_, (NaiveDateTime, Option<f64>)>(
"SELECT timestamp, catchment_mean::float8 FROM openmeteo_rain
WHERE timestamp >= $1 ORDER BY timestamp",
)
.bind(start)
.fetch_all(&pool)
.await
{
Ok(rows) if !rows.is_empty() => Some(
rows.into_iter()
.map(|(t, v)| (t, v.unwrap_or(f64::NAN)))
.collect(),
),
Ok(_) => None,
Err(e) => {
tracing::warn!("openmeteo_rain read failed: {e}");
None
}
};
let comparison = match (&gauge, &openmeteo) {
(Some(g), Some(o)) => compare(g, o),
_ => json!({"overlap_hours": 0}),
};
let body = json!({
"box": CATCHMENT_BOX,
"gauge": gauge.as_deref().map(series_rows).unwrap_or_default(),
"openmeteo": openmeteo.as_deref().map(series_rows).unwrap_or_default(),
"comparison_24h_sums": comparison,
});
Ok((to_body(&body), true))
}
},
)
.await?;
ok(body, stale)
}
pub const THAIWATER_URL: &str = "https://twa-api-public.thaiwater.net/v2/waterlevel";
/// `ThaiWaterClient._parse_ping_features`: the Ping-basin sensors.
pub fn parse_ping_features(payload: &Value) -> Vec<Value> {
let empty = Map::new();
let obj = |v: Option<&Value>| -> Map<String, Value> {
v.and_then(Value::as_object).cloned().unwrap_or_default()
};
let mut out = vec![];
let collections = payload
.get("data")
.and_then(Value::as_object)
.unwrap_or(&empty);
for collection in collections.values() {
let features = collection.get("features").and_then(Value::as_array);
for feature in features.into_iter().flatten() {
let props = obj(feature.get("properties"));
let basin = obj(props.get("basin"));
if basin.get("basin").and_then(Value::as_str) != Some("Ping") {
continue;
}
let geometry = obj(feature.get("geometry"));
let coords = geometry
.get("coordinates")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
if coords.len() < 2 {
continue;
}
let station = obj(props.get("station"));
let code = station
.get("stationCode")
.and_then(Value::as_str)
.unwrap_or("");
let get = |m: &Map<String, Value>, k: &str| m.get(k).cloned().unwrap_or(Value::Null);
out.push(json!({
"id": format!("thaiwater:{}", py_str(props.get("id").unwrap_or(&Value::Null))),
"station_code": code.split_once('-').map_or(code, |(_, rest)| rest),
"station_name": get(&station, "station"),
"latitude": coords[1],
"longitude": coords[0],
"timestamp": get(&props, "waterlevelDatetime"),
"water_level_msl": get(&props, "waterlevelMsl"),
"bank_percent": get(&props, "storagePercent"),
"distance_to_bank": get(&props, "diffWlBank"),
"river_name": get(&props, "riverName"),
"agency": get(&obj(props.get("agency")), "agencyShort"),
"source": "ThaiWater",
}));
}
}
out
}
/// GET /sensors/thaiwater: current ThaiWater water-level sensors in the Ping
/// basin, fetched live (cached for a minute; the feed updates every ~10 min).
pub async fn thaiwater_sensors(State(st): State<Shared>) -> ApiResult {
st.metrics.count("thaiwater_sensors");
let Some(key) = st.cfg.web.thaiwater_api_key.clone() else {
return Err(ApiError::new(503, "THAIWATER_API_KEY is not configured"));
};
let (http, timeout) = (st.http.clone(), st.cfg.web.request_timeout_s);
let (body, stale) = st
.caches
.sensors
.get("ping".into(), Duration::from_secs(60), None, move || {
let (http, key) = (http.clone(), key.clone());
async move {
let fetch = async {
http.get(THAIWATER_URL)
.header("Accept-Language", "en")
.header("x-api-key", key)
.timeout(Duration::from_secs(timeout))
.send()
.await?
.error_for_status()?
.json::<Value>()
.await
};
match fetch.await {
Ok(payload) => {
let sensors = parse_ping_features(&payload);
let keep = !sensors.is_empty();
Ok((to_body(&sensors), keep))
}
Err(e) => {
tracing::error!("Error fetching ThaiWater sensors: {e}");
Err(ApiError::new(502, "ThaiWater API unavailable"))
}
}
}
})
.await?;
ok(body, stale)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn thaiwater_features() {
let payload = json!({"data": {"a": {"features": [
{"properties": {"id": 2186, "basin": {"basin": "Ping"},
"station": {"stationCode": "FOP-MOU298", "station": "Mae Tom Bridge"},
"waterlevelDatetime": "2026-10-18T15:10:00+07:00", "waterlevelMsl": 794.41,
"storagePercent": -10.76, "diffWlBank": 1.9, "riverName": "Nam Mae Tuen",
"agency": {"agencyShort": "FOP"}},
"geometry": {"coordinates": [98.36419, 17.799877]}},
{"properties": {"id": 1, "basin": {"basin": "Wang"}},
"geometry": {"coordinates": [99.0, 18.0]}},
{"properties": {"basin": {"basin": "Ping"}}, "geometry": {"coordinates": [99.0]}},
{"properties": {"basin": {"basin": "Ping"}}, "geometry": {"coordinates": [99.0, 18.0]}}
]}}});
let s = parse_ping_features(&payload);
assert_eq!(s.len(), 2);
assert_eq!(
s[0].to_string(),
r#"{"id":"thaiwater:2186","station_code":"MOU298","station_name":"Mae Tom Bridge","latitude":17.799877,"longitude":98.36419,"timestamp":"2026-10-18T15:10:00+07:00","water_level_msl":794.41,"bank_percent":-10.76,"distance_to_bank":1.9,"river_name":"Nam Mae Tuen","agency":"FOP","source":"ThaiWater"}"#
);
assert_eq!(s[1]["id"], "thaiwater:None");
assert_eq!(s[1]["station_code"], "");
}
}
rust/collector/src/web/mod.rs
+335
View File
@@ -0,0 +1,335 @@
//! The website and read API: the port of the FastAPI app (src/web_api.py).
//!
//! Same routes, parameters, JSON fields, status codes, cache headers and
//! response caches as the Python app, checked route by route against it by
//! scripts/api_parity.py. It runs inside `ping-collector run` (WEB_LISTEN)
//! or alone (`ping-collector serve`). /docs is the same Swagger UI, fed by an
//! OpenAPI document built from the route table that also validates the
//! parameters (openapi.rs, params.rs).
mod cache;
mod data;
mod error;
mod forecasts;
mod hii;
mod openapi;
mod params;
mod pyfmt;
mod system;
#[cfg(test)]
mod tests;
use crate::config::Config;
use crate::stations::Stations;
use anyhow::{Context, Result};
use axum::Router;
use axum::body::Bytes;
use axum::extract::{Request, State};
use axum::handler::HandlerWithoutStateExt;
use axum::http::{HeaderValue, Method, StatusCode, header};
use axum::middleware::{self, Next};
use axum::response::{IntoResponse, Response};
use axum::routing::{get, post};
use cache::Swr;
use serde_json::{Map, Value, json};
use sqlx::PgPool;
use std::sync::{Arc, Mutex, RwLock};
use std::time::{Duration, Instant};
use tower_http::compression::CompressionLayer;
use tower_http::compression::predicate::{NotForContentType, Predicate, SizeAbove};
use tower_http::cors::{AllowHeaders, AllowMethods, AllowOrigin, Any, CorsLayer};
use tower_http::services::ServeDir;
pub struct AppState {
pub cfg: Config,
pub pool: PgPool,
pub http: reqwest::Client,
pub stations: Arc<RwLock<Stations>>,
/// The collector's status when it runs in this process; otherwise
/// /scraping/status reads SCRAPING_STATUS_FILE, as the Python app did.
pub status: Option<Arc<RwLock<Value>>>,
/// Wakes the collection loop (POST /scrape/trigger).
pub trigger: Option<Arc<tokio::sync::Notify>>,
pub dashboard: String,
pub metrics: Metrics,
pub started: Instant,
caches: Caches,
}
pub type Shared = Arc<AppState>;
struct Caches {
latest: Arc<Swr<Bytes>>,
history: Arc<Swr<Bytes>>,
hii: Arc<Swr<Bytes>>,
stats: Arc<Swr<Bytes>>,
health: Arc<Swr<Bytes>>,
sensors: Arc<Swr<Bytes>>,
/// body, older than FORECAST_MAX_AGE_SECONDS
forecast: Arc<Swr<(Bytes, bool)>>,
}
impl AppState {
pub fn new(
cfg: Config,
pool: PgPool,
http: reqwest::Client,
stations: Arc<RwLock<Stations>>,
status: Option<Arc<RwLock<Value>>>,
trigger: Option<Arc<tokio::sync::Notify>>,
) -> Shared {
let path = cfg.web.static_dir.join("dashboard.html");
let dashboard = std::fs::read_to_string(&path).unwrap_or_else(|e| {
tracing::error!("could not load the dashboard {}: {e}", path.display());
"<h1>Northern Thailand Ping River Monitor API</h1><p>See <code>/docs</code>.</p>".into()
});
Arc::new(AppState {
cfg,
pool,
http,
stations,
status,
trigger,
dashboard,
metrics: Metrics::default(),
started: Instant::now(),
// byte budgets: ~100 MB in all, inside the unit's MemoryMax
caches: Caches {
latest: Swr::new(4 * MB),
history: Swr::new(48 * MB),
hii: Swr::new(24 * MB),
stats: Swr::new(MB),
health: Swr::new(MB),
sensors: Swr::new(4 * MB),
forecast: Swr::new(4 * MB),
},
})
}
}
const MB: usize = 1 << 20;
fn secs(s: u64) -> Duration {
Duration::from_secs(s)
}
/// `/metrics`: request counters per endpoint, as the Python app kept them.
#[derive(Default)]
pub struct Metrics {
counters: Mutex<Map<String, Value>>,
}
impl Metrics {
pub fn count(&self, endpoint: &str) {
let key = format!("api_requests{{endpoint={endpoint}}}");
let mut c = self.counters.lock().unwrap();
let n = c.get(&key).and_then(Value::as_f64).unwrap_or(0.0);
c.insert(key, json!(n + 1.0));
}
fn counters(&self) -> Value {
Value::Object(self.counters.lock().unwrap().clone())
}
}
/// Cache-Control by path, as the Python app's middleware: lets browsers and
/// Cloudflare absorb repeat traffic. Max-ages mirror the data cadence.
const CACHE_CONTROL: &[(&str, &str)] = &[
("/static/", "public, max-age=3600"),
("/measurements/latest", "public, max-age=30"),
("/api/forecast/", "public, max-age=300"),
("/api/hii/", "public, max-age=60"),
("/measurements/history", "public, max-age=300"),
("/forecast", "public, max-age=120"),
("/api/stats", "public, max-age=300"),
("/stations", "public, max-age=300"),
];
/// Requests reported to Umami (server-side API analytics).
const UMAMI_PREFIXES: &[&str] = &[
"/api/",
"/measurements",
"/forecast",
"/stations",
"/sensors",
];
fn cache_control(path: &str) -> Option<&'static str> {
if path == "/" {
return Some("public, max-age=120");
}
CACHE_CONTROL
.iter()
.find(|(prefix, _)| path.starts_with(prefix))
.map(|(_, v)| *v)
}
async fn finish(State(st): State<Shared>, req: Request, next: Next) -> Response {
let path = req.uri().path().to_string();
let method = req.method().clone();
let (host, user_agent) = {
let header = |name: header::HeaderName| {
req.headers()
.get(name)
.and_then(|v| v.to_str().ok())
.map(String::from)
};
(
header(header::HOST).unwrap_or_else(|| "water.buildfor.life".into()),
header(header::USER_AGENT).unwrap_or_default(),
)
};
let mut resp = next.run(req).await;
if method == Method::GET
&& resp.status() == StatusCode::OK
&& !resp.headers().contains_key(header::CACHE_CONTROL)
&& let Some(v) = cache_control(&path)
{
resp.headers_mut()
.insert(header::CACHE_CONTROL, HeaderValue::from_static(v));
}
let web = &st.cfg.web;
if web.umami_track_api
&& !web.umami_website_id.is_empty()
&& UMAMI_PREFIXES.iter().any(|p| path.starts_with(p))
{
let event = json!({
"type": "event",
"payload": {
"website": web.umami_website_id,
"url": path,
"hostname": host,
"name": "api-request",
"data": {"method": method.as_str(), "status": resp.status().as_u16()},
},
});
let req = st
.http
.post(&web.umami_api_url)
.timeout(Duration::from_secs(3))
.header(
header::USER_AGENT,
if user_agent.is_empty() {
"api-client".into()
} else {
user_agent
},
)
.json(&event);
// analytics must never affect the response
tokio::spawn(async move {
let _ = req.send().await;
});
}
resp
}
async fn not_found() -> Response {
error::ApiError::new(404, "Not Found").into_response()
}
async fn method_not_allowed() -> Response {
error::ApiError::new(405, "Method Not Allowed").into_response()
}
fn cors(cfg: &Config) -> CorsLayer {
// as the Python app: any origin without credentials unless
// CORS_ALLOW_ORIGINS lists the allowed ones
let origins = &cfg.web.cors_allow_origins;
if origins.is_empty() {
return CorsLayer::new()
.allow_origin(Any)
.allow_methods(Any)
.allow_headers(Any);
}
let list: Vec<HeaderValue> = origins.iter().filter_map(|o| o.parse().ok()).collect();
CorsLayer::new()
.allow_origin(AllowOrigin::list(list))
.allow_credentials(true)
.allow_methods(AllowMethods::mirror_request())
.allow_headers(AllowHeaders::mirror_request())
}
pub fn router(st: Shared) -> Router {
let static_files =
ServeDir::new(st.cfg.web.static_dir.clone()).not_found_service(not_found.into_service());
// as Starlette's GZipMiddleware(minimum_size=500)
let compress = CompressionLayer::new().compress_when(
SizeAbove::new(500)
.and(NotForContentType::GRPC)
.and(NotForContentType::IMAGES)
.and(NotForContentType::SSE),
);
Router::new()
.route("/", get(system::root))
.route("/robots.txt", get(system::robots))
.route("/llms.txt", get(system::llms))
.route("/sitemap.xml", get(system::sitemap))
.route("/docs", get(openapi::swagger_ui))
.route("/redoc", get(openapi::redoc))
.route("/openapi.json", get(openapi::spec))
.route("/health", get(system::health))
.route("/metrics", get(system::metrics))
.route(
"/stations",
get(system::stations).post(system::create_station),
)
.route(
"/stations/{station_id}",
get(system::station)
.put(system::update_station)
.delete(system::delete_station),
)
.route("/sensors/thaiwater", get(hii::thaiwater_sensors))
.route("/api/hii/rainfall/latest", get(hii::rainfall_latest))
.route("/api/hii/rainfall/catchment", get(hii::rainfall_catchment))
.route("/api/hii/waterlevel/latest", get(hii::waterlevel_latest))
.route("/measurements/history/{station_code}", get(data::history))
.route("/forecast", get(forecasts::forecast))
.route(
"/api/forecast/history/{station_code}",
get(forecasts::history),
)
.route("/api/notifications", get(forecasts::notifications))
.route("/api/forecast/skill", get(forecasts::skill))
.route("/measurements/latest", get(data::latest))
.route("/measurements/station/{station_code}", get(data::station))
.route("/api/stats", get(data::stats))
.route("/scrape/trigger", post(system::trigger))
.route("/scraping/status", get(system::scraping_status))
.route("/config", get(system::config))
.nest_service("/static", static_files)
.fallback(not_found)
.method_not_allowed_fallback(method_not_allowed)
.layer(middleware::from_fn_with_state(st.clone(), finish))
.layer(cors(&st.cfg))
.layer(compress)
.with_state(st)
}
/// Serve until `shutdown` resolves. While the address is taken (the Python
/// app still running during a switch-over) binding is retried every 30 s.
pub async fn serve(
st: Shared,
listen: String,
shutdown: impl std::future::Future<Output = ()> + Send + 'static,
) -> Result<()> {
let mut shutdown = Box::pin(shutdown);
let listener = loop {
match tokio::net::TcpListener::bind(&listen).await {
Ok(l) => break l,
Err(e) => {
tracing::error!("website: cannot listen on {listen}: {e}; retrying in 30 s");
tokio::select! {
_ = tokio::time::sleep(Duration::from_secs(30)) => {}
_ = &mut shutdown => return Ok(()),
}
}
}
};
tracing::info!("website and API on http://{listen}");
axum::serve(listener, router(st))
.with_graceful_shutdown(shutdown)
.await
.context("web server")
}
rust/collector/src/web/openapi.rs
+430
View File
@@ -0,0 +1,430 @@
//! /docs, /redoc and /openapi.json. The document is built from the route
//! table below, whose parameter lists are the same constants the handlers
//! validate with, so the documented parameters are the accepted ones.
//! Summaries, operation ids and descriptions follow FastAPI's conventions
//! (from the Python handler names and docstrings) so /docs reads as before.
use super::params::{Param, python_title};
use super::{data, forecasts, hii, system};
use axum::http::header;
use axum::response::{IntoResponse, Response};
use serde_json::{Map, Value, json};
use std::sync::OnceLock;
pub const TITLE: &str = "Northern Thailand Ping River Monitor API";
const DESCRIPTION: &str =
"Real-time water level monitoring system for Northern Thailand's Ping River Basin stations";
const VERSION: &str = "3.1.3";
enum Resp {
Any,
Html,
Ref(&'static str),
ArrayOf(&'static str),
}
struct Route {
method: &'static str,
path: &'static str,
/// the Python handler's name (summary and operationId)
name: &'static str,
description: &'static str,
params: &'static [Param],
body: Option<&'static str>,
response: Resp,
}
const NONE: &[Param] = &[];
const ROUTES: &[Route] = &[
Route {
method: "get",
path: "/",
name: "root",
description: "Root endpoint with basic dashboard",
params: NONE,
body: None,
response: Resp::Html,
},
Route {
method: "get",
path: "/health",
name: "get_health",
description: "Get system health status",
params: NONE,
body: None,
response: Resp::Ref("HealthResponse"),
},
Route {
method: "get",
path: "/metrics",
name: "get_metrics",
description: "Get application metrics",
params: NONE,
body: None,
response: Resp::Ref("MetricsResponse"),
},
Route {
method: "get",
path: "/stations",
name: "get_stations",
description: "Get list of all monitoring stations",
params: NONE,
body: None,
response: Resp::ArrayOf("StationResponse"),
},
Route {
method: "post",
path: "/stations",
name: "create_station",
description: "Create a new monitoring station",
params: system::ADMIN,
body: Some("StationCreateModel"),
response: Resp::Ref("StationResponse"),
},
Route {
method: "put",
path: "/stations/{station_id}",
name: "update_station",
description: "Update an existing monitoring station",
params: system::ADMIN_STATION_ID,
body: Some("StationUpdateModel"),
response: Resp::Ref("StationResponse"),
},
Route {
method: "delete",
path: "/stations/{station_id}",
name: "delete_station",
description: "Delete a monitoring station",
params: system::ADMIN_STATION_ID,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/stations/{station_id}",
name: "get_station",
description: "Get details of a specific monitoring station",
params: system::STATION_ID,
body: None,
response: Resp::Ref("StationResponse"),
},
Route {
method: "get",
path: "/sensors/thaiwater",
name: "get_thaiwater_sensors",
description: "Get current ThaiWater water-level sensors in the Ping basin.",
params: NONE,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/api/hii/rainfall/latest",
name: "get_hii_rainfall_latest",
description: "Latest rainfall reading per HII station (Ping basin, collected hourly).",
params: hii::LATEST,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/api/hii/rainfall/catchment",
name: "get_hii_rainfall_catchment",
description: "Upper-Ping catchment-mean hourly rain: HII gauges vs the Open-Meteo\nseries the flood model actually uses, plus their agreement over the window.\n\nEvidence-gathering endpoint (docs/FLOOD_FORECASTING.md, HII gauge rain):\nthe gauge table only exists since 2026-08 so it cannot be a training\nfeature yet; this makes the two sources' relationship observable meanwhile.",
params: hii::CATCHMENT,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/api/hii/waterlevel/latest",
name: "get_hii_waterlevel_latest",
description: "Latest water-level reading per HII station (Ping basin, m MSL).",
params: hii::LATEST,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/measurements/history/{station_code}",
name: "get_postgres_history",
description: "Get historical measurements for a station from the configured database.",
params: data::HISTORY,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/forecast",
name: "get_flood_forecasts",
description: "Flood-risk forecasts per station for the 6/12/24 h horizons.",
params: NONE,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/api/forecast/history/{station_code}",
name: "get_forecast_history",
description: "Issued model forecasts for a station — what the model predicted, when.\n\nPopulated by the hourly precompute; enables predicted-vs-actual charts.",
params: forecasts::HISTORY,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/api/notifications",
name: "get_notifications_config",
description: "Public ntfy settings so the dashboard can offer subscribe links.",
params: NONE,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/api/forecast/skill",
name: "get_forecast_skill",
description: "Is the model getting better? Issued forecasts verified against what the\nriver then did, per model version, with a persistence baseline.\n\nRead from forecast_history (what each deployed version predicted, hourly)\njoined to water_measurements; no retraining involved. Cached like the HII\nfeeds because the join is a few hundred correlated subqueries.",
params: forecasts::SKILL,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/measurements/latest",
name: "get_latest_measurements",
description: "Get latest measurements from all stations",
params: data::LATEST,
body: None,
response: Resp::ArrayOf("MeasurementResponse"),
},
Route {
method: "get",
path: "/measurements/station/{station_code}",
name: "get_station_measurements",
description: "Get measurements for a specific station",
params: data::STATION,
body: None,
response: Resp::ArrayOf("MeasurementResponse"),
},
Route {
method: "get",
path: "/api/stats",
name: "get_database_stats",
description: "Get database coverage statistics (totals, date range, hourly coverage)",
params: NONE,
body: None,
response: Resp::Any,
},
Route {
method: "post",
path: "/scrape/trigger",
name: "trigger_scraping",
description: "Trigger manual data scraping",
params: system::ADMIN,
body: None,
response: Resp::Any,
},
Route {
method: "get",
path: "/scraping/status",
name: "get_scraping_status",
description: "Get current scraping status",
params: NONE,
body: None,
response: Resp::Ref("ScrapingStatusResponse"),
},
Route {
method: "get",
path: "/config",
name: "get_config",
description: "Get current configuration (sensitive data masked)",
params: system::ADMIN,
body: None,
response: Resp::Any,
},
];
/// FastAPI's `generate_unique_id`: name + path with non-word characters as
/// `_`, then the method.
fn operation_id(r: &Route) -> String {
let p: String = format!("{}{}", r.name, r.path)
.chars()
.map(|c| {
if c.is_alphanumeric() || c == '_' {
c
} else {
'_'
}
})
.collect();
format!("{p}_{}", r.method)
}
fn reference(name: &str) -> Value {
json!({ "$ref": format!("#/components/schemas/{name}") })
}
fn operation(r: &Route) -> Value {
let op_id = operation_id(r);
let ok_content = match r.response {
Resp::Html => json!({"text/html": {"schema": {"type": "string"}}}),
Resp::Any => json!({"application/json": {"schema": {}}}),
Resp::Ref(name) => json!({"application/json": {"schema": reference(name)}}),
Resp::ArrayOf(name) => json!({"application/json": {"schema": {
"type": "array",
"items": reference(name),
"title": format!("Response {}", python_title(&op_id.replace('_', " "))),
}}}),
};
let mut responses = Map::new();
responses.insert(
"200".into(),
json!({"description": "Successful Response", "content": ok_content}),
);
if !r.params.is_empty() || r.body.is_some() {
responses.insert(
"422".into(),
json!({"description": "Validation Error",
"content": {"application/json": {"schema": reference("HTTPValidationError")}}}),
);
}
let mut op = Map::new();
op.insert(
"summary".into(),
json!(python_title(&r.name.replace('_', " "))),
);
op.insert("description".into(), json!(r.description));
op.insert("operationId".into(), json!(op_id));
if !r.params.is_empty() {
op.insert(
"parameters".into(),
Value::Array(r.params.iter().map(Param::openapi).collect()),
);
}
if let Some(body) = r.body {
op.insert(
"requestBody".into(),
json!({"required": true, "content": {"application/json": {"schema": reference(body)}}}),
);
}
op.insert("responses".into(), Value::Object(responses));
Value::Object(op)
}
pub fn document() -> &'static str {
static DOC: OnceLock<String> = OnceLock::new();
DOC.get_or_init(|| {
let mut paths = Map::new();
for r in ROUTES {
let entry = paths
.entry(r.path)
.or_insert_with(|| Value::Object(Map::new()));
entry
.as_object_mut()
.expect("object")
.insert(r.method.into(), operation(r));
}
let schemas: Value = serde_json::from_str(include_str!("openapi_components.json"))
.expect("openapi_components.json is valid JSON");
json!({
"openapi": "3.1.0",
"info": {"title": TITLE, "description": DESCRIPTION, "version": VERSION},
"paths": paths,
"components": {"schemas": schemas},
})
.to_string()
})
}
pub async fn spec() -> Response {
([(header::CONTENT_TYPE, "application/json")], document()).into_response()
}
fn html(body: String) -> Response {
([(header::CONTENT_TYPE, "text/html; charset=utf-8")], body).into_response()
}
/// The Swagger UI page FastAPI serves (assets from the jsDelivr CDN).
pub async fn swagger_ui() -> Response {
html(format!(
r##"<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link type="text/css" rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui.css">
<title>{TITLE} - Swagger UI</title>
</head>
<body>
<div id="swagger-ui">
</div>
<script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui-bundle.js"></script>
<script>
const ui = SwaggerUIBundle({{
url: '/openapi.json',
"dom_id": "#swagger-ui",
"layout": "BaseLayout",
"deepLinking": true,
"showExtensions": true,
"showCommonExtensions": true,
presets: [
SwaggerUIBundle.presets.apis,
SwaggerUIBundle.SwaggerUIStandalonePreset
],
}})
</script>
</body>
</html>
"##
))
}
pub async fn redoc() -> Response {
html(format!(
r#"<!DOCTYPE html>
<html>
<head>
<title>{TITLE} - ReDoc</title>
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>body {{ margin: 0; padding: 0; }}</style>
</head>
<body>
<noscript>ReDoc requires Javascript to function. Please enable it to browse the documentation.</noscript>
<redoc spec-url="/openapi.json"></redoc>
<script src="https://cdn.jsdelivr.net/npm/redoc@2/bundles/redoc.standalone.js"></script>
</body>
</html>
"#
))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fastapi_naming() {
let doc: Value = serde_json::from_str(document()).unwrap();
let op = &doc["paths"]["/measurements/history/{station_code}"]["get"];
assert_eq!(
op["operationId"],
"get_postgres_history_measurements_history__station_code__get"
);
assert_eq!(op["summary"], "Get Postgres History");
assert_eq!(op["parameters"].as_array().unwrap().len(), 5);
let latest = &doc["paths"]["/measurements/latest"]["get"]["responses"]["200"];
assert_eq!(
latest["content"]["application/json"]["schema"]["title"],
"Response Get Latest Measurements Measurements Latest Get"
);
assert!(doc["paths"]["/stations"]["post"]["responses"]["422"].is_object());
assert!(
doc["paths"]["/health"]["get"]["responses"]
.get("422")
.is_none()
);
assert!(doc["components"]["schemas"]["HTTPValidationError"].is_object());
}
}
rust/collector/src/web/openapi_components.json
+478
View File
@@ -0,0 +1,478 @@
{
"HTTPValidationError": {
"properties": {
"detail": {
"items": {
"$ref": "#/components/schemas/ValidationError"
},
"type": "array",
"title": "Detail"
}
},
"type": "object",
"title": "HTTPValidationError"
},
"HealthResponse": {
"properties": {
"overall_status": {
"type": "string",
"title": "Overall Status"
},
"timestamp": {
"type": "string",
"title": "Timestamp"
},
"checks": {
"additionalProperties": {
"additionalProperties": true,
"type": "object"
},
"type": "object",
"title": "Checks"
}
},
"type": "object",
"required": [
"overall_status",
"timestamp",
"checks"
],
"title": "HealthResponse"
},
"MeasurementResponse": {
"properties": {
"timestamp": {
"type": "string",
"format": "date-time",
"title": "Timestamp"
},
"station_code": {
"type": "string",
"title": "Station Code"
},
"station_name_en": {
"type": "string",
"title": "Station Name En"
},
"station_name_th": {
"type": "string",
"title": "Station Name Th"
},
"water_level": {
"type": "number",
"title": "Water Level"
},
"discharge": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Discharge"
},
"discharge_percent": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Discharge Percent"
},
"status": {
"type": "string",
"title": "Status",
"default": "active"
}
},
"type": "object",
"required": [
"timestamp",
"station_code",
"station_name_en",
"station_name_th",
"water_level"
],
"title": "MeasurementResponse"
},
"MetricsResponse": {
"properties": {
"counters": {
"additionalProperties": {
"type": "number"
},
"type": "object",
"title": "Counters"
},
"gauges": {
"additionalProperties": {
"type": "number"
},
"type": "object",
"title": "Gauges"
},
"histograms": {
"additionalProperties": {
"additionalProperties": {
"type": "number"
},
"type": "object"
},
"type": "object",
"title": "Histograms"
}
},
"type": "object",
"required": [
"counters",
"gauges",
"histograms"
],
"title": "MetricsResponse"
},
"ScrapingStatusResponse": {
"properties": {
"is_running": {
"type": "boolean",
"title": "Is Running"
},
"last_run": {
"anyOf": [
{
"type": "string",
"format": "date-time"
},
{
"type": "null"
}
],
"title": "Last Run"
},
"next_run": {
"anyOf": [
{
"type": "string",
"format": "date-time"
},
{
"type": "null"
}
],
"title": "Next Run"
},
"total_runs": {
"type": "integer",
"title": "Total Runs",
"default": 0
},
"successful_runs": {
"type": "integer",
"title": "Successful Runs",
"default": 0
},
"failed_runs": {
"type": "integer",
"title": "Failed Runs",
"default": 0
},
"feed_current": {
"anyOf": [
{
"type": "boolean"
},
{
"type": "null"
}
],
"title": "Feed Current"
},
"latest_reading": {
"anyOf": [
{
"type": "string",
"format": "date-time"
},
{
"type": "null"
}
],
"title": "Latest Reading"
},
"readings_saved": {
"type": "integer",
"title": "Readings Saved",
"default": 0
}
},
"type": "object",
"required": [
"is_running"
],
"title": "ScrapingStatusResponse"
},
"StationCreateModel": {
"properties": {
"station_code": {
"type": "string",
"title": "Station Code",
"description": "Station code (e.g., P.1, P.20)"
},
"thai_name": {
"type": "string",
"title": "Thai Name",
"description": "Thai name of the station"
},
"english_name": {
"type": "string",
"title": "English Name",
"description": "English name of the station"
},
"latitude": {
"anyOf": [
{
"type": "number",
"maximum": 90.0,
"minimum": -90.0
},
{
"type": "null"
}
],
"title": "Latitude",
"description": "Latitude coordinate"
},
"longitude": {
"anyOf": [
{
"type": "number",
"maximum": 180.0,
"minimum": -180.0
},
{
"type": "null"
}
],
"title": "Longitude",
"description": "Longitude coordinate"
},
"geohash": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Geohash",
"description": "Geohash for the location"
},
"status": {
"type": "string",
"title": "Status",
"description": "Station status",
"default": "active"
}
},
"type": "object",
"required": [
"station_code",
"thai_name",
"english_name"
],
"title": "StationCreateModel"
},
"StationResponse": {
"properties": {
"station_id": {
"type": "integer",
"title": "Station Id"
},
"station_code": {
"type": "string",
"title": "Station Code"
},
"thai_name": {
"type": "string",
"title": "Thai Name"
},
"english_name": {
"type": "string",
"title": "English Name"
},
"latitude": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Latitude"
},
"longitude": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Longitude"
},
"geohash": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Geohash"
},
"status": {
"type": "string",
"title": "Status",
"default": "active"
}
},
"type": "object",
"required": [
"station_id",
"station_code",
"thai_name",
"english_name"
],
"title": "StationResponse"
},
"StationUpdateModel": {
"properties": {
"thai_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Thai Name",
"description": "Thai name of the station"
},
"english_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "English Name",
"description": "English name of the station"
},
"latitude": {
"anyOf": [
{
"type": "number",
"maximum": 90.0,
"minimum": -90.0
},
{
"type": "null"
}
],
"title": "Latitude",
"description": "Latitude coordinate"
},
"longitude": {
"anyOf": [
{
"type": "number",
"maximum": 180.0,
"minimum": -180.0
},
{
"type": "null"
}
],
"title": "Longitude",
"description": "Longitude coordinate"
},
"geohash": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Geohash",
"description": "Geohash for the location"
},
"status": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Status",
"description": "Station status"
}
},
"type": "object",
"title": "StationUpdateModel"
},
"ValidationError": {
"properties": {
"loc": {
"items": {
"anyOf": [
{
"type": "string"
},
{
"type": "integer"
}
]
},
"type": "array",
"title": "Location"
},
"msg": {
"type": "string",
"title": "Message"
},
"type": {
"type": "string",
"title": "Error Type"
},
"input": {
"title": "Input"
},
"ctx": {
"type": "object",
"title": "Context"
}
},
"type": "object",
"required": [
"loc",
"msg",
"type"
],
"title": "ValidationError"
}
}
rust/collector/src/web/params.rs
+450
View File
@@ -0,0 +1,450 @@
//! Query/path parameters: one declaration per route drives both the
//! validation (pydantic-style 422 errors, as FastAPI) and the OpenAPI
//! document behind /docs, so the two cannot drift apart.
use super::error::ApiError;
use chrono::{NaiveDate, NaiveDateTime};
use serde_json::{Map, Value, json};
use std::collections::HashMap;
#[derive(Clone, Copy, Debug, PartialEq)]
pub enum In {
Query,
Path,
Header,
}
#[derive(Clone, Copy, Debug, PartialEq)]
pub enum Kind {
Int,
Str,
Date,
}
#[derive(Clone, Copy, Debug)]
pub enum Default {
None,
Int(i64),
Str(&'static str),
}
#[derive(Clone, Copy, Debug)]
pub struct Param {
pub name: &'static str,
pub loc: In,
pub kind: Kind,
pub required: bool,
/// `Optional[...] = None`
pub nullable: bool,
pub default: Default,
pub ge: Option<i64>,
pub le: Option<i64>,
pub description: Option<&'static str>,
}
impl Param {
const fn new(name: &'static str, loc: In, kind: Kind) -> Self {
Param {
name,
loc,
kind,
required: matches!(loc, In::Path),
nullable: false,
default: Default::None,
ge: None,
le: None,
description: None,
}
}
pub const fn path_str(name: &'static str) -> Self {
Self::new(name, In::Path, Kind::Str)
}
pub const fn path_int(name: &'static str) -> Self {
Self::new(name, In::Path, Kind::Int)
}
pub const fn int(name: &'static str, default: i64) -> Self {
Param {
default: Default::Int(default),
..Self::new(name, In::Query, Kind::Int)
}
}
pub const fn opt_int(name: &'static str) -> Self {
Param {
nullable: true,
..Self::new(name, In::Query, Kind::Int)
}
}
pub const fn str(name: &'static str, default: &'static str) -> Self {
Param {
default: Default::Str(default),
..Self::new(name, In::Query, Kind::Str)
}
}
pub const fn opt_date(name: &'static str, description: &'static str) -> Self {
Param {
nullable: true,
description: Some(description),
..Self::new(name, In::Query, Kind::Date)
}
}
/// The optional `X-API-Key` header of the admin endpoints.
pub const fn api_key() -> Self {
Param {
nullable: true,
..Self::new("X-API-Key", In::Header, Kind::Str)
}
}
pub const fn ge(self, v: i64) -> Self {
Param {
ge: Some(v),
..self
}
}
pub const fn le(self, v: i64) -> Self {
Param {
le: Some(v),
..self
}
}
fn loc_name(&self) -> &'static str {
match self.loc {
In::Query => "query",
In::Path => "path",
In::Header => "header",
}
}
/// pydantic's default title: `snake_case` / `X-API-Key` -> `Snake Case` / `X-Api-Key`
pub fn title(&self) -> String {
python_title(&self.name.replace('_', " "))
}
/// The OpenAPI parameter object, as FastAPI writes it.
pub fn openapi(&self) -> Value {
let mut base = Map::new();
base.insert(
"type".into(),
json!(match self.kind {
Kind::Int => "integer",
Kind::Str | Kind::Date => "string",
}),
);
if self.kind == Kind::Date {
base.insert("format".into(), json!("date"));
}
if let Some(le) = self.le {
base.insert("maximum".into(), json!(le));
}
if let Some(ge) = self.ge {
base.insert("minimum".into(), json!(ge));
}
let mut schema = if self.nullable {
let mut m = Map::new();
m.insert(
"anyOf".into(),
json!([Value::Object(base), {"type": "null"}]),
);
m
} else {
base
};
match self.default {
Default::Int(d) => {
schema.insert("default".into(), json!(d));
}
Default::Str(d) => {
schema.insert("default".into(), json!(d));
}
Default::None => {}
}
if let Some(d) = self.description {
schema.insert("description".into(), json!(d));
}
schema.insert("title".into(), json!(self.title()));
let mut p = Map::new();
p.insert("name".into(), json!(self.name));
p.insert("in".into(), json!(self.loc_name()));
p.insert("required".into(), json!(self.required));
p.insert("schema".into(), Value::Object(schema));
if let Some(d) = self.description {
p.insert("description".into(), json!(d));
}
Value::Object(p)
}
}
/// Python's `str.title()` for ASCII.
pub fn python_title(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut prev_letter = false;
for c in s.chars() {
out.extend(if prev_letter {
c.to_lowercase().collect::<Vec<_>>()
} else {
c.to_uppercase().collect::<Vec<_>>()
});
prev_letter = c.is_alphabetic();
}
out
}
#[derive(Clone, Debug, PartialEq)]
pub enum Val {
Int(i64),
Str(String),
Date(NaiveDate),
None,
}
#[derive(Debug, Default)]
pub struct Args(HashMap<&'static str, Val>);
impl Args {
pub fn int(&self, name: &str) -> i64 {
match self.0.get(name) {
Some(Val::Int(v)) => *v,
other => panic!("parameter {name}: expected an int, got {other:?}"),
}
}
pub fn opt_int(&self, name: &str) -> Option<i64> {
match self.0.get(name) {
Some(Val::Int(v)) => Some(*v),
_ => None,
}
}
pub fn str(&self, name: &str) -> &str {
match self.0.get(name) {
Some(Val::Str(v)) => v,
other => panic!("parameter {name}: expected a string, got {other:?}"),
}
}
pub fn date(&self, name: &str) -> Option<NaiveDate> {
match self.0.get(name) {
Some(Val::Date(d)) => Some(*d),
_ => None,
}
}
/// Python's `str()` of the value (cache keys built like the Python app's).
pub fn py_str(&self, name: &str) -> String {
match self.0.get(name) {
Some(Val::Int(v)) => v.to_string(),
Some(Val::Str(s)) => s.clone(),
Some(Val::Date(d)) => d.format("%Y-%m-%d").to_string(),
Some(Val::None) | None => "None".into(),
}
}
}
/// The query string as a map; a repeated name keeps its last value (as
/// Starlette's `QueryParams.get`).
pub fn query_map(raw: Option<&str>) -> HashMap<String, String> {
form_urlencoded::parse(raw.unwrap_or("").as_bytes())
.map(|(k, v)| (k.into_owned(), v.into_owned()))
.collect()
}
fn error(kind: &str, p: &Param, msg: String, input: Value, ctx: Option<Value>) -> Value {
let mut e = Map::new();
e.insert("type".into(), json!(kind));
e.insert("loc".into(), json!([p.loc_name(), p.name]));
e.insert("msg".into(), json!(msg));
e.insert("input".into(), input);
if let Some(c) = ctx {
e.insert("ctx".into(), c);
}
Value::Object(e)
}
fn parse_date(s: &str) -> Result<NaiveDate, &'static str> {
if s.len() < 10 {
return Err("input is too short");
}
if let Ok(d) = NaiveDate::parse_from_str(s, "%Y-%m-%d") {
return Ok(d);
}
// pydantic accepts a datetime at midnight for a date
for fmt in [
"%Y-%m-%dT%H:%M:%S%.f",
"%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S",
] {
if let Ok(t) = NaiveDateTime::parse_from_str(s, fmt) {
if t.time() == chrono::NaiveTime::MIN {
return Ok(t.date());
}
return Err("Datetimes provided to dates should have zero time - e.g. be exact dates");
}
}
Err("invalid date format, expected YYYY-MM-DD")
}
/// Validate every declared parameter; all problems are reported at once.
pub fn parse(
spec: &[Param],
query: &HashMap<String, String>,
path: &[(&str, &str)],
headers: &axum::http::HeaderMap,
) -> Result<Args, ApiError> {
let mut args = Args::default();
let mut errors = vec![];
for p in spec {
let raw: Option<String> = match p.loc {
In::Query => query.get(p.name).cloned(),
In::Path => path
.iter()
.find(|(k, _)| *k == p.name)
.map(|(_, v)| v.to_string()),
In::Header => headers
.get(p.name)
.and_then(|v| v.to_str().ok())
.map(String::from),
};
let Some(raw) = raw else {
if p.required {
errors.push(error(
"missing",
p,
"Field required".into(),
Value::Null,
None,
));
continue;
}
let v = match p.default {
Default::Int(d) => Val::Int(d),
Default::Str(d) => Val::Str(d.into()),
Default::None => Val::None,
};
args.0.insert(p.name, v);
continue;
};
let v = match p.kind {
Kind::Str => Val::Str(raw),
Kind::Int => match raw.trim().parse::<i64>() {
Err(_) => {
errors.push(error(
"int_parsing",
p,
"Input should be a valid integer, unable to parse string as an integer"
.into(),
json!(raw),
None,
));
continue;
}
Ok(v) => {
if let Some(ge) = p.ge
&& v < ge
{
errors.push(error(
"greater_than_equal",
p,
format!("Input should be greater than or equal to {ge}"),
json!(raw),
Some(json!({ "ge": ge })),
));
continue;
}
if let Some(le) = p.le
&& v > le
{
errors.push(error(
"less_than_equal",
p,
format!("Input should be less than or equal to {le}"),
json!(raw),
Some(json!({ "le": le })),
));
continue;
}
Val::Int(v)
}
},
Kind::Date => match parse_date(&raw) {
Ok(d) => Val::Date(d),
Err(why) => {
errors.push(error(
"date_from_datetime_parsing",
p,
format!("Input should be a valid date or datetime, {why}"),
json!(raw),
Some(json!({ "error": why })),
));
continue;
}
},
};
args.0.insert(p.name, v);
}
if errors.is_empty() {
Ok(args)
} else {
Err(ApiError::validation(errors))
}
}
#[cfg(test)]
mod tests {
use super::*;
const SPEC: &[Param] = &[
Param::path_str("station_code"),
Param::int("hours", 168).ge(1),
Param::int("limit", 50000).ge(1).le(100000),
Param::opt_date("start", "First day (overrides hours)"),
Param::opt_int("horizon").ge(1).le(48),
];
fn run(q: &str) -> Result<Args, ApiError> {
parse(
SPEC,
&query_map(Some(q)),
&[("station_code", "P.1")],
&axum::http::HeaderMap::new(),
)
}
#[test]
fn defaults_and_values() {
let a = run("hours=6&start=2026-10-15&hours=12").unwrap();
assert_eq!(a.int("hours"), 12); // last wins
assert_eq!(a.int("limit"), 50000);
assert_eq!(a.str("station_code"), "P.1");
assert_eq!(a.date("start"), NaiveDate::from_ymd_opt(2026, 10, 15));
assert_eq!(a.opt_int("horizon"), None);
assert_eq!(a.py_str("horizon"), "None");
}
#[test]
fn errors_like_fastapi() {
// FastAPI: {"detail":[{"type":"greater_than_equal","loc":["query","hours"],
// "msg":"Input should be greater than or equal to 1","input":"0","ctx":{"ge":1}}]}
let e = run("hours=0&limit=x").unwrap_err();
assert_eq!(e.status.as_u16(), 422);
assert_eq!(
e.detail,
json!([
{"type": "greater_than_equal", "loc": ["query", "hours"],
"msg": "Input should be greater than or equal to 1", "input": "0", "ctx": {"ge": 1}},
{"type": "int_parsing", "loc": ["query", "limit"],
"msg": "Input should be a valid integer, unable to parse string as an integer",
"input": "x"}
])
);
}
#[test]
fn openapi_parameter_objects() {
assert_eq!(
SPEC[2].openapi().to_string(),
r#"{"name":"limit","in":"query","required":false,"schema":{"type":"integer","maximum":100000,"minimum":1,"default":50000,"title":"Limit"}}"#
);
assert_eq!(
SPEC[4].openapi().to_string(),
r#"{"name":"horizon","in":"query","required":false,"schema":{"anyOf":[{"type":"integer","maximum":48,"minimum":1},{"type":"null"}],"title":"Horizon"}}"#
);
assert_eq!(Param::api_key().title(), "X-Api-Key");
assert_eq!(python_title("station code"), "Station Code");
}
}
rust/collector/src/web/pyfmt.rs
+158
View File
@@ -0,0 +1,158 @@
//! Output conventions of the Python app that the API keeps: `isoformat()`,
//! `round()`, numpy's summation order and Python's `str()` of JSON scalars.
use chrono::{NaiveDate, NaiveDateTime, Timelike};
use serde_json::Value;
/// `datetime.isoformat()`: microseconds only when they are not zero.
pub fn isoformat(t: NaiveDateTime) -> String {
let base = t.format("%Y-%m-%dT%H:%M:%S");
match t.nanosecond() / 1000 {
0 => base.to_string(),
us => format!("{base}.{us:06}"),
}
}
/// `str(datetime)`: as `isoformat()` with a space instead of the `T`.
pub fn py_str_datetime(t: NaiveDateTime) -> String {
isoformat(t).replacen('T', " ", 1)
}
/// `datetime.combine(d, datetime.max.time())`
pub fn end_of_day(d: NaiveDate) -> NaiveDateTime {
d.and_hms_micro_opt(23, 59, 59, 999_999)
.expect("valid time")
}
pub fn start_of_day(d: NaiveDate) -> NaiveDateTime {
d.and_hms_opt(0, 0, 0).expect("valid time")
}
/// `round(x, ndigits)` for floats: the correctly rounded decimal, ties to
/// even (Rust's fixed-precision formatting rounds the same way).
pub fn py_round(x: f64, ndigits: usize) -> f64 {
if !x.is_finite() {
return x;
}
format!("{x:.ndigits$}")
.parse()
.expect("formatted float parses")
}
/// numpy's `np.add.reduce` on a contiguous float64 array (pairwise
/// summation, 8 accumulators per block of up to 128), which pandas'
/// `Series.mean()` uses without bottleneck.
pub fn numpy_sum(a: &[f64]) -> f64 {
0.0 + pairwise(a)
}
fn pairwise(a: &[f64]) -> f64 {
let n = a.len();
if n < 8 {
let mut res = -0.0;
for x in a {
res += x;
}
res
} else if n <= 128 {
let mut r = [0.0; 8];
r.copy_from_slice(&a[..8]);
let mut i = 8;
while i < n - (n % 8) {
for j in 0..8 {
r[j] += a[i + j];
}
i += 8;
}
let mut res = ((r[0] + r[1]) + (r[2] + r[3])) + ((r[4] + r[5]) + (r[6] + r[7]));
while i < n {
res += a[i];
i += 1;
}
res
} else {
let mut n2 = n / 2;
n2 -= n2 % 8;
pairwise(&a[..n2]) + pairwise(&a[n2..])
}
}
/// `Series.mean()` (NaN-free input).
pub fn numpy_mean(a: &[f64]) -> f64 {
numpy_sum(a) / a.len() as f64
}
/// `str(x)` of a JSON scalar as Python would print the decoded value.
pub fn py_str(v: &Value) -> String {
match v {
Value::Null => "None".into(),
Value::Bool(true) => "True".into(),
Value::Bool(false) => "False".into(),
Value::String(s) => s.clone(),
other => other.to_string(),
}
}
/// A float as JSON (`NaN` and infinities, which Python refuses to encode,
/// become null).
pub fn num(x: Option<f64>) -> Value {
x.filter(|x| x.is_finite()).map_or(Value::Null, Value::from)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn isoformat_drops_zero_microseconds() {
let d = NaiveDate::from_ymd_opt(2026, 10, 18).unwrap();
assert_eq!(
isoformat(d.and_hms_opt(14, 0, 0).unwrap()),
"2026-10-18T14:00:00"
);
assert_eq!(
isoformat(d.and_hms_micro_opt(15, 30, 38, 454_399).unwrap()),
"2026-10-18T15:30:38.454399"
);
assert_eq!(isoformat(end_of_day(d)), "2026-10-18T23:59:59.999999");
assert_eq!(
py_str_datetime(d.and_hms_opt(21, 0, 0).unwrap()),
"2026-10-18 21:00:00"
);
}
#[test]
fn rounding_matches_python() {
// values from CPython 3.11: round(x, n)
for (x, n, want) in [
(0.125, 2, 0.12),
(0.375, 2, 0.38),
(2.675, 2, 2.67),
(1.0005, 3, 1.0),
(0.0875, 3, 0.087),
(-0.0049, 2, -0.0),
(93.35, 1, 93.3),
(24.6, 2, 24.6),
(1.2345e-7, 3, 0.0),
] {
assert_eq!(py_round(x, n), want, "round({x}, {n})");
}
}
#[test]
fn numpy_pairwise_sum_order() {
// np.add.reduce / pd.Series.mean (numpy 1.26), where a naive loop
// gives 20.000000000000014, 1027.6428571428573, 1481.0200000000002
assert_eq!(numpy_sum(&[0.1; 200]), 20.0);
let series = |n: usize| -> Vec<f64> {
(0..n)
.map(|i| ((i * 7919) % 1000) as f64 / 7.0 * 0.013 + 0.1)
.collect()
};
assert_eq!(numpy_sum(&series(1000)), 1027.642857142857);
assert_eq!(numpy_sum(&series(1441)), 1481.02);
assert_eq!(numpy_mean(&series(1441)), 1.0277723802914642);
assert_eq!(numpy_sum(&series(300)), 308.84999999999997);
assert_eq!(numpy_sum(&[]), 0.0);
}
}
rust/collector/src/web/system.rs
+635
View File
@@ -0,0 +1,635 @@
//! The dashboard, health, metrics, collector status and the admin endpoints.
use super::Shared;
use super::error::{ApiError, ApiResult, ok, ok_json, to_body};
use super::params::{self, Param};
use super::pyfmt::{isoformat, py_str_datetime};
use crate::stations::StationInfo;
use crate::util::bangkok_now;
use axum::body::Bytes;
use axum::extract::{Path, State};
use axum::http::{HeaderMap, StatusCode, header};
use axum::response::{IntoResponse, Response};
use serde_json::{Map, Value, json};
use std::time::{Duration, Instant};
/// GET /: the dashboard (src/static/dashboard.html, loaded at start).
pub async fn root(State(st): State<Shared>) -> Response {
(
[(header::CONTENT_TYPE, "text/html; charset=utf-8")],
st.dashboard.clone(),
)
.into_response()
}
async fn static_file(st: &Shared, name: &str, content_type: &'static str) -> Response {
match tokio::fs::read(st.cfg.web.static_dir.join(name)).await {
Ok(bytes) => ([(header::CONTENT_TYPE, content_type)], bytes).into_response(),
Err(_) => ApiError::new(404, "Not Found").into_response(),
}
}
/// Crawler/indexing files at the domain root.
pub async fn robots(State(st): State<Shared>) -> Response {
static_file(&st, "robots.txt", "text/plain; charset=utf-8").await
}
pub async fn llms(State(st): State<Shared>) -> Response {
static_file(&st, "llms.txt", "text/plain; charset=utf-8").await
}
pub async fn sitemap(State(st): State<Shared>) -> Response {
static_file(&st, "sitemap.xml", "application/xml").await
}
// ---------------------------------------------------------------- /health
struct Check {
status: &'static str,
message: String,
}
fn check(status: &'static str, message: impl Into<String>) -> Check {
Check {
status,
message: message.into(),
}
}
/// Data freshness. As the Python check it judges the newest reading of the
/// first station (`get_latest_measurements(limit=1)`: P.1) against 2 h. One
/// change, as in the notifier: while RID's report lags (2-10 h at night) but
/// the ThaiWater telemetry of the same gauges is current, the data is current.
async fn database_check(st: &Shared) -> Check {
let latest: Result<Option<(chrono::NaiveDateTime,)>, _> = sqlx::query_as(
"SELECT m.timestamp
FROM water_measurements m
JOIN stations s ON m.station_id = s.id
INNER JOIN (
SELECT station_id, MAX(timestamp) as max_timestamp
FROM water_measurements GROUP BY station_id
) latest ON m.station_id = latest.station_id AND m.timestamp = latest.max_timestamp
ORDER BY s.station_code LIMIT 1",
)
.fetch_optional(&st.pool)
.await;
let latest = match latest {
Ok(l) => l.map(|(t,)| t),
Err(e) => return check("unhealthy", format!("Database check failed: {e}")),
};
let now = bangkok_now();
let max_age = chrono::TimeDelta::hours(2);
if let Some(ts) = latest
&& now - ts > max_age
{
let codes = &ping_forecast::config::config().hii_fill_stations;
if let Ok(Some(hii)) = crate::db::latest_hii_timestamp(&st.pool, codes).await
&& now - hii <= max_age
{
return check(
"healthy",
format!(
"Database connection OK; RID report lags (latest {}), ThaiWater telemetry current (latest {})",
py_str_datetime(ts),
py_str_datetime(hii)
),
);
}
return check(
"degraded",
format!("Latest data is old: {}", py_str_datetime(ts)),
);
}
check("healthy", "Database connection OK")
}
/// The RID API answers (5 s, a liveness probe must fail fast).
async fn api_check(st: &Shared) -> Check {
match st
.http
.get(&st.cfg.rid_api_url)
.timeout(Duration::from_secs(5))
.send()
.await
{
Ok(r) if r.status() == StatusCode::OK => check("healthy", "API connection OK"),
Ok(r) => check(
"degraded",
format!("API returned status {}", r.status().as_u16()),
),
Err(e) => check("unhealthy", format!("API check failed: {e}")),
}
}
/// Resident memory of this process in MB (Linux).
pub fn rss_mb() -> Option<f64> {
let status = std::fs::read_to_string("/proc/self/status").ok()?;
let kb: f64 = status
.lines()
.find(|l| l.starts_with("VmRSS:"))?
.split_whitespace()
.nth(1)?
.parse()
.ok()?;
Some(kb / 1024.0)
}
async fn memory_check() -> Check {
match rss_mb() {
Some(mb) if mb > 1000.0 => check("degraded", format!("High memory usage: {mb:.1}MB")),
Some(mb) => check("healthy", format!("Memory usage OK: {mb:.1}MB")),
None => check("healthy", "Memory check skipped (no /proc/self/status)"),
}
}
async fn timed<F: std::future::Future<Output = Check>>(
f: F,
) -> (Check, f64, chrono::NaiveDateTime) {
let t = Instant::now();
let c = f.await;
(c, t.elapsed().as_secs_f64() * 1000.0, bangkok_now())
}
async fn compute_health(st: Shared) -> super::cache::Computed<Bytes> {
let results = [
("database", timed(database_check(&st)).await),
("api", timed(api_check(&st)).await),
("memory", timed(memory_check()).await),
];
let statuses: Vec<&str> = results.iter().map(|(_, (c, _, _))| c.status).collect();
let overall = if statuses.contains(&"unhealthy") {
"unhealthy"
} else if statuses.contains(&"degraded") {
"degraded"
} else {
"healthy"
};
let mut checks = Map::new();
for (name, (c, ms, at)) in results {
checks.insert(
name.into(),
json!({
"status": c.status,
"message": c.message,
"response_time_ms": ms,
"timestamp": isoformat(at),
}),
);
}
let body = json!({
"overall_status": overall,
"timestamp": isoformat(bangkok_now()),
"checks": checks,
});
Ok((to_body(&body), true))
}
/// GET /health: cached briefly (it probes the RID API), never older than 4 TTLs.
pub async fn health(State(st): State<Shared>) -> ApiResult {
st.metrics.count("health");
let ttl = st.cfg.web.health_cache_ttl_s;
let me = st.clone();
let (body, _) = st
.caches
.health
.get(
"health".into(),
Duration::from_secs(ttl),
Some(Duration::from_secs(4 * ttl)),
move || compute_health(me.clone()),
)
.await?;
ok(body, false)
}
/// GET /metrics
pub async fn metrics(State(st): State<Shared>) -> ApiResult {
st.metrics.count("metrics");
let mut gauges = Map::new();
gauges.insert(
"uptime_seconds".into(),
json!(st.started.elapsed().as_secs_f64()),
);
if let Some(mb) = rss_mb() {
gauges.insert("memory_rss_mb".into(), json!(mb));
}
gauges.insert("db_pool_connections".into(), json!(st.pool.size() as f64));
ok_json(&json!({
"counters": st.metrics.counters(),
"gauges": gauges,
"histograms": {},
}))
}
// -------------------------------------------------------------- stations
fn station_json(id: u32, s: &StationInfo, geohash: bool, status: &str) -> Value {
json!({
"station_id": id,
"station_code": s.code,
"thai_name": s.thai_name,
"english_name": s.english_name,
"latitude": s.latitude,
"longitude": s.longitude,
"geohash": if geohash { json!(s.geohash) } else { Value::Null },
"status": status,
})
}
/// GET /stations: the RID gauges (report column -> station).
pub async fn stations(State(st): State<Shared>) -> ApiResult {
st.metrics.count("stations");
let list: Vec<Value> = st
.stations
.read()
.unwrap()
.iter()
.map(|(id, s)| station_json(*id, s, false, "active"))
.collect();
ok_json(&list)
}
pub const STATION_ID: &[Param] = &[Param::path_int("station_id")];
pub const ADMIN_STATION_ID: &[Param] = &[Param::path_int("station_id"), Param::api_key()];
pub const ADMIN: &[Param] = &[Param::api_key()];
fn station_id(id: &str, h: &HeaderMap) -> Result<u32, ApiError> {
let a = params::parse(STATION_ID, &Default::default(), &[("station_id", id)], h)?;
// ids beyond u32 cannot exist
Ok(u32::try_from(a.int("station_id")).unwrap_or(u32::MAX))
}
fn not_found() -> ApiError {
ApiError::new(404, "Station not found")
}
/// GET /stations/{station_id}
pub async fn station(State(st): State<Shared>, Path(id): Path<String>, h: HeaderMap) -> ApiResult {
st.metrics.count("get_station");
let id = station_id(&id, &h)?;
let stations = st.stations.read().unwrap();
let s = stations.info(id).ok_or_else(not_found)?;
ok_json(&station_json(id, s, true, "active"))
}
/// `require_admin_key`: the X-API-Key header must match ADMIN_API_KEY;
/// with no key configured the admin endpoints are disabled.
fn require_admin(st: &Shared, h: &HeaderMap) -> Result<(), ApiError> {
let Some(want) = &st.cfg.web.admin_api_key else {
return Err(ApiError::new(
503,
"Admin API disabled: ADMIN_API_KEY is not configured on the server",
));
};
let given = h.get("x-api-key").map(|v| v.as_bytes()).unwrap_or_default();
// constant time, as secrets.compare_digest
let same = given.len() == want.len()
&& given
.iter()
.zip(want.as_bytes())
.fold(0u8, |acc, (a, b)| acc | (a ^ b))
== 0;
if same {
Ok(())
} else {
Err(ApiError::new(401, "Invalid or missing X-API-Key"))
}
}
fn body_object(body: &Bytes) -> Result<Map<String, Value>, ApiError> {
match serde_json::from_slice::<Value>(body) {
Ok(Value::Object(m)) => Ok(m),
Ok(other) => Err(ApiError::validation(vec![json!({
"type": "model_attributes_type", "loc": ["body"],
"msg": "Input should be a valid dictionary or object to extract fields from",
"input": other,
})])),
Err(e) => Err(ApiError::validation(vec![json!({
"type": "json_invalid", "loc": ["body", e.column()],
"msg": "JSON decode error", "input": {}, "ctx": {"error": e.to_string()},
})])),
}
}
/// Field checks of StationCreateModel / StationUpdateModel.
struct Fields<'a> {
body: &'a Map<String, Value>,
errors: Vec<Value>,
}
impl<'a> Fields<'a> {
fn err(&mut self, kind: &str, field: &str, msg: String, ctx: Option<Value>) {
let mut e = json!({
"type": kind, "loc": ["body", field], "msg": msg,
"input": self.body.get(field).cloned().unwrap_or(Value::Object(self.body.clone())),
});
if let Some(c) = ctx {
e["ctx"] = c;
}
self.errors.push(e);
}
fn string(&mut self, field: &str, required: bool) -> Option<String> {
match self.body.get(field) {
None if required => {
self.err("missing", field, "Field required".into(), None);
None
}
None | Some(Value::Null) if !required => None,
Some(Value::String(s)) => Some(s.clone()),
_ => {
self.err(
"string_type",
field,
"Input should be a valid string".into(),
None,
);
None
}
}
}
fn coord(&mut self, field: &str, limit: f64) -> Option<f64> {
let v = match self.body.get(field) {
None | Some(Value::Null) => return None,
Some(Value::Number(n)) => n.as_f64(),
Some(Value::String(s)) => s.trim().parse().ok(),
_ => None,
};
let Some(x) = v else {
self.err(
"float_parsing",
field,
"Input should be a valid number, unable to parse string as a number".into(),
None,
);
return None;
};
if x < -limit {
self.err(
"greater_than_equal",
field,
format!("Input should be greater than or equal to {}", -limit),
Some(json!({"ge": -limit})),
);
} else if x > limit {
self.err(
"less_than_equal",
field,
format!("Input should be less than or equal to {limit}"),
Some(json!({"le": limit})),
);
}
Some(x)
}
fn finish(self) -> Result<(), ApiError> {
if self.errors.is_empty() {
Ok(())
} else {
Err(ApiError::validation(self.errors))
}
}
}
fn persist(st: &Shared) -> bool {
let stations = st.stations.read().unwrap();
match stations.save(&st.cfg.station_config_path) {
Ok(()) => true,
Err(e) => {
tracing::error!(
"Failed to persist station mapping to {}: {e:#}",
st.cfg.station_config_path.display()
);
false
}
}
}
/// POST /stations (admin)
pub async fn create_station(State(st): State<Shared>, h: HeaderMap, body: Bytes) -> ApiResult {
st.metrics.count("create_station");
require_admin(&st, &h)?;
let body = body_object(&body)?;
let mut f = Fields {
body: &body,
errors: vec![],
};
let code = f.string("station_code", true);
let thai = f.string("thai_name", true);
let english = f.string("english_name", true);
let latitude = f.coord("latitude", 90.0);
let longitude = f.coord("longitude", 180.0);
let geohash = f.string("geohash", false);
let status = f.string("status", false).unwrap_or_else(|| "active".into());
f.finish()?;
let info = StationInfo {
code: code.unwrap_or_default(),
thai_name: thai.unwrap_or_default(),
english_name: english.unwrap_or_default(),
latitude,
longitude,
geohash,
};
let id = {
let mut stations = st.stations.write().unwrap();
let id = stations.next_id();
stations.insert(id, info.clone());
id
};
if !persist(&st) {
st.stations.write().unwrap().remove(id);
return Err(ApiError::new(500, "Failed to persist new station"));
}
tracing::info!("Created new station: {} ({})", info.code, info.english_name);
ok_json(&station_json(id, &info, true, &status))
}
/// PUT /stations/{station_id} (admin)
pub async fn update_station(
State(st): State<Shared>,
Path(id): Path<String>,
h: HeaderMap,
body: Bytes,
) -> ApiResult {
st.metrics.count("update_station");
require_admin(&st, &h)?;
let id = station_id(&id, &h)?;
let body = body_object(&body)?;
let mut f = Fields {
body: &body,
errors: vec![],
};
let thai = f.string("thai_name", false);
let english = f.string("english_name", false);
let latitude = f.coord("latitude", 90.0);
let longitude = f.coord("longitude", 180.0);
let geohash = f.string("geohash", false);
let status = f.string("status", false);
f.finish()?;
let (original, updated) = {
let mut stations = st.stations.write().unwrap();
let s = stations.info_mut(id).ok_or_else(not_found)?;
let original = s.clone();
if let Some(v) = thai {
s.thai_name = v;
}
if let Some(v) = english {
s.english_name = v;
}
if latitude.is_some() {
s.latitude = latitude;
}
if longitude.is_some() {
s.longitude = longitude;
}
if geohash.is_some() {
s.geohash = geohash;
}
(original, s.clone())
};
if !persist(&st) {
st.stations.write().unwrap().insert(id, original);
return Err(ApiError::new(500, "Failed to persist station update"));
}
tracing::info!("Updated station {id}: {}", updated.code);
ok_json(&station_json(
id,
&updated,
true,
status
.as_deref()
.filter(|s| !s.is_empty())
.unwrap_or("active"),
))
}
/// DELETE /stations/{station_id} (admin)
pub async fn delete_station(
State(st): State<Shared>,
Path(id): Path<String>,
h: HeaderMap,
) -> ApiResult {
st.metrics.count("delete_station");
require_admin(&st, &h)?;
let id = station_id(&id, &h)?;
let removed = st
.stations
.write()
.unwrap()
.remove(id)
.ok_or_else(not_found)?;
if !persist(&st) {
st.stations.write().unwrap().insert(id, removed);
return Err(ApiError::new(500, "Failed to persist station deletion"));
}
tracing::info!("Deleted station {id}: {}", removed.code);
ok_json(&json!({"message": format!("Station {} deleted successfully", removed.code)}))
}
// ------------------------------------------------------------ collection
/// GET /scraping/status: the collector's last cycle (in this process, or
/// SCRAPING_STATUS_FILE when serving alone).
pub async fn scraping_status(State(st): State<Shared>) -> ApiResult {
st.metrics.count("scraping_status");
let status: Value = match &st.status {
Some(s) => s.read().unwrap().clone(),
None => st
.cfg
.status_file
.as_ref()
.and_then(|p| std::fs::read_to_string(p).ok())
.and_then(|t| serde_json::from_str(&t).ok())
.unwrap_or_else(|| json!({"is_running": false})),
};
// ScrapingStatusResponse's fields and defaults first, then what the
// collector adds (version, forecast run)
let defaults = [
("is_running", json!(false)),
("last_run", Value::Null),
("next_run", Value::Null),
("total_runs", json!(0)),
("successful_runs", json!(0)),
("failed_runs", json!(0)),
("feed_current", Value::Null),
("latest_reading", Value::Null),
("readings_saved", json!(0)),
];
let mut out = Map::new();
for (k, d) in defaults {
out.insert(k.into(), status.get(k).cloned().unwrap_or(d));
}
for (k, v) in status.as_object().into_iter().flatten() {
if !out.contains_key(k) {
out.insert(k.clone(), v.clone());
}
}
ok_json(&Value::Object(out))
}
/// POST /scrape/trigger (admin): poll every source now.
pub async fn trigger(State(st): State<Shared>, h: HeaderMap) -> ApiResult {
st.metrics.count("scrape_trigger");
require_admin(&st, &h)?;
let Some(trigger) = &st.trigger else {
return Err(ApiError::new(
409,
"Collection does not run in this process (ping-collector serve)",
));
};
let running = st
.status
.as_ref()
.is_some_and(|s| s.read().unwrap()["is_running"] == true);
if running {
return Err(ApiError::new(409, "Scraping already in progress"));
}
trigger.notify_one();
ok_json(&json!({"message": "Scraping triggered", "status": "started"}))
}
/// GET /config (admin): the settings, secrets masked.
pub async fn config(State(st): State<Shared>, h: HeaderMap) -> ApiResult {
st.metrics.count("config");
require_admin(&st, &h)?;
let c = &st.cfg;
let w = &c.web;
let mut out = json!({
"DB_TYPE": "postgresql",
"DATABASE_URL": c.redacted_database_url(),
"API_URL": c.rid_api_url,
"REQUEST_TIMEOUT": w.request_timeout_s,
"RID_POLL_OFFSET_MINUTES": c.poll_offset_minutes,
"RID_RETRY_MINUTES": c.retry_minutes,
"GAP_FILL_DAYS_ON_START": c.gap_fill_days_on_start,
"ENABLE_HII_COLLECTION": c.hii_enabled,
"ENABLE_RESERVOIR_COLLECTION": c.dams_enabled,
"ENABLE_RAIN_COLLECTION": c.rain_enabled,
"FORECAST_IN_COLLECTOR": c.forecast_enabled,
"MODELS_DIR": c.models_dir.display().to_string(),
"STATION_CONFIG_PATH": c.station_config_path.display().to_string(),
"SCRAPING_STATUS_FILE": c.status_file.as_ref().map(|p| p.display().to_string()),
"NTFY_SERVER": c.ntfy_server,
"NTFY_TOPIC_PREFIX": c.ntfy_topic_prefix,
"MATRIX_ENABLED": c.matrix.is_some(),
"WEB_LISTEN": w.listen,
"STATIC_DIR": w.static_dir.display().to_string(),
"CORS_ALLOW_ORIGINS": w.cors_allow_origins,
"HII_CACHE_TTL_SECONDS": w.hii_cache_ttl_s,
"LATEST_CACHE_TTL_SECONDS": w.latest_cache_ttl_s,
"HEALTH_CACHE_TTL_SECONDS": w.health_cache_ttl_s,
"FORECAST_MAX_AGE_SECONDS": w.forecast_max_age_s,
"UMAMI_TRACK_API": w.umami_track_api,
"collector": concat!("ping-collector ", env!("CARGO_PKG_VERSION")),
});
if let Some(m) = out.as_object_mut() {
for (k, v) in m.iter_mut() {
let k = k.to_lowercase();
if (k.contains("password") || k.contains("secret")) && !v.is_null() {
*v = json!("********");
}
}
}
ok_json(&out)
}
rust/collector/src/web/tests.rs
+293
View File
@@ -0,0 +1,293 @@
//! Route tests: the router as the server runs it, against a throwaway
//! schema (TEST_DATABASE_URL; see db_tests.rs) or, for the routes that do
//! not touch the database, a pool that never connects. Output formats are
//! the FastAPI app's (scripts/api_parity.py compares the two live).
use super::{AppState, router};
use crate::config::Config;
use crate::db_tests::TestDb;
use crate::stations::Stations;
use axum::Router;
use axum::body::Body;
use axum::http::{HeaderMap, Request, StatusCode};
use serde_json::{Value, json};
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
use tower::ServiceExt;
fn config(extra: &[(&str, &str)]) -> Config {
let static_dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../../src/static");
let mut vars: HashMap<String, String> = [
(
"POSTGRES_CONNECTION_STRING",
"postgres://test@localhost:1/none",
),
("UMAMI_TRACK_API", "false"),
("STATIC_DIR", static_dir),
("NTFY_SERVER", "https://ntfy.example"),
("STATION_CONFIG_PATH", "/nonexistent/stations.json"),
]
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect();
for (k, v) in extra {
vars.insert(k.to_string(), v.to_string());
}
Config::from_vars(&|k| vars.get(k).cloned()).unwrap()
}
fn app(cfg: Config, pool: sqlx::PgPool) -> Router {
let state = AppState::new(
cfg,
pool,
reqwest::Client::new(),
Arc::new(RwLock::new(Stations::bundled())),
None,
None,
);
router(state)
}
async fn call(
app: &Router,
method: &str,
uri: &str,
headers: &[(&str, &str)],
) -> (StatusCode, HeaderMap, Vec<u8>) {
let mut req = Request::builder().method(method).uri(uri);
for (k, v) in headers {
req = req.header(*k, *v);
}
let resp = app
.clone()
.oneshot(req.body(Body::empty()).unwrap())
.await
.unwrap();
let (parts, body) = resp.into_parts();
let bytes = axum::body::to_bytes(body, usize::MAX).await.unwrap();
(parts.status, parts.headers, bytes.to_vec())
}
async fn get_json(app: &Router, uri: &str) -> (StatusCode, HeaderMap, Value) {
let (s, h, b) = call(app, "GET", uri, &[]).await;
let v = serde_json::from_slice(&b)
.unwrap_or_else(|e| panic!("{uri}: {e}: {}", String::from_utf8_lossy(&b)));
(s, h, v)
}
fn header<'a>(h: &'a HeaderMap, name: &str) -> Option<&'a str> {
h.get(name).and_then(|v| v.to_str().ok())
}
#[tokio::test]
async fn routes_without_database() {
let pool = sqlx::PgPool::connect_lazy("postgres://test@localhost:1/none").unwrap();
let app = app(config(&[]), pool);
let (s, h, stations) = get_json(&app, "/stations").await;
assert_eq!(s, StatusCode::OK);
assert_eq!(header(&h, "cache-control"), Some("public, max-age=300"));
assert_eq!(header(&h, "content-type"), Some("application/json"));
assert_eq!(stations.as_array().unwrap().len(), 16);
// the list leaves geohash out, the single station has it (as FastAPI)
assert_eq!(stations[7]["station_code"], "P.1");
assert_eq!(stations[7]["geohash"], Value::Null);
let (_, _, p1) = get_json(&app, "/stations/8").await;
assert_eq!(
p1.as_object().unwrap().keys().collect::<Vec<_>>(),
[
"station_id",
"station_code",
"thai_name",
"english_name",
"latitude",
"longitude",
"geohash",
"status"
]
);
let (s, _, e) = get_json(&app, "/stations/abc").await;
assert_eq!(s, StatusCode::UNPROCESSABLE_ENTITY);
assert_eq!(e["detail"][0]["type"], "int_parsing");
assert_eq!(e["detail"][0]["loc"], json!(["path", "station_id"]));
assert_eq!(
get_json(&app, "/stations/99").await.2,
json!({"detail": "Station not found"})
);
assert_eq!(
get_json(&app, "/nope").await.2,
json!({"detail": "Not Found"})
);
let (s, _, b) = call(&app, "DELETE", "/forecast", &[]).await;
assert_eq!(s, StatusCode::METHOD_NOT_ALLOWED);
assert_eq!(b, br#"{"detail":"Method Not Allowed"}"#);
let (s, _, e) = get_json(&app, "/config").await;
assert_eq!(s, StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
e["detail"],
"Admin API disabled: ADMIN_API_KEY is not configured on the server"
);
let (s, _, e) = get_json(&app, "/measurements/history/P.1?hours=0").await;
assert_eq!(s, StatusCode::UNPROCESSABLE_ENTITY);
assert_eq!(e["detail"][0]["ctx"], json!({"ge": 1}));
let (_, _, n) = get_json(&app, "/api/notifications").await;
assert_eq!(n["topics"]["warning"], "ping-warning");
assert_eq!(n["server"], "https://ntfy.example");
let (s, h, spec) = get_json(&app, "/openapi.json").await;
assert_eq!(s, StatusCode::OK);
assert_eq!(header(&h, "content-encoding"), None);
assert!(spec["paths"]["/forecast"]["get"].is_object());
let (s, h, html) = call(&app, "GET", "/", &[("accept-encoding", "gzip")]).await;
assert_eq!(s, StatusCode::OK);
assert_eq!(header(&h, "cache-control"), Some("public, max-age=120"));
assert_eq!(header(&h, "content-encoding"), Some("gzip"));
assert!(html.len() > 100);
let (s, h, _) = call(&app, "GET", "/docs", &[]).await;
assert_eq!(s, StatusCode::OK);
assert_eq!(header(&h, "content-type"), Some("text/html; charset=utf-8"));
let (s, h, _) = call(&app, "GET", "/static/custom-markers.json", &[]).await;
assert_eq!(s, StatusCode::OK);
assert_eq!(header(&h, "cache-control"), Some("public, max-age=3600"));
let (s, h, _) = call(&app, "GET", "/robots.txt", &[]).await;
assert_eq!(s, StatusCode::OK);
assert_eq!(
header(&h, "content-type"),
Some("text/plain; charset=utf-8")
);
assert_eq!(
call(&app, "GET", "/static/nope.json", &[]).await.0,
StatusCode::NOT_FOUND
);
// admin key
let app = super::router(AppState::new(
config(&[("ADMIN_API_KEY", "k3y")]),
sqlx::PgPool::connect_lazy("postgres://test@localhost:1/none").unwrap(),
reqwest::Client::new(),
Arc::new(RwLock::new(Stations::bundled())),
None,
None,
));
assert_eq!(
call(&app, "GET", "/config", &[("x-api-key", "nope")])
.await
.0,
StatusCode::UNAUTHORIZED
);
let (s, _, b) = call(&app, "GET", "/config", &[("x-api-key", "k3y")]).await;
assert_eq!(s, StatusCode::OK, "{}", String::from_utf8_lossy(&b));
assert!(String::from_utf8_lossy(&b).contains("postgres://test:***@localhost:1/none"));
let (s, _, b) = call(&app, "POST", "/scrape/trigger", &[("x-api-key", "k3y")]).await;
assert_eq!(s, StatusCode::CONFLICT, "{}", String::from_utf8_lossy(&b));
}
async fn seed(pool: &sqlx::PgPool) {
for sql in [
"INSERT INTO stations (id, station_code, thai_name, english_name) VALUES
(1, 'P.20', 'บ้านเชียงดาว', 'Ban Chiang Dao'), (8, 'P.1', 'สะพานนวรัฐ', 'Nawarat Bridge')",
// P.1 hourly 2026-10-01 00:00 .. 10-03 23:00, one missing discharge, a hole at 10-02 05:00
"INSERT INTO water_measurements (timestamp, station_id, water_level, discharge, discharge_percent, status)
SELECT t, 8, round((1.2 + extract(hour from t) / 100.0)::numeric, 3),
CASE WHEN t = timestamp '2026-10-02 03:00' THEN NULL ELSE 55.5 END, 12.94, 'active'
FROM generate_series(timestamp '2026-10-01 00:00', timestamp '2026-10-03 23:00', interval '1 hour') t
WHERE t <> timestamp '2026-10-02 05:00'",
"INSERT INTO water_measurements (timestamp, station_id, water_level, discharge, discharge_percent, status)
VALUES (timestamp '2026-10-03 22:00', 1, 0.95, NULL, NULL, 'active')",
"INSERT INTO forecast_history (as_of, station_code, horizon_hours, predicted_max_level, p_warning,
p_danger, current_level, model_version, source)
SELECT t, 'P.1', 6, 1.5, 0.01, 0.0, 1.2, CASE WHEN t < timestamp '2026-10-02 00:00' THEN 'v1' ELSE 'v2' END, 'model'
FROM generate_series(timestamp '2026-10-01 00:00', timestamp '2026-10-03 12:00', interval '1 hour') t",
"INSERT INTO forecast_current (id, computed_at, model_versions, payload)
VALUES ('all', timestamp '2026-10-03 23:10', 'v2', '[{\"station_code\":\"P.1\",\"horizon_hours\":6}]')",
] {
sqlx::query(sql).execute(pool).await.unwrap();
}
}
#[tokio::test]
async fn routes_against_the_database() {
let Some(db) = TestDb::new().await else {
return;
};
seed(&db.pool).await;
let app = app(
config(&[("FORECAST_MAX_AGE_SECONDS", "60")]),
db.pool.clone(),
);
let (s, h, latest) = get_json(&app, "/measurements/latest?limit=5").await;
assert_eq!(s, StatusCode::OK);
assert_eq!(header(&h, "cache-control"), Some("public, max-age=30"));
// ORDER BY station_code: P.1 before P.20
assert_eq!(
latest,
json!([
{"timestamp": "2026-10-03T23:00:00", "station_code": "P.1", "station_name_en": "Nawarat Bridge",
"station_name_th": "สะพานนวรัฐ", "water_level": 1.43, "discharge": 55.5,
"discharge_percent": 12.94, "status": "active"},
{"timestamp": "2026-10-03T22:00:00", "station_code": "P.20", "station_name_en": "Ban Chiang Dao",
"station_name_th": "บ้านเชียงดาว", "water_level": 0.95, "discharge": null,
"discharge_percent": null, "status": "active"}
])
);
let (_, _, h) = get_json(
&app,
"/measurements/history/P.1?start=2026-10-02&end=2026-10-02&limit=6",
)
.await;
let rows = h.as_array().unwrap();
assert_eq!(rows.len(), 6);
assert_eq!(rows[0]["timestamp"], "2026-10-02T00:00:00");
// discharge missing in the database: 20 * H, rounded
assert_eq!(rows[3]["discharge"], json!(24.6));
assert_eq!(rows[5]["timestamp"], "2026-10-02T06:00:00"); // the hole at 05:00
let (s, _, e) = get_json(
&app,
"/measurements/history/P.1?start=2026-10-03&end=2026-10-02",
)
.await;
assert_eq!(s, StatusCode::BAD_GATEWAY);
assert_eq!(e["detail"], "Measurement history unavailable");
let (_, _, stats) = get_json(&app, "/api/stats").await;
assert_eq!(stats["rid_measurements"], 72);
assert_eq!(stats["rid_station_count"], 2);
assert_eq!(stats["first_timestamp"], "2026-10-01T00:00:00");
assert_eq!(stats["days_spanned"], 3);
assert_eq!(stats["coverage_percent"], json!(98.6)); // 71 of 72 hours
let (_, _, fh) = get_json(&app, "/api/forecast/history/P.1?start=2026-10-03&horizon=6").await;
assert_eq!(fh.as_array().unwrap().len(), 13);
assert_eq!(fh[0]["predicted_max_level"], json!(1.5));
let (s, h, f) = get_json(&app, "/forecast").await;
assert_eq!(s, StatusCode::OK);
assert_eq!(f[0]["station_code"], "P.1");
// computed long ago: served, marked
assert_eq!(header(&h, "x-data-stale"), Some("true"));
let (_, _, sk) = get_json(&app, "/api/forecast/skill?station_code=P.1&horizon=6").await;
let versions = sk["versions"].as_array().unwrap();
assert_eq!(versions.len(), 2);
assert_eq!(versions[0]["model_version"], "v1");
assert_eq!(versions[0]["n"], 24);
assert_eq!(versions[0]["enough_data"], true);
assert_eq!(sk["current"]["model_version"], "v2");
assert_eq!(sk["trend"]["previous_version"], "v1");
let (_, _, health) = get_json(&app, "/health").await;
assert_eq!(health["checks"]["database"]["status"], "degraded"); // data from 2026-10-03
assert!(
health["checks"]["database"]["message"]
.as_str()
.unwrap()
.starts_with("Latest data is old: 2026-10-03 23:00:00")
);
db.drop().await;
}