docs: Rust rewrite assessment + forecast parity spike

spikes/rust-forecast ports build_features (P.1 path) and HistGradientBoosting
inference to Rust and checks them against the pandas/sklearn output on two
fixtures (2024 record flood, summer 2026): 244,800 feature cells and every
regression prediction bit-identical, classifier heads to the last ulp;
0.25 ms vs 158 ms per station. docs/RUST_REWRITE.md: what a rewrite would
mean (Rust service, Python training), costs, phased plan, Python alternative.

docs/RUST_REWRITE.md

@@ -0,0 +1,105 @@
+# A Rust rewrite: assessment
+
+Status: evaluation, nothing decided. Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
+
+## Short version
+
+- **Feasible.** The risky part, serving the scikit-learn forecast from Rust with
+  no train/serve skew, works: the spike reproduces the pandas feature matrix
+  and the HistGradientBoosting regression output bit for bit (classifier heads
+  to the last ulp).
+- **Training stays Python.** There is no Rust HGB that trains like
+  scikit-learn, and the evaluation harness (`scripts/evaluate_variants.py`) is
+  where model work happens. A "Rust rewrite" means a Rust service (collectors,
+  API, forecast serving, ntfy) plus a Python offline job (monthly retrain and
+  evaluation) that exports models for Rust.
+- **Performance is not a reason.** The forecast takes 158 ms per station in Python
+  and 0.25 ms in Rust, but it runs once an hour. Load tests showed the site
+  is bandwidth-bound once caches are hit.
+- **Real reasons, if any:** one process instead of leader-elected uvicorn workers
+  (the root of per-worker state bugs like `/scraping/status`), a fraction of the
+  memory (324 MB today), typed parsing and SQL, one deployable binary, and the
+  same stack as proplan (axum + sqlx).
+- **Recommendation:** no big-bang rewrite. If Rust is wanted, start with the
+  collector + notifier daemon (phase 1 below), which stands alone and can be
+  abandoned without loss. The cheap Python alternative is at the end.
+
+## What exists (Python, `src/`)
+
+| Area | LOC | Fate in Rust |
+| --- | ---: | --- |
+| Collectors: RID hourly, HII/ThaiWater, RID dams, Open-Meteo | 2,500 | port; typed serde payloads |
+| Web API (20 routes), schemas, health, metrics | 2,300 | port with identical JSON (golden-file tests) |
+| DB adapters: Postgres, SQLite, MySQL, InfluxDB, VictoriaMetrics | 1,300 | Postgres only via sqlx, existing 13 tables, no data migration |
+| ntfy notifications; Matrix alerting (CLI path) | 1,050 | port ntfy; Matrix only if still used |
+| ML serving: predict, features, rain/dam/HII fill | ~1,300 | port; parity-tested (spike) |
+| ML training + evaluation | ~1,700 | **stays Python** |
+| CLI (`main.py`) | 650 | small clap CLI (`fill-gaps`, test cycle) |
+| `static/dashboard.html` (164 KB) | | unchanged, served as a static file |
+| tests | 3,900 | Rust unit tests + parity fixtures + API golden files |
+
+## Target shape
+
+One binary, one process (tokio):
+
+- tasks: RID / HII / dams / rain collectors on their schedules; forecast
+  precompute when new readings land; ntfy notifier. Shared state is an
+  `Arc<RwLock<..>>`, so there's no leader election, no status hand-off file and
+  no executor sizing.
+- axum + tower-http: same routes and JSON, compression (gzip/brotli also covers
+  the 257 KB river GeoJSON), static dashboard, CORS; `utoipa` if `/docs` stays.
+- sqlx against the existing schema. Python and Rust can run against the same
+  database during migration.
+- models: `scripts/retrain.sh` also exports each bundle to JSON (spike format,
+  ~4.9 MB per station vs 6.2 MB joblib). Rust loads them at start and reloads on
+  change. A bundle whose `feature_names` Rust cannot build is refused and the
+  station falls back to the heuristic, as Python does today.
+
+## The permanent cost: the Python/Rust model contract
+
+Every feature added in `features.py` must be ported to Rust in the same change,
+or the new bundle is refused. That has to be enforced by CI: Python writes a parity
+fixture per station (as `export_fixture.py` does), and a Rust test asserts
+bit-identical features and regression output. The spike found two traps that
+silently flip tree branches (HGB thresholds often equal real data values):
+serde_json's default float parsing (1 ulp off; enable `float_roundtrip`) and
+pandas' Kahan-compensated rolling sums/means (replicated in `rolling_kahan()`).
+This slows model iteration, which is the most active part of the repo.
+
+## Benefits and costs
+
+| For | Against |
+| --- | --- |
+| Single process: no leader election, no per-worker state | Two languages forever; parity CI is mandatory |
+| Memory ~324 MB -> est. 60-80 MB (mostly tree arrays; less with u32/u16 nodes) | Weeks of work with little user-visible change |
+| Typed RID/HII parsing, compile-checked SQL | Re-introduces bugs into recently hardened code (tz, alerts, security) |
+| One static binary; no venv on the server for the API | Server still needs Python + sklearn for the monthly retrain |
+| Same stack as proplan | Feature work = Python + Rust change every time |
+
+## Plan, if pursued (each phase ships on its own)
+
+0. **Spike: done.** P.1 features + HGB inference, bit-identical.
+1. **Collector + notifier daemon** (~1.5k LOC): RID (the fixed semantics: store
+   late readings, retry while behind, re-fetch yesterday's holes), HII, dams,
+   Open-Meteo, ntfy, `notification_state`. The Python API runs with collection
+   off and becomes a stateless reader. Shadow one week first: compare stored
+   hours against RID upstream.
+2. **Forecast crate** (~2k LOC): all 15 stations, dam/HII/rain inputs, heuristic
+   fallback, probabilities and stages, `forecast_history`; parity CI for every
+   station on a flood window and a dry window.
+3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
+   then a Caddy switch. Python keeps only training/evaluation.
+4. **Cleanup:** drop the multi-backend adapters, demo databases and the
+   Python web stack.
+
+Open questions before phase 1: is Matrix alerting still used? Must `/docs`
+(OpenAPI UI) stay? Is SQLite dev mode needed? Where does the monthly retrain
+run if the server stops having a Python environment for the API?
+
+## Cheaper alternative (stay Python)
+
+Most of the operational gain comes from taking collection out of the web
+workers: run the collector as its own systemd unit (one process) and the
+uvicorn workers as read-only API. That's about a day of work. It removes leader
+election and per-worker state, but doesn't bring the memory, typing or
+single-binary benefits.