grabowski
6ae67dec63
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock).
125 lines
7.0 KiB
Markdown
125 lines
7.0 KiB
Markdown
# A Rust rewrite: assessment
|
|
|
|
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17;
|
|
phase 2 (forecast + ntfy) is built and parity-verified, cutover pending; see
|
|
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
|
|
|
|
## Short version
|
|
|
|
- **Feasible.** The risky part, serving the scikit-learn forecast from Rust with
|
|
no train/serve skew, works: the spike reproduces the pandas feature matrix
|
|
and the HistGradientBoosting regression output bit for bit (classifier heads
|
|
to the last ulp).
|
|
- **Training stays Python.** There is no Rust HGB that trains like
|
|
scikit-learn, and the evaluation harness (`scripts/evaluate_variants.py`) is
|
|
where model work happens. A "Rust rewrite" means a Rust service (collectors,
|
|
API, forecast serving, ntfy) plus a Python offline job (monthly retrain and
|
|
evaluation) that exports models for Rust.
|
|
- **Performance is not a reason.** The forecast takes 158 ms per station in Python
|
|
and 0.25 ms in Rust, but it runs once an hour. Load tests showed the site
|
|
is bandwidth-bound once caches are hit.
|
|
- **Real reasons, if any:** one process instead of leader-elected uvicorn workers
|
|
(the root of per-worker state bugs like `/scraping/status`), a fraction of the
|
|
memory (324 MB today), typed parsing and SQL, one deployable binary, and the
|
|
same stack as proplan (axum + sqlx).
|
|
- **Recommendation:** no big-bang rewrite. If Rust is wanted, start with the
|
|
collector + notifier daemon (phase 1 below), which stands alone and can be
|
|
abandoned without loss. The cheap Python alternative is at the end.
|
|
|
|
## What exists (Python, `src/`)
|
|
|
|
| Area | LOC | Fate in Rust |
|
|
| --- | ---: | --- |
|
|
| Collectors: RID hourly, HII/ThaiWater, RID dams, Open-Meteo | 2,500 | port; typed serde payloads |
|
|
| Web API (20 routes), schemas, health, metrics | 2,300 | port with identical JSON (golden-file tests) |
|
|
| DB adapters: Postgres, SQLite, MySQL, InfluxDB, VictoriaMetrics | 1,300 | Postgres only via sqlx, existing 13 tables, no data migration |
|
|
| ntfy notifications; Matrix alerting (CLI path) | 1,050 | port ntfy; Matrix only if still used |
|
|
| ML serving: predict, features, rain/dam/HII fill | ~1,300 | port; parity-tested (spike) |
|
|
| ML training + evaluation | ~1,700 | **stays Python** |
|
|
| CLI (`main.py`) | 650 | small clap CLI (`fill-gaps`, test cycle) |
|
|
| `static/dashboard.html` (164 KB) | | unchanged, served as a static file |
|
|
| tests | 3,900 | Rust unit tests + parity fixtures + API golden files |
|
|
|
|
## Target shape
|
|
|
|
One binary, one process (tokio):
|
|
|
|
- tasks: RID / HII / dams / rain collectors on their schedules; forecast
|
|
precompute when new readings land; ntfy notifier. Shared state is an
|
|
`Arc<RwLock<..>>`, so there's no leader election, no status hand-off file and
|
|
no executor sizing.
|
|
- axum + tower-http: same routes and JSON, compression (gzip/brotli also covers
|
|
the 257 KB river GeoJSON), static dashboard, CORS; `utoipa` if `/docs` stays.
|
|
- sqlx against the existing schema. Python and Rust can run against the same
|
|
database during migration.
|
|
- models: `train_all()` also exports each bundle (`models/flood_<station>.json.gz`,
|
|
1.2-2.3 MB), `scripts/retrain.sh` promotes it. Rust reloads changed files on
|
|
its next run. A bundle whose `feature_names` Rust cannot build falls back to
|
|
the heuristic, as Python does.
|
|
|
|
## The permanent cost: the Python/Rust model contract
|
|
|
|
Every feature added in `features.py` must be ported to Rust in the same change,
|
|
or the new bundle is refused. That has to be enforced by CI: Python writes a parity
|
|
fixture per station (as `export_fixture.py` does), and a Rust test asserts
|
|
bit-identical features and regression output. The spike found two traps that
|
|
silently flip tree branches (HGB thresholds often equal real data values):
|
|
serde_json's default float parsing (1 ulp off; enable `float_roundtrip`) and
|
|
pandas' Kahan-compensated rolling sums/means (replicated in `rolling_kahan()`).
|
|
This slows model iteration, which is the most active part of the repo.
|
|
|
|
## Benefits and costs
|
|
|
|
| For | Against |
|
|
| --- | --- |
|
|
| Single process: no leader election, no per-worker state | Two languages forever; parity CI is mandatory |
|
|
| Memory ~324 MB -> est. 60-80 MB (mostly tree arrays; less with u32/u16 nodes) | Weeks of work with little user-visible change |
|
|
| Typed RID/HII parsing, compile-checked SQL | Re-introduces bugs into recently hardened code (tz, alerts, security) |
|
|
| One static binary; no venv on the server for the API | Server still needs Python + sklearn for the monthly retrain |
|
|
| Same stack as proplan | Feature work = Python + Rust change every time |
|
|
|
|
## Plan (each phase ships on its own)
|
|
|
|
| Phase | Scope | Status |
|
|
| --- | --- | --- |
|
|
| 0 | Spike: P.1 features + HGB inference | done, bit-identical |
|
|
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
|
|
| 2 | Forecast crate + ntfy | built 2026-10-18, parity-verified; cutover with `install_collector.sh --cutover-forecast` |
|
|
| 3 | Read API | |
|
|
| 4 | Cleanup | |
|
|
|
|
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
|
|
1. **Collector daemon** (`rust/collector`, ~1.5k LOC incl. tests): RID with the
|
|
fixed semantics (store late readings, retry while behind, re-fetch
|
|
yesterday's holes), HII, dams, Open-Meteo, into the same tables. The Python
|
|
API runs with `COLLECTION_ENABLED=false` and re-forecasts when the newest
|
|
RID reading in the database advances. ntfy moved to phase 2: the outlook
|
|
alert needs the forecasts, which are still Python. Cutover and rollback
|
|
are one command each (`scripts/install_collector.sh`).
|
|
2. **Forecast crate + ntfy** (`rust/forecast`, run by `ping-collector` with
|
|
`FORECAST_IN_COLLECTOR=true`): all stations, HII/rain/dam inputs, heuristic
|
|
fallback, probabilities and stages, `forecast_history`, `forecast_current`
|
|
(served by the API) and `notification_state`. Parity: fixtures from the
|
|
Python production path (CI: P.1 model; locally all 15), a live comparison
|
|
script, and 751 notifications replayed word for word. Porting found a
|
|
Python serving bug, fixed first: forecast rain (`rain_fc24`) was always
|
|
NaN at serving time (docs/FLOOD_FORECASTING.md section 5). The ntfy change:
|
|
ThaiWater mirror readings keep level alerts and the feed check current
|
|
while RID lags at night. The Python API then only serves.
|
|
3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
|
|
then a Caddy switch. Python keeps only training/evaluation.
|
|
4. **Cleanup:** drop the multi-backend adapters, demo databases and the
|
|
Python web stack.
|
|
|
|
Open questions before phase 3: is Matrix alerting still used? Must `/docs`
|
|
(OpenAPI UI) stay? Is SQLite dev mode needed? Where does the monthly retrain
|
|
run if the server stops having a Python environment for the API?
|
|
|
|
## Cheaper alternative (stay Python)
|
|
|
|
Most of the operational gain comes from taking collection out of the web
|
|
workers: run the collector as its own systemd unit (one process) and the
|
|
uvicorn workers as read-only API. That's about a day of work. It removes leader
|
|
election and per-worker state, but doesn't bring the memory, typing or
|
|
single-binary benefits.
|