grabowski
e3694d7591
spikes/rust-forecast ports build_features (P.1 path) and HistGradientBoosting inference to Rust and checks them against the pandas/sklearn output on two fixtures (2024 record flood, summer 2026): 244,800 feature cells and every regression prediction bit-identical, classifier heads to the last ulp; 0.25 ms vs 158 ms per station. docs/RUST_REWRITE.md: what a rewrite would mean (Rust service, Python training), costs, phased plan, Python alternative.
2.1 KiB
2.1 KiB
Rust forecast spike
Answers one question: can the flood forecast be served from Rust without train/serve skew, while training stays in Python (scikit-learn)? Assessment and migration plan: docs/RUST_REWRITE.md.
export/export_fixture.py: dumpsmodels/flood_P.1.joblib(all 9 HGB heads) as flat tree arrays, plus two fixtures frommodels/cache(Aug-Oct 2024 record flood, Jun-Aug 2026): the observed hourly grid of P.1 and its 7 upstream gauges, rain, the pandas feature matrix and every head's sklearn prediction.src/features.rs: port ofsrc/ml/features.py::build_features(P.1 path: lags, rises, rolling stats, upstream leads, season, rain).src/hgb.rs: HistGradientBoosting inference (~90 lines).src/main.rs: compares bit-for-bit and times serving-shaped work.
cd spikes/rust-forecast/export
uv run --project ../../.. python export_fixture.py # writes ../data/*.json (gitignored)
cd .. && cargo run --release
uv run --project ../.. python export/time_python.py # Python timing for comparison
Result (model hgb-v3+cbb3bf7, 3,600 hourly rows):
| Result | |
|---|---|
| Features, 68 columns x 3,600 rows | 244,800 / 244,800 cells bit-identical |
| Regression heads (max_6/12/24) | bit-identical on every row |
| Classifier heads (warn/danger) | equal to the last ulp (<= 2.2e-16, exp vs scipy expit) |
| One station: 336 h window -> features + 9 heads | Rust 0.25 ms, Python 158 ms |
| Model load | Rust 16 ms (JSON), Python 199 ms (joblib) |
Two traps the spike had to get past, both of which flip tree branches because HGB thresholds often equal real data values exactly:
serde_jsonparses floats best-effort (can be 1 ulp off) unless thefloat_roundtripfeature is on.- pandas
rolling().sum()/.mean()are incremental with Kahan compensation, a "run of identical values" shortcut and a sign clamp. A plain window sum differs in the last bits;rolling_kahan()replicates pandas exactly.
Not covered yet: dam features (Mae Ngat), HII gap fill, Open-Meteo catchment mean, the heuristic fallback, stage probabilities, the other 14 stations.