feat(forecast): Rust forecast crate, bit-for-bit with the Python serving path
CI / Test suite (push) Successful in 1m2s
CI / Format & lint (push) Successful in 22s
Rust / fmt, clippy, test, build (push) Successful in 9m53s
Security / Dependency vulnerabilities (push) Successful in 1m52s
Security / Static analysis (push) Successful in 20s
Security / License report (push) Successful in 1m41s

Phase 2a of docs/RUST_REWRITE.md. rust/forecast (ping-forecast) ports the
serving half of src/ml: the HII mirror gap fill, the hourly grid, every
feature (pandas' Kahan rolling sums included), the Mae Ngat dam frame,
HistGradientBoosting tree inference, the heuristic fallback, stale-threshold
handling and the P.1 flood-stage probabilities.

Hand-over from Python (src/ml/export.py):
- models/flood_<station>.json.gz next to each joblib bundle; train_all() writes
  it, retrain.sh promotes it, `python -m src.ml.export` exports existing ones.
- rust/forecast/flood_config.json: thresholds, stages, leads and notify
  constants, compiled in. tests/test_model_export.py fails while it differs
  from the Python source.

Parity (rust/forecast/tests/parity.rs, fixtures from
testdata/gen_forecast_fixtures.py on the production code path): four
scenarios incl. the 2024 flood with RID lagging behind HII. Merged readings,
all features and all levels identical; with all 15 local models 377/384
probabilities bit-identical, the rest within 2 ulp (exp).

Python refactor without behaviour change: predict.forecasts_from_long, and
get_latest_forecasts/load_latest/serving_frame take now/rain for
reproducible comparisons; the HII query orders its rows.
This commit is contained in:
grabowski committed 2026-10-18 14:21:56 +07:00
1 parent 6d4beae65a
commit 5c4f464383
31 files changed
+2889 -29

No files matched your search

+104
View File
@@ -0,0 +1,104 @@
//! Inference for scikit-learn HistGradientBoosting heads exported as flat
//! tree arrays (src/ml/export.py). Mirrors sklearn's `_predictor.pyx`: NaN
//! follows `missing_go_to_left`, otherwise `x <= threshold` goes left; raw =
//! baseline + leaf values summed in tree order (sklearn's order, so the
//! regression output is bit-identical).
use anyhow::{Result, bail};
use serde::Deserialize;
fn thresholds<'de, D: serde::Deserializer<'de>>(d: D) -> Result<Vec<f64>, D::Error> {
let raw: Vec<Option<f64>> = Deserialize::deserialize(d)?;
Ok(raw
.into_iter()
.map(|t| t.unwrap_or(f64::INFINITY))
.collect())
}
#[derive(Debug, Deserialize)]
pub struct Tree {
feature: Vec<usize>,
#[serde(deserialize_with = "thresholds")]
threshold: Vec<f64>,
missing_left: Vec<bool>,
left: Vec<usize>,
right: Vec<usize>,
leaf: Vec<bool>,
value: Vec<f64>,
}
impl Tree {
#[inline]
fn predict(&self, x: &[f64]) -> f64 {
let mut n = 0;
while !self.leaf[n] {
let v = x[self.feature[n]];
let go_left = if v.is_nan() {
self.missing_left[n]
} else {
v <= self.threshold[n]
};
n = if go_left { self.left[n] } else { self.right[n] };
}
self.value[n]
}
fn validate(&self, n_features: usize) -> Result<()> {
let n = self.value.len();
if n == 0
|| [
self.feature.len(),
self.threshold.len(),
self.missing_left.len(),
self.left.len(),
self.right.len(),
self.leaf.len(),
]
.iter()
.any(|&l| l != n)
{
bail!("tree arrays have different lengths");
}
for i in 0..n {
if !self.leaf[i]
&& (self.feature[i] >= n_features || self.left[i] >= n || self.right[i] >= n)
{
bail!("tree node {i} points outside the tree or the feature vector");
}
}
Ok(())
}
}
#[derive(Debug, Deserialize, PartialEq, Eq, Clone, Copy)]
#[serde(rename_all = "lowercase")]
pub enum Kind {
Regressor,
Classifier,
}
#[derive(Debug, Deserialize)]
pub struct Head {
pub kind: Kind,
baseline: f64,
trees: Vec<Tree>,
}
impl Head {
/// Regressor: the prediction. Classifier: P(class 1), as
/// `predict_proba(X)[:, 1]` (scipy's `expit`).
pub fn predict(&self, x: &[f64]) -> f64 {
let mut raw = self.baseline;
for t in &self.trees {
raw += t.predict(x);
}
match self.kind {
Kind::Regressor => raw,
Kind::Classifier => 1.0 / (1.0 + (-raw).exp()),
}
}
pub fn validate(&self, n_features: usize) -> Result<()> {
self.trees.iter().try_for_each(|t| t.validate(n_features))
}
}