grabowski
5c4f464383
CI / Test suite (push) Successful in 1m2s
CI / Format & lint (push) Successful in 22s
Rust / fmt, clippy, test, build (push) Successful in 9m53s
Security / Dependency vulnerabilities (push) Successful in 1m52s
Security / Static analysis (push) Successful in 20s
Security / License report (push) Successful in 1m41s
Phase 2a of docs/RUST_REWRITE.md. rust/forecast (ping-forecast) ports the serving half of src/ml: the HII mirror gap fill, the hourly grid, every feature (pandas' Kahan rolling sums included), the Mae Ngat dam frame, HistGradientBoosting tree inference, the heuristic fallback, stale-threshold handling and the P.1 flood-stage probabilities. Hand-over from Python (src/ml/export.py): - models/flood_<station>.json.gz next to each joblib bundle; train_all() writes it, retrain.sh promotes it, `python -m src.ml.export` exports existing ones. - rust/forecast/flood_config.json: thresholds, stages, leads and notify constants, compiled in. tests/test_model_export.py fails while it differs from the Python source. Parity (rust/forecast/tests/parity.rs, fixtures from testdata/gen_forecast_fixtures.py on the production code path): four scenarios incl. the 2024 flood with RID lagging behind HII. Merged readings, all features and all levels identical; with all 15 local models 377/384 probabilities bit-identical, the rest within 2 ulp (exp). Python refactor without behaviour change: predict.forecasts_from_long, and get_latest_forecasts/load_latest/serving_frame take now/rain for reproducible comparisons; the HII query orders its rows.
87 lines
2.9 KiB
Rust
87 lines
2.9 KiB
Rust
//! Long readings -> complete hourly grid (`features.make_hourly_grid`).
|
|
|
|
use crate::rolling::{Series, nan_series};
|
|
use chrono::{Duration, NaiveDateTime, Timelike};
|
|
use std::collections::BTreeMap;
|
|
|
|
/// One reading in long format; NaN = missing.
|
|
#[derive(Clone, Debug, PartialEq)]
|
|
pub struct Row {
|
|
pub timestamp: NaiveDateTime,
|
|
pub station_code: String,
|
|
pub water_level: f64,
|
|
pub discharge: f64,
|
|
}
|
|
|
|
pub fn floor_hour(t: NaiveDateTime) -> NaiveDateTime {
|
|
t.with_nanosecond(0)
|
|
.and_then(|t| t.with_second(0))
|
|
.and_then(|t| t.with_minute(0))
|
|
.unwrap_or(t)
|
|
}
|
|
|
|
/// Every hour from the first to the last reading; per station the observed
|
|
/// (unfilled) level and discharge, NaN where nothing was recorded.
|
|
pub struct Grid {
|
|
pub times: Vec<NaiveDateTime>,
|
|
pub level: BTreeMap<String, Series>,
|
|
pub discharge: BTreeMap<String, Series>,
|
|
}
|
|
|
|
impl Grid {
|
|
/// Timestamps are floored to the hour; for a duplicate (station, hour)
|
|
/// the later row wins (pandas `drop_duplicates(keep="last")`).
|
|
pub fn from_rows(rows: &[Row]) -> Grid {
|
|
let mut cells: BTreeMap<(String, NaiveDateTime), (f64, f64)> = BTreeMap::new();
|
|
for r in rows {
|
|
cells.insert(
|
|
(r.station_code.clone(), floor_hour(r.timestamp)),
|
|
(r.water_level, r.discharge),
|
|
);
|
|
}
|
|
let (Some(first), Some(last)) = (
|
|
cells.keys().map(|k| k.1).min(),
|
|
cells.keys().map(|k| k.1).max(),
|
|
) else {
|
|
return Grid {
|
|
times: vec![],
|
|
level: BTreeMap::new(),
|
|
discharge: BTreeMap::new(),
|
|
};
|
|
};
|
|
let n = ((last - first).num_hours() + 1) as usize;
|
|
let times: Vec<NaiveDateTime> = (0..n).map(|i| first + Duration::hours(i as i64)).collect();
|
|
let mut level: BTreeMap<String, Series> = BTreeMap::new();
|
|
let mut discharge: BTreeMap<String, Series> = BTreeMap::new();
|
|
for ((code, t), (l, q)) in cells {
|
|
let i = (t - first).num_hours() as usize;
|
|
level.entry(code.clone()).or_insert_with(|| nan_series(n))[i] = l;
|
|
discharge.entry(code).or_insert_with(|| nan_series(n))[i] = q;
|
|
}
|
|
Grid {
|
|
times,
|
|
level,
|
|
discharge,
|
|
}
|
|
}
|
|
|
|
pub fn len(&self) -> usize {
|
|
self.times.len()
|
|
}
|
|
|
|
pub fn is_empty(&self) -> bool {
|
|
self.times.is_empty()
|
|
}
|
|
|
|
pub fn index_of(&self, t: NaiveDateTime) -> Option<usize> {
|
|
let first = *self.times.first()?;
|
|
let i = (t - first).num_hours();
|
|
(i >= 0 && (i as usize) < self.len() && self.times[i as usize] == t).then_some(i as usize)
|
|
}
|
|
|
|
/// The station's last hour with an observed level.
|
|
pub fn last_observed(&self, station: &str) -> Option<usize> {
|
|
self.level.get(station)?.iter().rposition(|x| !x.is_nan())
|
|
}
|
|
}
|