feat: rolling-origin event-aware evaluation harness for model variants
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s

One fold per monsoon season (train <= 30 Apr, test Jun-Nov, 2021-2025)
replaces the single fixed holdout that contained only ~4 warning events.
Metrics are what matters operationally: sustained first-alert lead vs
each observed 3.70m crossing (two consecutive alerting samples required;
lookback floored at the previous event's end so multi-peak floods can't
launder lead credit), peak error from the prediction actually issued 24h
before the peak (3h match tolerance, null on outages), false-alarm
episodes (12h gap tolerance), MAE / flood-regime MAE, and a Brier score
on warning exceedance — included because sigma cancels algebraically in
any p>=0.5 alert metric, so lead times compare predictors while Brier
compares uncertainty models.

Variants: baseline_abs (current), rise (target = future max - current
level), rise_weighted (flood-regime sample weights 1x->5x), and
rise_quantile (q50/q90 heads, spread-implied sigma). Harness verified by
a 3-agent adversarial review (features bit-identical across fold
cutoffs; three metric flaws found and fixed before first use).

Also: features.build_labels/build_matrix gain stats_end so the rescue
quantile is computed from pre-cutoff data only, closing the label-
construction leak flagged in the earlier ML review.
This commit is contained in:
grabowski committed 2026-08-12 15:19:26 +07:00
1 parent 98023243af
commit a0086086a2
3 files changed
+398 -6

No files matched your search

+16 -6
View File
@@ -272,9 +272,17 @@ def _future_window_stats(col: pd.Series, horizon_h: int) -> Tuple[pd.Series, pd.
def build_labels(
grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)
grid: HourlyGrid,
station: str,
horizons: Tuple[int, ...] = (6, 12, 24),
stats_end: Optional[str] = None,
) -> pd.DataFrame:
"""Build max-level and threshold-exceedance labels for one target station."""
"""Build max-level and threshold-exceedance labels for one target station.
``stats_end`` bounds the data used for label-construction statistics (the
rescue quantile below): pass the training cutoff during evaluation so
test-period extremes cannot influence which training rows receive labels.
"""
idx = grid.observed.index
observed_level = _series(grid.observed, station, "water_level", idx)
warn_thr, danger_thr = get_thresholds(station)
@@ -283,10 +291,11 @@ def build_labels(
# NOT to warn_thr: coupling it to the configurable threshold made raising a
# station's threshold silently shrink its regression training set (P.5 lost
# 34% of rows and +46% MAE when its warning went 3.0 -> 4.55).
stats_level = (
observed_level.loc[: pd.Timestamp(stats_end)] if stats_end else observed_level
)
rescue_thr = (
float(observed_level.quantile(0.975))
if observed_level.notna().any()
else np.inf
float(stats_level.quantile(0.975)) if stats_level.notna().any() else np.inf
)
out: Dict[str, pd.Series] = {}
@@ -321,11 +330,12 @@ def build_matrix(
df_long: pd.DataFrame,
station: str,
horizons: Tuple[int, ...] = (6, 12, 24),
stats_end: Optional[str] = None,
) -> Tuple[pd.DataFrame, pd.DataFrame, dict]:
"""Build (X, Y, meta) training/inference matrices for one station."""
grid = make_hourly_grid(df_long)
X = build_features(grid, station)
Y = build_labels(grid, station, horizons)
Y = build_labels(grid, station, horizons, stats_end=stats_end)
keep = X["obs_age_h"].notna()
train_start = TRAIN_START.get(station)