feat: rolling-origin event-aware evaluation harness for model variants
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s
One fold per monsoon season (train <= 30 Apr, test Jun-Nov, 2021-2025) replaces the single fixed holdout that contained only ~4 warning events. Metrics are what matters operationally: sustained first-alert lead vs each observed 3.70m crossing (two consecutive alerting samples required; lookback floored at the previous event's end so multi-peak floods can't launder lead credit), peak error from the prediction actually issued 24h before the peak (3h match tolerance, null on outages), false-alarm episodes (12h gap tolerance), MAE / flood-regime MAE, and a Brier score on warning exceedance — included because sigma cancels algebraically in any p>=0.5 alert metric, so lead times compare predictors while Brier compares uncertainty models. Variants: baseline_abs (current), rise (target = future max - current level), rise_weighted (flood-regime sample weights 1x->5x), and rise_quantile (q50/q90 heads, spread-implied sigma). Harness verified by a 3-agent adversarial review (features bit-identical across fold cutoffs; three metric flaws found and fixed before first use). Also: features.build_labels/build_matrix gain stats_end so the rescue quantile is computed from pre-cutoff data only, closing the label- construction leak flagged in the earlier ML review.
This commit is contained in:
1 parent
98023243af
commit
a0086086a2
3 files changed
+398
-6
No files matched your search
+16
-6
@@ -272,9 +272,17 @@ def _future_window_stats(col: pd.Series, horizon_h: int) -> Tuple[pd.Series, pd.
|
||||
|
||||
|
||||
def build_labels(
|
||||
grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)
|
||||
grid: HourlyGrid,
|
||||
station: str,
|
||||
horizons: Tuple[int, ...] = (6, 12, 24),
|
||||
stats_end: Optional[str] = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Build max-level and threshold-exceedance labels for one target station."""
|
||||
"""Build max-level and threshold-exceedance labels for one target station.
|
||||
|
||||
``stats_end`` bounds the data used for label-construction statistics (the
|
||||
rescue quantile below): pass the training cutoff during evaluation so
|
||||
test-period extremes cannot influence which training rows receive labels.
|
||||
"""
|
||||
idx = grid.observed.index
|
||||
observed_level = _series(grid.observed, station, "water_level", idx)
|
||||
warn_thr, danger_thr = get_thresholds(station)
|
||||
@@ -283,10 +291,11 @@ def build_labels(
|
||||
# NOT to warn_thr: coupling it to the configurable threshold made raising a
|
||||
# station's threshold silently shrink its regression training set (P.5 lost
|
||||
# 34% of rows and +46% MAE when its warning went 3.0 -> 4.55).
|
||||
stats_level = (
|
||||
observed_level.loc[: pd.Timestamp(stats_end)] if stats_end else observed_level
|
||||
)
|
||||
rescue_thr = (
|
||||
float(observed_level.quantile(0.975))
|
||||
if observed_level.notna().any()
|
||||
else np.inf
|
||||
float(stats_level.quantile(0.975)) if stats_level.notna().any() else np.inf
|
||||
)
|
||||
|
||||
out: Dict[str, pd.Series] = {}
|
||||
@@ -321,11 +330,12 @@ def build_matrix(
|
||||
df_long: pd.DataFrame,
|
||||
station: str,
|
||||
horizons: Tuple[int, ...] = (6, 12, 24),
|
||||
stats_end: Optional[str] = None,
|
||||
) -> Tuple[pd.DataFrame, pd.DataFrame, dict]:
|
||||
"""Build (X, Y, meta) training/inference matrices for one station."""
|
||||
grid = make_hourly_grid(df_long)
|
||||
X = build_features(grid, station)
|
||||
Y = build_labels(grid, station, horizons)
|
||||
Y = build_labels(grid, station, horizons, stats_end=stats_end)
|
||||
|
||||
keep = X["obs_age_h"].notna()
|
||||
train_start = TRAIN_START.get(station)
|
||||
|
||||
Reference in new issue
Block a user