fix: backtest/review findings in the flood-ML package
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.12) (push) Failing after 26s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 14s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.12) (push) Failing after 26s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 14s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
From the adversarial review and threshold backtest (swarm verification): - predict.py: when a bundle's trained thresholds differ from the current config (deploy before retrain), skip its stale classifier heads and derive p_warning/p_danger from the regression + sigma against the CURRENT thresholds - the dashboard can no longer show contradictory old-threshold classifier output next to new-threshold stages - features.py: decouple the low-coverage regression-label rescue from the warning threshold (now the station's own p97.5 level); the old coupling silently dropped 34% of P.5's regression training rows and cost +46% MAE when its threshold rose - features.py: P.82 danger 3.80 -> 3.75 (3.80 was above the station's 8-year maximum of 3.78, so danger could never train or fire) - data.py / predict.py: anchor models/cache paths to the repo root; the relative paths silently returned zero rows when run from another CWD - annotate P.4A thresholds as low-confidence (11 supporting readings) 47 tests pass. Retrain required for the label-rescue and P.82 changes to reach the classifier heads.
This commit is contained in:
1 parent
9cac9c4d2a
commit
ecd34177bb
4 files changed
+312
-76
No files matched your search
+56
-13
@@ -18,9 +18,8 @@ logger = logging.getLogger(__name__)
|
||||
# Static configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Per-station (warning, danger) level thresholds in meters. "*" is the default
|
||||
# applied to any station without an explicit override.
|
||||
# Per-station (warning, danger) levels in metres on each gauge's own datum.
|
||||
# "*" is the default applied to any station without an explicit override.
|
||||
# Calibrated 2026-08-10 from the DB's discharge_percent (RID % of channel
|
||||
# capacity): warning = median level at 75-85% capacity, danger = median level
|
||||
# at 95-105%. P.1 instead uses the official Chiang Mai inundation map keyed to
|
||||
@@ -32,6 +31,9 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
|
||||
"P.103": (5.95, 6.75),
|
||||
"P.20": (2.35, 2.80),
|
||||
"P.21": (3.20, 3.60),
|
||||
# P.4A: LOW CONFIDENCE — its sensor was dead 2019-2024 and only 11 readings
|
||||
# ever reached 3.40 m, so the capacity calibration rests on very few points.
|
||||
# It only affects the heuristic sigmoid (P.4A is NOT_TRAINABLE).
|
||||
"P.4A": (3.40, 3.90),
|
||||
"P.5": (4.55, 4.95),
|
||||
"P.67": (2.45, 2.90),
|
||||
@@ -39,7 +41,9 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
|
||||
"P.76": (5.35, 5.45),
|
||||
"P.77": (2.85, 3.35),
|
||||
"P.81": (5.15, 6.30),
|
||||
"P.82": (3.40, 3.80),
|
||||
# P.82 never reached 100% capacity in the record (max level 3.78, max 96.4%);
|
||||
# danger sits just below the observed maximum so the head can actually train.
|
||||
"P.82": (3.40, 3.75),
|
||||
"P.84": (3.45, 3.90),
|
||||
"P.85": (2.90, 3.35),
|
||||
"P.87": (3.75, 4.05),
|
||||
@@ -73,8 +77,23 @@ BASIN_ANCHOR = "P.1"
|
||||
# lists, for each station, the (upstream_code, lead_hours) pairs to use as
|
||||
# routed-upstream input features when forecasting `station`.
|
||||
UPSTREAM_LEADS: Dict[str, List[Tuple[str, int]]] = {
|
||||
"P.1": [("P.103", 1), ("P.67", 7), ("P.21", 9), ("P.75", 12), ("P.4A", 12), ("P.92", 15), ("P.20", 17)],
|
||||
"P.103": [("P.67", 6), ("P.21", 8), ("P.75", 11), ("P.4A", 11), ("P.92", 14), ("P.20", 16)],
|
||||
"P.1": [
|
||||
("P.103", 1),
|
||||
("P.67", 7),
|
||||
("P.21", 9),
|
||||
("P.75", 12),
|
||||
("P.4A", 12),
|
||||
("P.92", 15),
|
||||
("P.20", 17),
|
||||
],
|
||||
"P.103": [
|
||||
("P.67", 6),
|
||||
("P.21", 8),
|
||||
("P.75", 11),
|
||||
("P.4A", 11),
|
||||
("P.92", 14),
|
||||
("P.20", 16),
|
||||
],
|
||||
"P.21": [("P.67", 1), ("P.75", 3), ("P.4A", 3), ("P.92", 6), ("P.20", 8)],
|
||||
"P.67": [("P.75", 5), ("P.4A", 5), ("P.92", 8), ("P.20", 10)],
|
||||
"P.75": [("P.92", 3), ("P.20", 5)],
|
||||
@@ -140,9 +159,13 @@ def make_hourly_grid(df_long: pd.DataFrame) -> HourlyGrid:
|
||||
df["timestamp"] = pd.to_datetime(df["timestamp"]).dt.floor("h")
|
||||
df = df.drop_duplicates(subset=["station_code", "timestamp"], keep="last")
|
||||
|
||||
full_index = pd.date_range(df["timestamp"].min(), df["timestamp"].max(), freq="h", name="timestamp")
|
||||
full_index = pd.date_range(
|
||||
df["timestamp"].min(), df["timestamp"].max(), freq="h", name="timestamp"
|
||||
)
|
||||
|
||||
wide = df.pivot(index="timestamp", columns="station_code", values=["water_level", "discharge"])
|
||||
wide = df.pivot(
|
||||
index="timestamp", columns="station_code", values=["water_level", "discharge"]
|
||||
)
|
||||
wide = wide.reorder_levels([1, 0], axis=1).sort_index(axis=1)
|
||||
wide = wide.reindex(full_index)
|
||||
|
||||
@@ -154,7 +177,9 @@ def make_hourly_grid(df_long: pd.DataFrame) -> HourlyGrid:
|
||||
return HourlyGrid(observed=observed, filled=filled, mask=mask)
|
||||
|
||||
|
||||
def _series(grid_frame: pd.DataFrame, station: str, field: str, index: pd.Index) -> pd.Series:
|
||||
def _series(
|
||||
grid_frame: pd.DataFrame, station: str, field: str, index: pd.Index
|
||||
) -> pd.Series:
|
||||
"""Fetch a (station, field) column, or an all-NaN series if the station is absent."""
|
||||
if (station, field) in grid_frame.columns:
|
||||
return grid_frame[(station, field)]
|
||||
@@ -214,7 +239,9 @@ def build_features(grid: HourlyGrid, station: str) -> pd.DataFrame:
|
||||
cols[f"{upstream_code}_level_lag_{lead_h}"] = u_level.shift(lead_h)
|
||||
cols[f"{upstream_code}_level_lag_{lead_h + 3}"] = u_level.shift(lead_h + 3)
|
||||
cols[f"{upstream_code}_rise_6_lag_{lead_h}"] = u_rise_6.shift(lead_h)
|
||||
cols[f"{upstream_code}_rollmax_24_lag_{near_lag}"] = u_rollmax_24.shift(near_lag)
|
||||
cols[f"{upstream_code}_rollmax_24_lag_{near_lag}"] = u_rollmax_24.shift(
|
||||
near_lag
|
||||
)
|
||||
|
||||
if station != BASIN_ANCHOR:
|
||||
p1_level = _series(grid.filled, BASIN_ANCHOR, "water_level", idx)
|
||||
@@ -244,11 +271,23 @@ def _future_window_stats(col: pd.Series, horizon_h: int) -> Tuple[pd.Series, pd.
|
||||
return fut_max, fut_count
|
||||
|
||||
|
||||
def build_labels(grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)) -> pd.DataFrame:
|
||||
def build_labels(
|
||||
grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)
|
||||
) -> pd.DataFrame:
|
||||
"""Build max-level and threshold-exceedance labels for one target station."""
|
||||
idx = grid.observed.index
|
||||
observed_level = _series(grid.observed, station, "water_level", idx)
|
||||
warn_thr, danger_thr = get_thresholds(station)
|
||||
# Low-coverage windows are still usable regression labels when they contain a
|
||||
# rare high reading. Anchor "rare" to the station's own distribution (p97.5),
|
||||
# NOT to warn_thr: coupling it to the configurable threshold made raising a
|
||||
# station's threshold silently shrink its regression training set (P.5 lost
|
||||
# 34% of rows and +46% MAE when its warning went 3.0 -> 4.55).
|
||||
rescue_thr = (
|
||||
float(observed_level.quantile(0.975))
|
||||
if observed_level.notna().any()
|
||||
else np.inf
|
||||
)
|
||||
|
||||
out: Dict[str, pd.Series] = {}
|
||||
for horizon_h in horizons:
|
||||
@@ -264,7 +303,7 @@ def build_labels(grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6,
|
||||
exceed_danger[fut_max >= danger_thr] = 1.0
|
||||
exceed_danger[enough_cov & exceed_danger.isna()] = 0.0
|
||||
|
||||
max_level_valid = fut_max.where(enough_cov | (fut_max >= warn_thr))
|
||||
max_level_valid = fut_max.where(enough_cov | (fut_max >= rescue_thr))
|
||||
|
||||
out[f"max_level_{horizon_h}"] = max_level_valid
|
||||
out[f"exceed_warn_{horizon_h}"] = exceed_warn
|
||||
@@ -297,13 +336,17 @@ def build_matrix(
|
||||
Y = Y.loc[keep]
|
||||
|
||||
positive_counts = {
|
||||
col: int(Y[col].sum()) for col in Y.columns if col.startswith("exceed_") and Y[col].notna().any()
|
||||
col: int(Y[col].sum())
|
||||
for col in Y.columns
|
||||
if col.startswith("exceed_") and Y[col].notna().any()
|
||||
}
|
||||
meta = {
|
||||
"station_code": station,
|
||||
"n_rows": int(len(X)),
|
||||
"span": (
|
||||
(X.index.min().isoformat(), X.index.max().isoformat()) if len(X) else (None, None)
|
||||
(X.index.min().isoformat(), X.index.max().isoformat())
|
||||
if len(X)
|
||||
else (None, None)
|
||||
),
|
||||
"positive_counts": positive_counts,
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user