fix: backtest/review findings in the flood-ML package
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.12) (push) Failing after 26s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 14s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s

From the adversarial review and threshold backtest (swarm verification):

- predict.py: when a bundle's trained thresholds differ from the current
  config (deploy before retrain), skip its stale classifier heads and
  derive p_warning/p_danger from the regression + sigma against the
  CURRENT thresholds - the dashboard can no longer show contradictory
  old-threshold classifier output next to new-threshold stages
- features.py: decouple the low-coverage regression-label rescue from
  the warning threshold (now the station's own p97.5 level); the old
  coupling silently dropped 34% of P.5's regression training rows and
  cost +46% MAE when its threshold rose
- features.py: P.82 danger 3.80 -> 3.75 (3.80 was above the station's
  8-year maximum of 3.78, so danger could never train or fire)
- data.py / predict.py: anchor models/cache paths to the repo root; the
  relative paths silently returned zero rows when run from another CWD
- annotate P.4A thresholds as low-confidence (11 supporting readings)

47 tests pass. Retrain required for the label-rescue and P.82 changes
to reach the classifier heads.
This commit is contained in:
grabowski committed 2026-08-10 15:57:01 +07:00
1 parent 9cac9c4d2a
commit ecd34177bb
4 files changed
+312 -76

No files matched your search

+56 -13
View File
@@ -18,9 +18,8 @@ logger = logging.getLogger(__name__)
# Static configuration
# ---------------------------------------------------------------------------
# Per-station (warning, danger) level thresholds in meters. "*" is the default
# applied to any station without an explicit override.
# Per-station (warning, danger) levels in metres on each gauge's own datum.
# "*" is the default applied to any station without an explicit override.
# Calibrated 2026-08-10 from the DB's discharge_percent (RID % of channel
# capacity): warning = median level at 75-85% capacity, danger = median level
# at 95-105%. P.1 instead uses the official Chiang Mai inundation map keyed to
@@ -32,6 +31,9 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
"P.103": (5.95, 6.75),
"P.20": (2.35, 2.80),
"P.21": (3.20, 3.60),
# P.4A: LOW CONFIDENCE — its sensor was dead 2019-2024 and only 11 readings
# ever reached 3.40 m, so the capacity calibration rests on very few points.
# It only affects the heuristic sigmoid (P.4A is NOT_TRAINABLE).
"P.4A": (3.40, 3.90),
"P.5": (4.55, 4.95),
"P.67": (2.45, 2.90),
@@ -39,7 +41,9 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
"P.76": (5.35, 5.45),
"P.77": (2.85, 3.35),
"P.81": (5.15, 6.30),
"P.82": (3.40, 3.80),
# P.82 never reached 100% capacity in the record (max level 3.78, max 96.4%);
# danger sits just below the observed maximum so the head can actually train.
"P.82": (3.40, 3.75),
"P.84": (3.45, 3.90),
"P.85": (2.90, 3.35),
"P.87": (3.75, 4.05),
@@ -73,8 +77,23 @@ BASIN_ANCHOR = "P.1"
# lists, for each station, the (upstream_code, lead_hours) pairs to use as
# routed-upstream input features when forecasting `station`.
UPSTREAM_LEADS: Dict[str, List[Tuple[str, int]]] = {
"P.1": [("P.103", 1), ("P.67", 7), ("P.21", 9), ("P.75", 12), ("P.4A", 12), ("P.92", 15), ("P.20", 17)],
"P.103": [("P.67", 6), ("P.21", 8), ("P.75", 11), ("P.4A", 11), ("P.92", 14), ("P.20", 16)],
"P.1": [
("P.103", 1),
("P.67", 7),
("P.21", 9),
("P.75", 12),
("P.4A", 12),
("P.92", 15),
("P.20", 17),
],
"P.103": [
("P.67", 6),
("P.21", 8),
("P.75", 11),
("P.4A", 11),
("P.92", 14),
("P.20", 16),
],
"P.21": [("P.67", 1), ("P.75", 3), ("P.4A", 3), ("P.92", 6), ("P.20", 8)],
"P.67": [("P.75", 5), ("P.4A", 5), ("P.92", 8), ("P.20", 10)],
"P.75": [("P.92", 3), ("P.20", 5)],
@@ -140,9 +159,13 @@ def make_hourly_grid(df_long: pd.DataFrame) -> HourlyGrid:
df["timestamp"] = pd.to_datetime(df["timestamp"]).dt.floor("h")
df = df.drop_duplicates(subset=["station_code", "timestamp"], keep="last")
full_index = pd.date_range(df["timestamp"].min(), df["timestamp"].max(), freq="h", name="timestamp")
full_index = pd.date_range(
df["timestamp"].min(), df["timestamp"].max(), freq="h", name="timestamp"
)
wide = df.pivot(index="timestamp", columns="station_code", values=["water_level", "discharge"])
wide = df.pivot(
index="timestamp", columns="station_code", values=["water_level", "discharge"]
)
wide = wide.reorder_levels([1, 0], axis=1).sort_index(axis=1)
wide = wide.reindex(full_index)
@@ -154,7 +177,9 @@ def make_hourly_grid(df_long: pd.DataFrame) -> HourlyGrid:
return HourlyGrid(observed=observed, filled=filled, mask=mask)
def _series(grid_frame: pd.DataFrame, station: str, field: str, index: pd.Index) -> pd.Series:
def _series(
grid_frame: pd.DataFrame, station: str, field: str, index: pd.Index
) -> pd.Series:
"""Fetch a (station, field) column, or an all-NaN series if the station is absent."""
if (station, field) in grid_frame.columns:
return grid_frame[(station, field)]
@@ -214,7 +239,9 @@ def build_features(grid: HourlyGrid, station: str) -> pd.DataFrame:
cols[f"{upstream_code}_level_lag_{lead_h}"] = u_level.shift(lead_h)
cols[f"{upstream_code}_level_lag_{lead_h + 3}"] = u_level.shift(lead_h + 3)
cols[f"{upstream_code}_rise_6_lag_{lead_h}"] = u_rise_6.shift(lead_h)
cols[f"{upstream_code}_rollmax_24_lag_{near_lag}"] = u_rollmax_24.shift(near_lag)
cols[f"{upstream_code}_rollmax_24_lag_{near_lag}"] = u_rollmax_24.shift(
near_lag
)
if station != BASIN_ANCHOR:
p1_level = _series(grid.filled, BASIN_ANCHOR, "water_level", idx)
@@ -244,11 +271,23 @@ def _future_window_stats(col: pd.Series, horizon_h: int) -> Tuple[pd.Series, pd.
return fut_max, fut_count
def build_labels(grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)) -> pd.DataFrame:
def build_labels(
grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)
) -> pd.DataFrame:
"""Build max-level and threshold-exceedance labels for one target station."""
idx = grid.observed.index
observed_level = _series(grid.observed, station, "water_level", idx)
warn_thr, danger_thr = get_thresholds(station)
# Low-coverage windows are still usable regression labels when they contain a
# rare high reading. Anchor "rare" to the station's own distribution (p97.5),
# NOT to warn_thr: coupling it to the configurable threshold made raising a
# station's threshold silently shrink its regression training set (P.5 lost
# 34% of rows and +46% MAE when its warning went 3.0 -> 4.55).
rescue_thr = (
float(observed_level.quantile(0.975))
if observed_level.notna().any()
else np.inf
)
out: Dict[str, pd.Series] = {}
for horizon_h in horizons:
@@ -264,7 +303,7 @@ def build_labels(grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6,
exceed_danger[fut_max >= danger_thr] = 1.0
exceed_danger[enough_cov & exceed_danger.isna()] = 0.0
max_level_valid = fut_max.where(enough_cov | (fut_max >= warn_thr))
max_level_valid = fut_max.where(enough_cov | (fut_max >= rescue_thr))
out[f"max_level_{horizon_h}"] = max_level_valid
out[f"exceed_warn_{horizon_h}"] = exceed_warn
@@ -297,13 +336,17 @@ def build_matrix(
Y = Y.loc[keep]
positive_counts = {
col: int(Y[col].sum()) for col in Y.columns if col.startswith("exceed_") and Y[col].notna().any()
col: int(Y[col].sum())
for col in Y.columns
if col.startswith("exceed_") and Y[col].notna().any()
}
meta = {
"station_code": station,
"n_rows": int(len(X)),
"span": (
(X.index.min().isoformat(), X.index.max().isoformat()) if len(X) else (None, None)
(X.index.min().isoformat(), X.index.max().isoformat())
if len(X)
else (None, None)
),
"positive_counts": positive_counts,
}