feat: Mae Ngat dam features — built, evaluated, defaulted OFF
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 27s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped

src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame
(daily row visible from 07:00 its own date, ffill capped at 48 h) and is
plumbed through features/train/predict/evaluate exactly like rain, gated
to the six mainstem stations below the Mae Ngat confluence.

The experiment concludes as a documented NEGATIVE result: on the 2024
record-flood backtest every dam-feature subset costs 1-3 h of first-alert
lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID
report lags up to 31 h and describes yesterday's benign absorbing
reservoir during fast onset. Features therefore default OFF (--dam
opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness
variants, excluded from the default variant set). The ablation also
isolated the HII gap-fill as lead-neutral: the acceptance gate holds at
13 h with fill enabled, and docs/img charts are regenerated with the
shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5.

Review-swarm fixes: evaluate.py skips variants whose feature family is
absent instead of crashing the run; --dam forwards --db-url and warns
loudly when no dam history loads; an empty DB result can no longer wipe
a good dam cache; run-level metrics version claims v4 only when a dam
station is actually in the set.
This commit is contained in:
2026-08-13 20:42:21 +07:00
parent 6af6fbe02c
commit 28b62e5a36
12 changed files with 464 additions and 25 deletions
+111
View File
@@ -0,0 +1,111 @@
"""Mae Ngat reservoir series for the flood models.
rid_reservoir_daily (collected hourly by src/rid_reservoir.py, backfilled to
2018) holds daily storage/inflow/outflow for every RID large dam. Mae Ngat
Somboon Chon (DAM_ID 200103) is the only large dam upstream of Chiang Mai:
in Oct 2024 its inflow hit 19-22 MCM/day and storage 114% of usable capacity
days around the P.1 crossing — upstream state no river gauge carries.
Leakage rule: RID publishes the daily report for date D on the morning of D,
so the row becomes visible to features at D 07:00 local time, never earlier.
Forward-fill is capped at FFILL_LIMIT_H so a stalled collector degrades to
NaN (HGB-native) instead of silently serving stale reservoir state.
Known residual optimism: the collector upserts keep-last (and re-fetches
yesterday), so the stored row for date D is RID's FINAL revision, which
training then back-dates to D 07:00 — values live serving may not have had
that morning. This bias works IN FAVOR of dam features, so the 2026-08-13
negative result (they cost 1-3 h of alert lead) holds a fortiori; but any
future POSITIVE result must first validate intraday row stability or shift
the flow columns to D+1 07:00.
"""
import datetime
import logging
from pathlib import Path
from typing import Optional
import pandas as pd
from ..rid_reservoir import MAE_NGAT_DAM_ID
from .data import CACHE_DIR, resolve_db_url
logger = logging.getLogger(__name__)
REPORT_HOUR = 7 # daily value valid from 07:00 local on its own date
FFILL_LIMIT_H = 48 # two missed daily reports -> NaN, not stale state
DAM_COLUMNS = ("storage_pct", "inflow_mcm", "outflow_mcm")
CACHE_FILE = f"dam_{MAE_NGAT_DAM_ID}.csv.gz"
def load_daily(
db_url: Optional[str] = None,
dam_id: str = MAE_NGAT_DAM_ID,
start: Optional[datetime.date] = None,
cache_dir: Path = CACHE_DIR,
) -> Optional[pd.DataFrame]:
"""Daily dam rows indexed by date. DB first, on-disk cache as fallback."""
cache_path = Path(cache_dir) / CACHE_FILE
resolved = resolve_db_url(db_url)
if resolved:
try:
from sqlalchemy import create_engine, text
query = (
"SELECT date, storage_pct, inflow_mcm, outflow_mcm "
"FROM rid_reservoir_daily WHERE dam_id = :dam_id"
)
params = {"dam_id": dam_id}
if start is not None:
query += " AND date >= :start"
params["start"] = start
engine = create_engine(resolved, pool_pre_ping=True)
with engine.connect() as conn:
daily = pd.read_sql(
text(query + " ORDER BY date"), conn, params=params
)
daily["date"] = pd.to_datetime(daily["date"])
daily = daily.set_index("date")
for col in DAM_COLUMNS:
daily[col] = pd.to_numeric(daily[col], errors="coerce")
# Only full, NON-EMPTY loads refresh the cache: a truncated or
# freshly-recreated table must not wipe a good fallback archive.
if start is None and not daily.empty:
cache_path.parent.mkdir(parents=True, exist_ok=True)
daily.to_csv(cache_path, compression="gzip")
return daily
except Exception as error:
logger.warning(f"dam series DB load failed: {error}")
if cache_path.exists():
logger.warning("falling back to on-disk cache for the dam series")
return pd.read_csv(cache_path, index_col=0, parse_dates=True)
return None
def hourly_frame(daily: Optional[pd.DataFrame]) -> Optional[pd.DataFrame]:
"""Step the daily rows onto an hourly grid, each valid from D 07:00."""
if daily is None or daily.empty:
return None
frame = daily.copy()
frame.index = pd.to_datetime(frame.index) + pd.Timedelta(hours=REPORT_HOUR)
frame = frame[~frame.index.duplicated(keep="last")].sort_index()
hourly_index = pd.date_range(
frame.index.min(),
frame.index.max() + pd.Timedelta(hours=FFILL_LIMIT_H),
freq="h",
)
return frame.reindex(hourly_index).ffill(limit=FFILL_LIMIT_H)
def load_history(db_url: Optional[str] = None) -> Optional[pd.DataFrame]:
"""Full hourly Mae Ngat history for training; None when unavailable."""
return hourly_frame(load_daily(db_url))
def serving_frame(
db_url: Optional[str] = None, days: int = 21
) -> Optional[pd.DataFrame]:
"""Recent hourly dam state for inference (covers the 336 h feature window
plus the 72 h storage-delta lag)."""
start = datetime.date.today() - datetime.timedelta(days=days)
return hourly_frame(load_daily(db_url, start=start))