feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s

Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103;
results in models/eval_variants.json) showed the absolute-level target
alerting AT the crossing on essentially every event, while the rise
target (future max - current level, level added back at serving) gives
+6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than
weighted/quantile variants, and ~11% better MAE. Weighted and quantile
variants rejected: more false alarms, no Brier-score calibration gain.

Ported to production: train.py fits rise in both eval and refit passes
(sigma/metrics computed in absolute space), bundles stamped hgb-v2 with
regression_target='rise', predict.py adds the level back for v2 and
stays compatible with v1 bundles, backtest_render.py mirrors the same
math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h
BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and
the record-peak underprediction is gone (rise models can exceed the
training max). The >=12h acceptance gate still fails honestly at +6h —
closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at
6/12/24h; docs updated throughout.
This commit is contained in:
2026-08-12 15:43:29 +07:00
parent a0086086a2
commit 21e9d2e114
8 changed files with 814 additions and 44 deletions
+22 -7
View File
@@ -126,7 +126,8 @@ def _p_warning_series(
"""Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
if head is not None:
return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
predicted_max = pd.Series(reg.predict(X), index=X.index)
# reg predicts the RISE over current level; add the level back
predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"]
return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
@@ -240,14 +241,22 @@ def train_station(
)
horizon_metrics: dict = {}
# --- regression head (max level) ---
# --- regression head (rise to future max) ---
# Target = future max MINUS current level ("rise"). Rises are far more
# stationary than absolute stages, which softens the cannot-exceed-
# training-max ceiling: on the rolling-origin harness (2026-08-12) the
# rise target moved P.1 first-alert leads from +0h to +6/+46h and cut
# the 2024 record-peak underprediction. Prediction = rise + level.
reg_labeled = eval_Y[max_col].notna()
reg = None
if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
rise_target = (
eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"]
)
reg = _safe_fit(
_make_regressor(hgb_overrides),
eval_X.loc[reg_labeled],
eval_Y.loc[reg_labeled, max_col],
rise_target,
f"max_{h}",
skipped_heads,
)
@@ -259,7 +268,10 @@ def train_station(
test_labeled = Y_test[max_col].notna()
if test_labeled.sum() > 0:
y_true = Y_test.loc[test_labeled, max_col]
y_pred = reg.predict(X_test.loc[test_labeled])
y_pred = (
reg.predict(X_test.loc[test_labeled])
+ X_test.loc[test_labeled, "level"].to_numpy()
)
residuals = y_true.to_numpy() - y_pred
sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
horizon_metrics["n_test"] = int(test_labeled.sum())
@@ -367,7 +379,7 @@ def train_station(
reg = _safe_fit(
_make_regressor(hgb_overrides),
X.loc[labeled],
Y.loc[labeled, max_col],
Y.loc[labeled, max_col] - X.loc[labeled, "level"], # rise target
head_key,
skipped_heads,
)
@@ -401,7 +413,10 @@ def train_station(
bundle = {
"station_code": station,
"model_version": f"hgb-v1+{_git_short_sha()}",
"model_version": f"hgb-v2+{_git_short_sha()}",
# v2: regression heads predict the RISE over the current level; the
# serving side must add the level back. Old v1 bundles lack this key.
"regression_target": "rise",
"trained_at": datetime.datetime.now().isoformat(),
"sklearn_version": sklearn.__version__,
"feature_names": feature_names,
@@ -428,7 +443,7 @@ def train_all(
"""Train and save every requested station's models. Returns the metrics.json payload."""
models_dir = Path(models_dir)
models_dir.mkdir(parents=True, exist_ok=True)
model_version = f"hgb-v1+{_git_short_sha()}"
model_version = f"hgb-v2+{_git_short_sha()}"
station_results: Dict[str, dict] = {}
for station in stations: