feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.
This commit is contained in:
+22
-7
@@ -126,7 +126,8 @@ def _p_warning_series(
|
||||
"""Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
|
||||
if head is not None:
|
||||
return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
|
||||
predicted_max = pd.Series(reg.predict(X), index=X.index)
|
||||
# reg predicts the RISE over current level; add the level back
|
||||
predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"]
|
||||
return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
|
||||
|
||||
|
||||
@@ -240,14 +241,22 @@ def train_station(
|
||||
)
|
||||
horizon_metrics: dict = {}
|
||||
|
||||
# --- regression head (max level) ---
|
||||
# --- regression head (rise to future max) ---
|
||||
# Target = future max MINUS current level ("rise"). Rises are far more
|
||||
# stationary than absolute stages, which softens the cannot-exceed-
|
||||
# training-max ceiling: on the rolling-origin harness (2026-08-12) the
|
||||
# rise target moved P.1 first-alert leads from +0h to +6/+46h and cut
|
||||
# the 2024 record-peak underprediction. Prediction = rise + level.
|
||||
reg_labeled = eval_Y[max_col].notna()
|
||||
reg = None
|
||||
if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
|
||||
rise_target = (
|
||||
eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"]
|
||||
)
|
||||
reg = _safe_fit(
|
||||
_make_regressor(hgb_overrides),
|
||||
eval_X.loc[reg_labeled],
|
||||
eval_Y.loc[reg_labeled, max_col],
|
||||
rise_target,
|
||||
f"max_{h}",
|
||||
skipped_heads,
|
||||
)
|
||||
@@ -259,7 +268,10 @@ def train_station(
|
||||
test_labeled = Y_test[max_col].notna()
|
||||
if test_labeled.sum() > 0:
|
||||
y_true = Y_test.loc[test_labeled, max_col]
|
||||
y_pred = reg.predict(X_test.loc[test_labeled])
|
||||
y_pred = (
|
||||
reg.predict(X_test.loc[test_labeled])
|
||||
+ X_test.loc[test_labeled, "level"].to_numpy()
|
||||
)
|
||||
residuals = y_true.to_numpy() - y_pred
|
||||
sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
|
||||
horizon_metrics["n_test"] = int(test_labeled.sum())
|
||||
@@ -367,7 +379,7 @@ def train_station(
|
||||
reg = _safe_fit(
|
||||
_make_regressor(hgb_overrides),
|
||||
X.loc[labeled],
|
||||
Y.loc[labeled, max_col],
|
||||
Y.loc[labeled, max_col] - X.loc[labeled, "level"], # rise target
|
||||
head_key,
|
||||
skipped_heads,
|
||||
)
|
||||
@@ -401,7 +413,10 @@ def train_station(
|
||||
|
||||
bundle = {
|
||||
"station_code": station,
|
||||
"model_version": f"hgb-v1+{_git_short_sha()}",
|
||||
"model_version": f"hgb-v2+{_git_short_sha()}",
|
||||
# v2: regression heads predict the RISE over the current level; the
|
||||
# serving side must add the level back. Old v1 bundles lack this key.
|
||||
"regression_target": "rise",
|
||||
"trained_at": datetime.datetime.now().isoformat(),
|
||||
"sklearn_version": sklearn.__version__,
|
||||
"feature_names": feature_names,
|
||||
@@ -428,7 +443,7 @@ def train_all(
|
||||
"""Train and save every requested station's models. Returns the metrics.json payload."""
|
||||
models_dir = Path(models_dir)
|
||||
models_dir.mkdir(parents=True, exist_ok=True)
|
||||
model_version = f"hgb-v1+{_git_short_sha()}"
|
||||
model_version = f"hgb-v2+{_git_short_sha()}"
|
||||
|
||||
station_results: Dict[str, dict] = {}
|
||||
for station in stations:
|
||||
|
||||
Reference in New Issue
Block a user