feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.
This commit is contained in:
@@ -160,3 +160,60 @@ def serving_series() -> Optional[pd.Series]:
|
||||
except Exception as error:
|
||||
logger.warning(f"Open-Meteo forecast fetch failed: {error}")
|
||||
return None
|
||||
|
||||
|
||||
def save_to_db(df: pd.DataFrame, engine, db_type: str) -> int:
|
||||
"""Upsert per-point + catchment-mean hourly rain into openmeteo_rain.
|
||||
|
||||
Called by the leader worker's hourly precompute with the live forecast
|
||||
frame, so the DB accumulates both what fell (past rows are the model
|
||||
analysis) and what was forecast (future rows, overwritten as they become
|
||||
past). The ML training path reads Open-Meteo's own archive, not this
|
||||
table — this is for dashboards, SQL analysis, and source independence.
|
||||
"""
|
||||
if df is None or df.empty:
|
||||
return 0
|
||||
from sqlalchemy import text
|
||||
|
||||
point_cols = [p[0] for p in CATCHMENT_POINTS]
|
||||
ddl_cols = ", ".join(f"{c} NUMERIC(6,2)" for c in point_cols)
|
||||
ddl = (
|
||||
"CREATE TABLE IF NOT EXISTS openmeteo_rain ("
|
||||
"timestamp TIMESTAMP PRIMARY KEY, "
|
||||
f"{ddl_cols}, catchment_mean NUMERIC(6,2), "
|
||||
"created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP)"
|
||||
)
|
||||
cols = ["timestamp"] + point_cols + ["catchment_mean"]
|
||||
placeholders = ", ".join(f":{c}" for c in cols)
|
||||
updates = ", ".join(
|
||||
f"{c} = "
|
||||
+ (f"VALUES({c})" if db_type == "mysql" else f"EXCLUDED.{c}")
|
||||
for c in cols[1:]
|
||||
)
|
||||
if db_type == "mysql":
|
||||
sql = (
|
||||
f"INSERT INTO openmeteo_rain ({', '.join(cols)}) VALUES ({placeholders}) "
|
||||
f"ON DUPLICATE KEY UPDATE {updates}"
|
||||
)
|
||||
else:
|
||||
sql = (
|
||||
f"INSERT INTO openmeteo_rain ({', '.join(cols)}) VALUES ({placeholders}) "
|
||||
f"ON CONFLICT (timestamp) DO UPDATE SET {updates}"
|
||||
)
|
||||
mean = df.mean(axis=1)
|
||||
params = [
|
||||
{
|
||||
"timestamp": ts.to_pydatetime(),
|
||||
**{c: (None if pd.isna(row[c]) else float(row[c])) for c in point_cols},
|
||||
"catchment_mean": None if pd.isna(mean.loc[ts]) else float(mean.loc[ts]),
|
||||
}
|
||||
for ts, row in df.iterrows()
|
||||
]
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
conn.execute(text(ddl))
|
||||
conn.execute(text(sql), params)
|
||||
return len(params)
|
||||
except Exception as error:
|
||||
logger.error(f"openmeteo_rain save failed: {error}")
|
||||
return 0
|
||||
|
||||
Reference in New Issue
Block a user