feat: Open-Meteo catchment rainfall series + rain features + harness variant
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Generate API Documentation (push) Successful in 8s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Generate API Documentation (push) Successful in 8s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s
src/ml/rain.py fetches hourly precipitation for five upper-Ping catchment points (Chiang Dao, Mae Taeng, Mae Ngat, Mae Rim, city) from the Open-Meteo forecast-model archive (2021-03 onward, no API key, Bangkok-local timestamps, year-chunked local cache) plus the live forecast endpoint for serving (trailing days + next 48h). features.build_features/build_matrix accept the catchment-mean series and add rain_6h/24h/72h trailing sums and rain_fc24 — the forward 24h sum, a genuine forecast feature (archived forecasts at training time, a real weather forecast at serving; never contains river data). Columns exist only when a series is provided: HGB rejects all-NaN columns at fit, so no-rain training omits them and serving passes an empty series for alignment. evaluate.py gains the rise_rain variant (and --no-rain) so the harness can judge whether rain beats the deployed rise baseline.
This commit is contained in:
+162
@@ -0,0 +1,162 @@
|
||||
"""Open-Meteo rainfall series for the upper Ping catchment.
|
||||
|
||||
One consistent source for training AND serving: the Open-Meteo forecast-model
|
||||
archive (historical-forecast-api, 2021-03 onward) supplies hourly
|
||||
precipitation at five catchment points above P.1; the live forecast endpoint
|
||||
supplies the same series for recent days plus the next 48 h. Timestamps are
|
||||
Asia/Bangkok local, matching the measurement grid. Rows before 2021-03 simply
|
||||
have no rain data — HistGradientBoosting handles the NaNs natively.
|
||||
|
||||
The forward-looking sum built from this series is a legitimate *forecast*
|
||||
feature, not label leakage: the series never contains river observations, and
|
||||
at serving time the future values come from an actual weather forecast.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Iterable, List, Optional, Tuple
|
||||
|
||||
import pandas as pd
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# (name, lat, lon) — upper Ping catchment above P.1, headwaters to city
|
||||
CATCHMENT_POINTS: Tuple[Tuple[str, float, float], ...] = (
|
||||
("chiang_dao", 19.37, 98.97),
|
||||
("mae_taeng", 19.12, 98.94),
|
||||
("mae_ngat", 19.17, 99.05),
|
||||
("mae_rim", 18.92, 98.92),
|
||||
("chiang_mai", 18.79, 99.00),
|
||||
)
|
||||
|
||||
HISTORY_URL = "https://historical-forecast-api.open-meteo.com/v1/forecast"
|
||||
FORECAST_URL = "https://api.open-meteo.com/v1/forecast"
|
||||
HISTORY_START = "2021-03-23" # archive begins here
|
||||
CACHE_FILE = "rain_openmeteo.csv.gz"
|
||||
|
||||
|
||||
def _points_params() -> dict:
|
||||
return {
|
||||
"latitude": ",".join(str(lat) for _, lat, _ in CATCHMENT_POINTS),
|
||||
"longitude": ",".join(str(lon) for _, _, lon in CATCHMENT_POINTS),
|
||||
"hourly": "precipitation",
|
||||
"timezone": "Asia/Bangkok",
|
||||
}
|
||||
|
||||
|
||||
def _parse_multi(payload, columns: Iterable[str]) -> pd.DataFrame:
|
||||
"""Open-Meteo returns a list when multiple coordinates are requested."""
|
||||
results = payload if isinstance(payload, list) else [payload]
|
||||
frames = {}
|
||||
for name, result in zip(columns, results):
|
||||
hourly = result.get("hourly", {})
|
||||
idx = pd.to_datetime(hourly.get("time", []))
|
||||
frames[name] = pd.Series(hourly.get("precipitation", []), index=idx)
|
||||
df = pd.DataFrame(frames)
|
||||
df.index.name = "timestamp"
|
||||
return df
|
||||
|
||||
|
||||
def fetch_history(
|
||||
start: str, end: str, session: Optional[requests.Session] = None
|
||||
) -> pd.DataFrame:
|
||||
"""Hourly precipitation for all catchment points over [start, end]."""
|
||||
session = session or requests.Session()
|
||||
response = session.get(
|
||||
HISTORY_URL,
|
||||
params={**_points_params(), "start_date": start, "end_date": end},
|
||||
timeout=120,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _parse_multi(response.json(), [p[0] for p in CATCHMENT_POINTS])
|
||||
|
||||
|
||||
def fetch_forecast(
|
||||
past_days: int = 5,
|
||||
forecast_days: int = 2,
|
||||
session: Optional[requests.Session] = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Recent + next-48h precipitation from the live forecast endpoint."""
|
||||
session = session or requests.Session()
|
||||
response = session.get(
|
||||
FORECAST_URL,
|
||||
params={
|
||||
**_points_params(),
|
||||
"past_days": past_days,
|
||||
"forecast_days": forecast_days,
|
||||
},
|
||||
timeout=60,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _parse_multi(response.json(), [p[0] for p in CATCHMENT_POINTS])
|
||||
|
||||
|
||||
def load_history(
|
||||
cache_dir: Path = Path("models/cache"),
|
||||
end: Optional[datetime.date] = None,
|
||||
refresh: bool = True,
|
||||
) -> Optional[pd.DataFrame]:
|
||||
"""Cached catchment rain history from 2021-03 to ~today.
|
||||
|
||||
Fetches year-sized chunks on first use (~6 requests), then only extends
|
||||
the tail. Returns None when the API is unreachable and no cache exists.
|
||||
"""
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
cache_path = cache_dir / CACHE_FILE
|
||||
end = end or datetime.date.today()
|
||||
|
||||
cached: Optional[pd.DataFrame] = None
|
||||
if cache_path.exists():
|
||||
cached = pd.read_csv(cache_path, index_col=0, parse_dates=True)
|
||||
|
||||
fetch_from = pd.Timestamp(HISTORY_START)
|
||||
if cached is not None and len(cached):
|
||||
fetch_from = cached.index.max() - pd.Timedelta(days=2) # re-fetch tail
|
||||
if not refresh and cached is not None:
|
||||
return cached
|
||||
|
||||
chunks: List[pd.DataFrame] = []
|
||||
cursor = fetch_from.date()
|
||||
try:
|
||||
while cursor <= end:
|
||||
chunk_end = min(
|
||||
datetime.date(cursor.year, 12, 31), end
|
||||
)
|
||||
chunks.append(
|
||||
fetch_history(cursor.isoformat(), chunk_end.isoformat())
|
||||
)
|
||||
cursor = datetime.date(cursor.year + 1, 1, 1)
|
||||
except Exception as error:
|
||||
logger.warning(f"Open-Meteo history fetch failed: {error}")
|
||||
if not chunks and cached is None:
|
||||
return None
|
||||
|
||||
if chunks:
|
||||
fresh = pd.concat(chunks)
|
||||
combined = (
|
||||
pd.concat([cached[cached.index < fresh.index.min()], fresh])
|
||||
if cached is not None
|
||||
else fresh
|
||||
)
|
||||
combined = combined[~combined.index.duplicated(keep="last")].sort_index()
|
||||
combined.to_csv(cache_path, compression="gzip")
|
||||
return combined
|
||||
return cached
|
||||
|
||||
|
||||
def catchment_mean(df: Optional[pd.DataFrame]) -> Optional[pd.Series]:
|
||||
"""Single catchment-average hourly rain series (mm)."""
|
||||
if df is None or df.empty:
|
||||
return None
|
||||
return df.mean(axis=1)
|
||||
|
||||
|
||||
def serving_series() -> Optional[pd.Series]:
|
||||
"""Catchment rain for inference: trailing days + the next 48 h forecast."""
|
||||
try:
|
||||
return catchment_mean(fetch_forecast())
|
||||
except Exception as error:
|
||||
logger.warning(f"Open-Meteo forecast fetch failed: {error}")
|
||||
return None
|
||||
Reference in New Issue
Block a user