B4L/Northern-Thailand-Ping-River-Monitor
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

chore: remove the Python service; Python is the training stack only

Browse Source 
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/backtest_render.py
+188 -75
View File
@@ -14,7 +14,7 @@ non-zero exit if the model gives less than 12 h of warning before the first
3.70 m crossing of the 2024 event.
Usage:
python scripts/backtest_render.py # uses FLOOD_ML_DB_URL/Config
python scripts/backtest_render.py # uses FLOOD_ML_DB_URL / POSTGRES_*
python scripts/backtest_render.py --db-url postgresql://...
"""
@@ -60,7 +60,11 @@ def fit_backtest_model(df_long: pd.DataFrame, train_end: str, use_dam: bool = Fa
rain_series = rain_mod.catchment_mean(rain_mod.load_history())
dam_frame = dam_mod.load_history() if use_dam else None
X, Y, _meta = features.build_matrix(
df_long, STATION, (HORIZON,), stats_end=train_end, rain=rain_series,
df_long,
STATION,
(HORIZON,),
stats_end=train_end,
rain=rain_series,
dam=dam_frame,
)
train_mask = X.index <= pd.Timestamp(train_end)
@@ -92,6 +96,7 @@ def event_series(df_long, X, reg, clf, window_start: str, window_end: str):
# whichever is more alarmed. The classifier alone proved unreliable on
# out-of-distribution extremes (silent on the 2024 record flood).
import numpy as np
p_clf = clf.predict_proba(Xw)[:, 1]
p_sig = 1.0 / (1.0 + np.exp(-(forecasts["pred_max"] - STAGE1) / 0.15))
forecasts["p_flood"] = np.maximum(p_clf, p_sig)
@@ -109,55 +114,107 @@ def _style_axes(ax):
ax.set_axisbelow(True)
def render(observed, forecasts, flood_start, first_alert, out_path, *,
title, subtitle, detail=False, show_stage7=False, peak_note=None):
def render(
observed,
forecasts,
flood_start,
first_alert,
out_path,
*,
title,
subtitle,
detail=False,
show_stage7=False,
peak_note=None,
):
fig, (ax, axp) = plt.subplots(
2, 1, figsize=(12.6, 7.6), sharex=True,
2,
1,
figsize=(12.6, 7.6),
sharex=True,
gridspec_kw={"height_ratios": [2.2, 1], "hspace": 0.12},
)
fig.patch.set_facecolor("white")
marker = dict(marker="o", markersize=3) if detail else {}
ax.plot(observed.index, observed.values, color=BLUE, linewidth=2.2,
label="Observed level" + (" (hourly)" if detail else ""), **marker)
ax.plot(
observed.index,
observed.values,
color=BLUE,
linewidth=2.2,
label="Observed level" + (" (hourly)" if detail else ""),
**marker,
)
marker = dict(marker="s", markersize=3) if detail else {}
ax.plot(forecasts.index, forecasts["pred_max"], color=AMBER, linewidth=2,
linestyle="--", label="Predicted 24 h peak (issued at that hour)", **marker)
ax.plot(
forecasts.index,
forecasts["pred_max"],
color=AMBER,
linewidth=2,
linestyle="--",
label="Predicted 24 h peak (issued at that hour)",
**marker,
)
ax.axhline(STAGE1, color=RED, linewidth=1, alpha=0.65)
ax.annotate(f"{STAGE1:.2f} m · stage 1 · flooding begins", xy=(0.06, STAGE1),
xycoords=("axes fraction", "data"), xytext=(0, 5),
textcoords="offset points", color=RED, fontsize=10.5)
ax.annotate(
f"{STAGE1:.2f} m · stage 1 · flooding begins",
xy=(0.06, STAGE1),
xycoords=("axes fraction", "data"),
xytext=(0, 5),
textcoords="offset points",
color=RED,
fontsize=10.5,
)
if show_stage7:
ax.axhline(STAGE7, color=RED, linewidth=1, alpha=0.65)
ax.annotate(f"{STAGE7:.2f} m · stage 7 · widespread", xy=(0.06, STAGE7),
xycoords=("axes fraction", "data"), xytext=(0, 5),
textcoords="offset points", color=RED, fontsize=10.5)
ax.annotate(
f"{STAGE7:.2f} m · stage 7 · widespread",
xy=(0.06, STAGE7),
xycoords=("axes fraction", "data"),
xytext=(0, 5),
textcoords="offset points",
color=RED,
fontsize=10.5,
)
if peak_note:
peak_ts = observed.idxmax()
ax.annotate(peak_note, xy=(peak_ts, observed.max()),
xytext=(12, 10), textcoords="offset points",
color=BLUE, fontsize=11.5, fontweight="bold")
ax.annotate(
peak_note,
xy=(peak_ts, observed.max()),
xytext=(12, 10),
textcoords="offset points",
color=BLUE,
fontsize=11.5,
fontweight="bold",
)
ax.set_ylabel("P.1 water level (m)", color=INK, fontsize=11.5)
ax.legend(loc="upper left", frameon=False, fontsize=10.5)
_style_axes(ax)
axp.plot(forecasts.index, forecasts["p_flood"], color=AMBER, linewidth=1.8)
axp.fill_between(forecasts.index, 0, forecasts["p_flood"],
color=AMBER, alpha=0.28)
axp.fill_between(forecasts.index, 0, forecasts["p_flood"], color=AMBER, alpha=0.28)
axp.axhline(0.5, color=INK, linewidth=0.9, linestyle=":", alpha=0.6)
axp.set_ylim(-0.02, 1.1)
axp.set_ylabel(f"P(flooding within {HORIZON} h)", color=INK, fontsize=11.5)
_style_axes(axp)
if first_alert is not None:
lead_h = None if flood_start is None else \
int((flood_start - first_alert).total_seconds() // 3600)
lead_txt = "" if lead_h is None else (
f"\n({lead_h} h before flooding began)" if lead_h >= 0
else f"\n({-lead_h} h after flooding began)"
lead_h = (
None
if flood_start is None
else int((flood_start - first_alert).total_seconds() // 3600)
)
lead_txt = (
""
if lead_h is None
else (
f"\n({lead_h} h before flooding began)"
if lead_h >= 0
else f"\n({-lead_h} h after flooding began)"
)
)
if detail and flood_start is not None:
for a in (ax, axp):
@@ -166,34 +223,61 @@ def render(observed, forecasts, flood_start, first_alert, out_path, *,
# Anchor labels away from each other in chronological order so a
# late alert (alert AFTER crossing) cannot overprint the labels.
events = sorted(
[(first_alert, "model alert", AMBER), (flood_start, "flooding begins", BLUE)]
[
(first_alert, "model alert", AMBER),
(flood_start, "flooding begins", BLUE),
]
)
for (ts, label, color), (offset, align) in zip(events, ((-8, "right"), (8, "left"))):
ax.annotate(f"{label}\n{ts:%d %b %H:%M}",
xy=(ts, observed.min()), xytext=(offset, 18),
textcoords="offset points", ha=align,
color=color, fontsize=11, fontweight="bold")
for (ts, label, color), (offset, align) in zip(
events, ((-8, "right"), (8, "left"))
):
ax.annotate(
f"{label}\n{ts:%d %b %H:%M}",
xy=(ts, observed.min()),
xytext=(offset, 18),
textcoords="offset points",
ha=align,
color=color,
fontsize=11,
fontweight="bold",
)
mid_y = observed.min() + (observed.max() - observed.min()) * 0.28
ax.annotate("", xy=(flood_start, mid_y), xytext=(first_alert, mid_y),
arrowprops=dict(arrowstyle="<->", color=INK, lw=1.3))
ax.annotate(
"",
xy=(flood_start, mid_y),
xytext=(first_alert, mid_y),
arrowprops=dict(arrowstyle="<->", color=INK, lw=1.3),
)
arrow_label = (
f"{lead_h} h warning" if lead_h >= 0 else f"alert {-lead_h} h late"
)
ax.annotate(arrow_label,
xy=(first_alert + (flood_start - first_alert) / 2, mid_y),
xytext=(0, 8), textcoords="offset points", ha="center",
color=INK, fontsize=11.5, fontweight="bold")
ax.annotate(
arrow_label,
xy=(first_alert + (flood_start - first_alert) / 2, mid_y),
xytext=(0, 8),
textcoords="offset points",
ha="center",
color=INK,
fontsize=11.5,
fontweight="bold",
)
else:
axp.annotate(f"first alert · {first_alert:%d %b %H:%M}{lead_txt}",
xy=(first_alert, 0.62), xytext=(10, 0),
textcoords="offset points", color=RED, fontsize=10.5,
bbox=dict(facecolor="white", alpha=0.75, edgecolor="none"))
axp.annotate(
f"first alert · {first_alert:%d %b %H:%M}{lead_txt}",
xy=(first_alert, 0.62),
xytext=(10, 0),
textcoords="offset points",
color=RED,
fontsize=10.5,
bbox=dict(facecolor="white", alpha=0.75, edgecolor="none"),
)
locator = mdates.DayLocator(interval=1 if detail else 3)
axp.xaxis.set_major_locator(locator)
axp.xaxis.set_major_formatter(mdates.DateFormatter("%d %b"))
fig.suptitle(f"{title}\n{subtitle}", x=0.07, y=0.985, ha="left",
fontsize=15, color=INK)
fig.suptitle(
f"{title}\n{subtitle}", x=0.07, y=0.985, ha="left", fontsize=15, color=INK
)
fig.subplots_adjust(top=0.885, left=0.07, right=0.97, bottom=0.07)
fig.savefig(out_path, dpi=110)
plt.close(fig)
@@ -204,16 +288,20 @@ def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--db-url", default=None)
parser.add_argument("--out-dir", default=os.path.join("docs", "img"))
parser.add_argument("--dam", action="store_true",
help="ablation: include Mae Ngat reservoir features "
"(2026-08 result: costs 1-3 h of alert lead)")
parser.add_argument("--no-hii-fill", action="store_true",
help="ablation: load without the HII gap-fill merge")
parser.add_argument(
"--dam",
action="store_true",
help="ablation: include Mae Ngat reservoir features "
"(2026-08 result: costs 1-3 h of alert lead)",
)
parser.add_argument(
"--no-hii-fill",
action="store_true",
help="ablation: load without the HII gap-fill merge",
)
args = parser.parse_args(argv)
df = data.load_measurements(
db_url=args.db_url, hii_fill=not args.no_hii_fill
)
df = data.load_measurements(db_url=args.db_url, hii_fill=not args.no_hii_fill)
if df.empty:
print("no measurement data available", file=sys.stderr)
return 1
@@ -222,41 +310,66 @@ def main(argv=None) -> int:
# --- October 2024 record flood: trained only on data before 1 Sep 2024 ---
X, reg, clf = fit_backtest_model(df, "2024-08-31", use_dam=args.dam)
obs, fc, flood_start, first_alert = event_series(
df, X, reg, clf, "2024-09-10", "2024-10-14 23:00")
df, X, reg, clf, "2024-09-10", "2024-10-14 23:00"
)
peak = float(obs.max())
render(obs, fc, flood_start, first_alert,
os.path.join(args.out_dir, "backtest-2024-p1.png"),
title="October 2024 flood: what the model saw coming",
subtitle="P.1 Nawarat Bridge — model trained only on data before 1 Sep 2024",
show_stage7=True, peak_note=f"record peak {peak:.2f} m")
render(
obs,
fc,
flood_start,
first_alert,
os.path.join(args.out_dir, "backtest-2024-p1.png"),
title="October 2024 flood: what the model saw coming",
subtitle="P.1 Nawarat Bridge — model trained only on data before 1 Sep 2024",
show_stage7=True,
peak_note=f"record peak {peak:.2f} m",
)
obs_d, fc_d, flood_d, alert_d = event_series(
df, X, reg, clf, "2024-09-21 18:00", "2024-09-28 06:00")
df, X, reg, clf, "2024-09-21 18:00", "2024-09-28 06:00"
)
lead_h = None
if alert_d is not None and flood_d is not None:
lead_h = int((flood_d - alert_d).total_seconds() // 3600)
render(obs_d, fc_d, flood_d, alert_d,
os.path.join(args.out_dir, "backtest-2024-p1-detail.png"),
title="Detection in detail: 22–28 September 2024, hour by hour",
subtitle=(
f"the model alerts {lead_h} h before the river crosses the flooding line"
if lead_h is not None and lead_h > 0
else "model alert vs the river crossing the flooding line"
),
detail=True)
render(
obs_d,
fc_d,
flood_d,
alert_d,
os.path.join(args.out_dir, "backtest-2024-p1-detail.png"),
title="Detection in detail: 22–28 September 2024, hour by hour",
subtitle=(
f"the model alerts {lead_h} h before the river crosses the flooding line"
if lead_h is not None and lead_h > 0
else "model alert vs the river crossing the flooding line"
),
detail=True,
)
# --- September 2025 flood: the deployed configuration (trained <= 2024) ---
X25, reg25, clf25 = fit_backtest_model(df, "2024-12-31", use_dam=args.dam)
obs25, fc25, flood25, alert25 = event_series(
df, X25, reg25, clf25, "2025-09-22", "2025-10-02 12:00")
pred_at_alert = float(fc25.loc[alert25:, "pred_max"].iloc[:24].max()) if alert25 is not None else None
df, X25, reg25, clf25, "2025-09-22", "2025-10-02 12:00"
)
pred_at_alert = (
float(fc25.loc[alert25:, "pred_max"].iloc[:24].max())
if alert25 is not None
else None
)
note = f"peak {float(obs25.max()):.2f} m" + (
f" (predicted {pred_at_alert:.2f} m)" if pred_at_alert is not None else "")
render(obs25, fc25, flood25, alert25,
os.path.join(args.out_dir, "backtest-2025-p1.png"),
title="The September 2025 flood — as forecast by the deployed configuration",
subtitle="model trained only on data through 2024; this event was never seen in training",
detail=True, peak_note=note)
f" (predicted {pred_at_alert:.2f} m)" if pred_at_alert is not None else ""
)
render(
obs25,
fc25,
flood25,
alert25,
os.path.join(args.out_dir, "backtest-2025-p1.png"),
title="The September 2025 flood — as forecast by the deployed configuration",
subtitle="model trained only on data through 2024; this event was never seen in training",
detail=True,
peak_note=note,
)
print(f"2024: flooding began {flood_start}, first alert {first_alert}")
print(f"2025: flooding began {flood25}, first alert {alert25}")