B4L/Northern-Thailand-Ping-River-Monitor
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

chore: remove the Python service; Python is the training stack only

Browse Source 
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/test_flood_forecast.py
+79 -12
View File
@@ -34,7 +34,9 @@ def make_synth(
level = base + noise
for pulse_start, width, peak_add in (pulses or {}).get(station, []):
t = np.arange(n_hours)
bump = peak_add * np.exp(-0.5 * ((t - (pulse_start + width / 2)) / (width / 4)) ** 2)
bump = peak_add * np.exp(
-0.5 * ((t - (pulse_start + width / 2)) / (width / 4)) ** 2
)
level = level + bump
discharge = 20.0 * level + rng.normal(0, 1.0, n_hours)
@@ -107,7 +109,12 @@ def test_label_coverage_gate():
idx = pd.date_range("2020-01-01", periods=n, freq="h")
levels = [1.0] * n
df = pd.DataFrame(
{"timestamp": idx, "station_code": "P.1", "water_level": levels, "discharge": [20.0] * n}
{
"timestamp": idx,
"station_code": "P.1",
"water_level": levels,
"discharge": [20.0] * n,
}
)
# Drop 70% of a future window (hours 21..26) for the row at t=20, no exceedance in it.
df_missing = df[~df["timestamp"].isin(idx[21:26])].copy()
@@ -119,7 +126,14 @@ def test_label_coverage_gate():
# Same sparse window, but WITH an observed exceedance inside it -> must be 1, not NaN.
df_with_peak = df_missing.copy()
peak_row = pd.DataFrame(
[{"timestamp": idx[22], "station_code": "P.1", "water_level": 5.0, "discharge": 100.0}]
[
{
"timestamp": idx[22],
"station_code": "P.1",
"water_level": 5.0,
"discharge": 100.0,
}
]
)
df_with_peak = pd.concat([df_with_peak, peak_row], ignore_index=True)
grid2 = features.make_hourly_grid(df_with_peak)
@@ -193,11 +207,20 @@ def test_train_smoke_and_roundtrip(tmp_path):
data_stations = ["P.1"] + upstream
target_stations = ["P.1", "P.20"]
n = 700
pulses = {station: [(start, 20, 2.0) for start in range(50, n - 50, 110)] for station in data_stations}
pulses = {
station: [(start, 20, 2.0) for start in range(50, n - 50, 110)]
for station in data_stations
}
df = make_synth(n, data_stations, seed=7, pulses=pulses)
metrics = train.train_all(
df, target_stations, models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 20}, use_rain=False, use_dam=False
df,
target_stations,
models_dir=tmp_path,
skip_eval=True,
hgb_overrides={"max_iter": 20},
use_rain=False,
use_dam=False,
)
assert metrics["stations"]["P.1"]["status"] == "trained"
assert metrics["stations"]["P.20"]["status"] == "trained"
@@ -220,7 +243,9 @@ def test_train_smoke_and_roundtrip(tmp_path):
def test_heuristic_fallback(tmp_path):
df = make_synth(50, ["P.1"], seed=3)
readings_by_station = {"P.1": df[["timestamp", "water_level", "discharge"]].to_dict("records")}
readings_by_station = {
"P.1": df[["timestamp", "water_level", "discharge"]].to_dict("records")
}
now = df["timestamp"].max()
forecasts = predict.get_forecasts(readings_by_station, models_dir=tmp_path, now=now)
@@ -235,7 +260,9 @@ def test_heuristic_fallback(tmp_path):
def _p1_synth(n: int = 300, seed: int = 11) -> pd.DataFrame:
upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
return make_synth(n, ["P.1"] + upstream, seed=seed, pulses={"P.1": [(100, 20, 2.0)]})
return make_synth(
n, ["P.1"] + upstream, seed=seed, pulses={"P.1": [(100, 20, 2.0)]}
)
def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
@@ -252,7 +279,15 @@ def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
# Case 1: the loader returns None (archive unreachable, no cache file)
monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: None)
with pytest.raises(train.RainUnavailableError, match="--no-rain"):
train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides=overrides,
use_rain=True,
use_dam=False,
)
assert not (tmp_path / "flood_P.1.joblib").exists()
assert not (tmp_path / "metrics.json").exists()
@@ -262,11 +297,27 @@ def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
monkeypatch.setattr(rain_mod, "load_history", boom)
with pytest.raises(train.RainUnavailableError, match="simulated Open-Meteo outage"):
train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides=overrides,
use_rain=True,
use_dam=False,
)
assert not (tmp_path / "flood_P.1.joblib").exists()
# Explicit opt-out still produces v2 bundles as before
metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=False, use_dam=False)
metrics = train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides=overrides,
use_rain=False,
use_dam=False,
)
assert metrics["model_version"].startswith("hgb-v2+")
assert (tmp_path / "flood_P.1.joblib").exists()
@@ -279,7 +330,15 @@ def test_train_with_rain_series_yields_v3(tmp_path, monkeypatch):
fake_rain = pd.DataFrame({"a": np.linspace(0, 1, len(idx)), "b": 0.5}, index=idx)
monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: fake_rain)
metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 10}, use_rain=True, use_dam=False)
metrics = train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides={"max_iter": 10},
use_rain=True,
use_dam=False,
)
assert metrics["model_version"].startswith("hgb-v3+")
bundle = joblib.load(tmp_path / "flood_P.1.joblib")
assert set(features.RAIN_FEATURES) <= set(bundle["feature_names"])
@@ -306,7 +365,15 @@ def test_feature_name_stability(tmp_path):
upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
data_stations = ["P.1"] + upstream
df = make_synth(300, data_stations, seed=11, pulses={"P.1": [(100, 20, 2.0)]})
train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 10}, use_rain=False, use_dam=False)
train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides={"max_iter": 10},
use_rain=False,
use_dam=False,
)
# Safe: loading the bundle this same test just wrote to tmp_path, not an external file.
bundle = joblib.load(tmp_path / "flood_P.1.joblib")