chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
+46
-48
@@ -62,8 +62,8 @@ P.5 before 2022-01-01; the rest are handled by the per-row coverage gates.
|
||||
|
||||
**How `src/ml/data.py` loads it.** `resolve_db_url()` picks a connection string in
|
||||
priority order: an explicit `--db-url` argument, then the `FLOOD_ML_DB_URL`
|
||||
environment variable, then `Config.get_database_config()` when `DB_TYPE` is
|
||||
`postgresql`, else `None`. `load_measurements()` then tries three tiers:
|
||||
environment variable, then the `POSTGRES_*` settings (`src/config.py`, the same
|
||||
`.env` the service reads), else `None`. `load_measurements()` then tries three tiers:
|
||||
|
||||
1. **PostgreSQL** (`_fetch_from_db`) — the primary path. NULL discharge stays
|
||||
NULL, which matters because the models must learn from the real missingness
|
||||
@@ -614,32 +614,28 @@ fix, the served features match what this section evaluates.
|
||||
|
||||
## 6. Deployment
|
||||
|
||||
### API
|
||||
### Serving
|
||||
|
||||
With `FORECAST_IN_COLLECTOR=true` the forecast is computed by `ping-collector`
|
||||
(`rust/forecast`, a parity-tested port of this serving path) from
|
||||
The forecast is computed by `ping-collector` (`rust/forecast`, a parity-tested
|
||||
port of `src/ml/predict.py`, which stays as the reference implementation) from
|
||||
`models/flood_<station>.json.gz`, which `train_all()` exports next to each
|
||||
joblib bundle; `/forecast` serves its `forecast_current` row and computes
|
||||
in-process only when that is older than `FORECAST_MAX_AGE_SECONDS` (2 h). See
|
||||
`rust/README.md`. The rest of this section describes the in-process path.
|
||||
joblib bundle. It runs after every collection cycle that brought newer
|
||||
readings and at least hourly, stores the rows in `forecast_history` (section 9's
|
||||
skill tracking) and `forecast_current`, and `GET /forecast` serves the stored
|
||||
row: one JSON row per station × horizon with the fields listed in section 1.
|
||||
A stored forecast older than `FORECAST_MAX_AGE_SECONDS` (2 h, i.e. the service
|
||||
has stopped forecasting) is still served, marked `X-Data-Stale: true`. See
|
||||
`rust/README.md`.
|
||||
|
||||
`GET /forecast` (`src/web_api.py`) returns one JSON row per station × horizon with
|
||||
the fields listed in section 1. Results are cached in-process for
|
||||
`FORECAST_TTL = 900` seconds (15 minutes), which matches the data cadence — the
|
||||
underlying readings do not update faster than hourly. Inference runs in a thread
|
||||
via `asyncio.to_thread` so it never blocks the event loop.
|
||||
A station without a model bundle, or whose bundle needs a feature the Rust side
|
||||
cannot build, gets the heuristic, as in Python. So **before the first training
|
||||
run `/forecast` returns 200 with an all-heuristic payload**, provided there is
|
||||
recent gauge data; you get an empty `200 []` only when there is no recent data
|
||||
at all. Judge deployment state by the `source` field, not the status code.
|
||||
|
||||
Failure modes: **503** if the `src.ml` package cannot be imported (missing
|
||||
scikit-learn, say), **502** on any other exception.
|
||||
|
||||
One behaviour worth knowing, because the code comments suggest otherwise: the
|
||||
endpoint's `FileNotFoundError` ("No trained flood models found") and `RuntimeError`
|
||||
handlers are unreachable — nothing in `src/ml/` raises either, and
|
||||
`predict._forecast_station` checks `bundle_path.exists()` and falls back to the
|
||||
heuristic instead. So **before the first training run `/forecast` returns 200 with
|
||||
an all-heuristic payload**, not a 503, provided there is recent gauge data; you
|
||||
get an empty `200 []` only when there is no recent data at all. Judge deployment
|
||||
state by the `source` field, not the status code.
|
||||
The Python serving path (`predict.get_latest_forecasts`) is not called by the
|
||||
service any more; `scripts/forecast_parity.py` runs both on the live database
|
||||
and `scripts/install.sh` does that on every install.
|
||||
|
||||
### Dashboard
|
||||
|
||||
@@ -661,13 +657,14 @@ The server already has the PostgreSQL connection configured, so no host override
|
||||
is needed:
|
||||
|
||||
```bash
|
||||
cd /path/to/Northern-Thailand-Ping-River-Monitor
|
||||
python scripts/train_flood_model.py --stations all
|
||||
cd /opt/thailand-water-monitor
|
||||
uv run python scripts/train_flood_model.py --stations all
|
||||
```
|
||||
|
||||
`resolve_db_url()` picks up `Config.get_database_config()` automatically when
|
||||
`DB_TYPE=postgresql`. The run writes fifteen `models/flood_{station}.joblib`
|
||||
bundles plus `models/metrics.json`.
|
||||
`resolve_db_url()` picks up the `POSTGRES_*` settings from `.env`. The run writes
|
||||
fifteen `models/flood_{station}.joblib` bundles, their `.json.gz` exports for the
|
||||
service, and `models/metrics.json`. The monthly timer does the same through
|
||||
`scripts/retrain.sh` with a staged promote (section 7).
|
||||
|
||||
### Artifacts and dependencies
|
||||
|
||||
@@ -733,7 +730,10 @@ gap between fetch and first model could not be reconstructed from the surviving
|
||||
artifacts. Do not attribute it to per-station training cost. Either way the
|
||||
conclusion holds: **retraining is minutes, not tens of minutes.**
|
||||
|
||||
**Inference** (15 bundles, 16 stations × 3 horizons = 48 rows):
|
||||
**Inference in Python** (15 bundles, 16 stations × 3 horizons = 48 rows; the
|
||||
retired in-process serving path, measured before the Rust forecaster took
|
||||
over: there the whole run takes milliseconds, 0.25 ms per station against
|
||||
158 ms, and the 15 exported models need about 70 MB resident):
|
||||
|
||||
| Measurement | Value |
|
||||
|---|---|
|
||||
@@ -753,9 +753,9 @@ one-time lazy `sklearn.ensemble` import — the remaining fourteen average 159 m
|
||||
Of the ~640 ms warm compute, model prediction is ~525 ms, the hourly grid 47 ms,
|
||||
and feature building 71 ms across all sixteen stations.
|
||||
|
||||
**Live-endpoint measurements** (a second, independent benchmark run against a real
|
||||
uvicorn instance of the app, same day, same workstation, RSS summed over the
|
||||
process tree):
|
||||
**Live-endpoint measurements of the retired Python API** (a second, independent
|
||||
benchmark run against a real uvicorn instance of the app, same day, same
|
||||
workstation, RSS summed over the process tree):
|
||||
|
||||
| Measurement | Value |
|
||||
|---|---|
|
||||
@@ -775,14 +775,11 @@ Training shows the opposite (2.3× slower single-threaded), so set the variable
|
||||
per process, never globally.
|
||||
|
||||
**Server sizing, in plain terms:** this is a small workload and almost any server
|
||||
runs it. **RAM is the binding constraint, not CPU.** Budget about **1 GB for the
|
||||
API process** so the ~335 MB steady state has headroom on top of the rest of the
|
||||
app; training peaks at only ~210–315 MB and can share the same box. No GPU
|
||||
anywhere. Pin thread counts per process — `OMP_NUM_THREADS=1` in the serving
|
||||
unit, `OMP_NUM_THREADS=4` for retraining so it cannot monopolise every core while
|
||||
the API is serving. And since a cold call costs seconds against a 2.4 ms cache
|
||||
hit, consider warming `/forecast` once at startup rather than letting a user
|
||||
absorb it.
|
||||
runs it. The service (`ping-collector`, Rust) stays under 100 MB resident with
|
||||
every model loaded (its unit caps it at 512 MB); the monthly retrain peaks at
|
||||
~210–315 MB and shares the same box. No GPU anywhere. The retrain unit sets
|
||||
`OMP_NUM_THREADS=4` so it cannot monopolise every core while the site is
|
||||
serving.
|
||||
|
||||
## 7. Retraining policy
|
||||
|
||||
@@ -821,15 +818,15 @@ fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
|
||||
time (`Persistent=true`, so a missed run catches up at boot). The unit runs
|
||||
`scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
|
||||
|
||||
1. trains all stations into `models/.staging/` (the API keeps serving the old
|
||||
bundles throughout);
|
||||
1. trains all stations into `models/.staging/` (the service keeps serving the
|
||||
old bundles throughout);
|
||||
2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
|
||||
least 14 trained stations (exit 3, staging discarded, old models untouched);
|
||||
3. renames the new bundles into `models/`, moving the previous generation to
|
||||
`models/.previous/` for rollback.
|
||||
|
||||
No API restart: `predict.py` reloads bundles by mtime on the next hourly
|
||||
precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
|
||||
No restart: ping-collector reloads changed `.json.gz` exports by mtime on its
|
||||
next forecast run. `systemctl list-timers water-monitor-retrain.timer` shows the next
|
||||
run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
|
||||
flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
|
||||
(`scripts/install.sh`) enables the timer.
|
||||
@@ -912,9 +909,10 @@ non-model-backed, and all of them are visible in the payload:
|
||||
horizon's head is in the bundle's `skipped_heads`, almost always because the
|
||||
station had fewer than 30 positive examples for that threshold.
|
||||
- A whole station flipping to heuristic after a code change — the feature-name
|
||||
check in `_model_forecast` caught a mismatch between the live feature builder
|
||||
and the stored `feature_names`. The fix is to retrain; the log line names the
|
||||
missing columns.
|
||||
check caught a mismatch between the feature builder and the stored
|
||||
`feature_names` (in Python `_model_forecast`; in Rust a bundle needing a
|
||||
feature `rust/forecast` cannot build). The fix is to retrain or to port the
|
||||
feature; the log line names the missing columns.
|
||||
|
||||
A live example from the 2026-08-10 cache: of 48 forecast rows, 42 came from
|
||||
models and 6 were heuristic — three for P.4A, which has no bundle by design, and
|
||||
|
||||
Reference in new issue
Block a user