chore: remove the Python service; Python is the training stack only

Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558

No files matched your search

+25 -10
View File
@@ -1,9 +1,17 @@
# A Rust rewrite: assessment
# The Rust rewrite
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17,
phase 2 (forecast + ntfy) and phase 3 (website/API + Matrix) since
2026-10-18; see
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
Status: **done** (2026-10-18). The service is `ping-collector`
([rust/README.md](../rust/README.md)); Python keeps model training,
evaluation and the archive backfills. Phases 1-3 replaced the Python
collector, the forecast + notifier and the website/API one at a time, each
checked against the Python side on production data before the switch;
phase 4 removed the Python service code. The migration tooling
(`scripts/collector_parity.py`, `scripts/api_parity.py`,
`scripts/install_collector.sh` with its cutover/rollback modes, the
`spikes/rust-forecast` spike) is in git history before the cleanup commit.
The assessment below was written before phase 1 and is kept as the record of
why and how.
## Short version
@@ -87,7 +95,7 @@ This slows model iteration, which is the most active part of the repo.
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
| 2 | Forecast crate + ntfy | in production since 2026-10-18 15:08 (`FORECAST_IN_COLLECTOR=true`); live check on production data: 48/48 rows identical, 96/96 probabilities bit-identical |
| 3 | Website + read API in `ping-collector`, Matrix like ntfy | in production since 2026-10-18 16:40 (`WEB_LISTEN`, `water-monitor.service` disabled); on production data 53 of 53 compared routes identical to FastAPI (`scripts/api_parity.py`) |
| 4 | Cleanup | |
| 4 | Cleanup: Python service code removed | done 2026-10-18 |
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
1. **Collector daemon** (`rust/collector`, ~1.5k LOC incl. tests): RID with the
@@ -96,7 +104,7 @@ This slows model iteration, which is the most active part of the repo.
API runs with `COLLECTION_ENABLED=false` and re-forecasts when the newest
RID reading in the database advances. ntfy moved to phase 2: the outlook
alert needs the forecasts, which are still Python. Cutover and rollback
are one command each (`scripts/install_collector.sh`).
were one command each (`scripts/install_collector.sh`, since removed).
2. **Forecast crate + ntfy** (`rust/forecast`, run by `ping-collector` with
`FORECAST_IN_COLLECTOR=true`): all stations, HII/rain/dam inputs, heuristic
fallback, probabilities and stages, `forecast_history`, `forecast_current`
@@ -130,9 +138,16 @@ This slows model iteration, which is the most active part of the repo.
their own `matrix:` state keys seeded from ntfy's, replacing the
`run.py --alert-check` cron job that repeated alerts every 15 minutes.
Python keeps training, evaluation and the monthly retrain timer.
4. **Cleanup:** drop FastAPI/uvicorn, the Python collector, notifier, CLI and
the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) and
demo databases; the Python environment shrinks to the training stack.
4. **Cleanup** (done): FastAPI/uvicorn, the Python collector, notifier, Matrix
alerting, CLI (`run.py`), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics), demo databases, Docker/Grafana files and the
migration scripts are gone, about 17k lines. The Python environment is the
training stack (pandas, scikit-learn, SQLAlchemy/psycopg2): `src/ml`, the
backfill clients/stores (`src/hii_collector.py`, `src/rid_reservoir.py`)
and `src/ml/predict.py` as the reference implementation the Rust forecaster
is tested against. PostgreSQL only; the Python database tests run against
a throwaway schema (`TEST_DATABASE_URL`, as the Rust ones). One installer,
`scripts/install.sh`.
Decided before phase 3 (2026-10-18): Matrix stays and behaves like ntfy;
`/docs` stays; SQLite dev mode goes; the monthly retrain stays on the same