chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
+25
-10
@@ -1,9 +1,17 @@
|
||||
# A Rust rewrite: assessment
|
||||
# The Rust rewrite
|
||||
|
||||
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17,
|
||||
phase 2 (forecast + ntfy) and phase 3 (website/API + Matrix) since
|
||||
2026-10-18; see
|
||||
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
|
||||
Status: **done** (2026-10-18). The service is `ping-collector`
|
||||
([rust/README.md](../rust/README.md)); Python keeps model training,
|
||||
evaluation and the archive backfills. Phases 1-3 replaced the Python
|
||||
collector, the forecast + notifier and the website/API one at a time, each
|
||||
checked against the Python side on production data before the switch;
|
||||
phase 4 removed the Python service code. The migration tooling
|
||||
(`scripts/collector_parity.py`, `scripts/api_parity.py`,
|
||||
`scripts/install_collector.sh` with its cutover/rollback modes, the
|
||||
`spikes/rust-forecast` spike) is in git history before the cleanup commit.
|
||||
|
||||
The assessment below was written before phase 1 and is kept as the record of
|
||||
why and how.
|
||||
|
||||
## Short version
|
||||
|
||||
@@ -87,7 +95,7 @@ This slows model iteration, which is the most active part of the repo.
|
||||
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
|
||||
| 2 | Forecast crate + ntfy | in production since 2026-10-18 15:08 (`FORECAST_IN_COLLECTOR=true`); live check on production data: 48/48 rows identical, 96/96 probabilities bit-identical |
|
||||
| 3 | Website + read API in `ping-collector`, Matrix like ntfy | in production since 2026-10-18 16:40 (`WEB_LISTEN`, `water-monitor.service` disabled); on production data 53 of 53 compared routes identical to FastAPI (`scripts/api_parity.py`) |
|
||||
| 4 | Cleanup | |
|
||||
| 4 | Cleanup: Python service code removed | done 2026-10-18 |
|
||||
|
||||
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
|
||||
1. **Collector daemon** (`rust/collector`, ~1.5k LOC incl. tests): RID with the
|
||||
@@ -96,7 +104,7 @@ This slows model iteration, which is the most active part of the repo.
|
||||
API runs with `COLLECTION_ENABLED=false` and re-forecasts when the newest
|
||||
RID reading in the database advances. ntfy moved to phase 2: the outlook
|
||||
alert needs the forecasts, which are still Python. Cutover and rollback
|
||||
are one command each (`scripts/install_collector.sh`).
|
||||
were one command each (`scripts/install_collector.sh`, since removed).
|
||||
2. **Forecast crate + ntfy** (`rust/forecast`, run by `ping-collector` with
|
||||
`FORECAST_IN_COLLECTOR=true`): all stations, HII/rain/dam inputs, heuristic
|
||||
fallback, probabilities and stages, `forecast_history`, `forecast_current`
|
||||
@@ -130,9 +138,16 @@ This slows model iteration, which is the most active part of the repo.
|
||||
their own `matrix:` state keys seeded from ntfy's, replacing the
|
||||
`run.py --alert-check` cron job that repeated alerts every 15 minutes.
|
||||
Python keeps training, evaluation and the monthly retrain timer.
|
||||
4. **Cleanup:** drop FastAPI/uvicorn, the Python collector, notifier, CLI and
|
||||
the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) and
|
||||
demo databases; the Python environment shrinks to the training stack.
|
||||
4. **Cleanup** (done): FastAPI/uvicorn, the Python collector, notifier, Matrix
|
||||
alerting, CLI (`run.py`), the multi-backend adapters (SQLite, MySQL,
|
||||
InfluxDB, VictoriaMetrics), demo databases, Docker/Grafana files and the
|
||||
migration scripts are gone, about 17k lines. The Python environment is the
|
||||
training stack (pandas, scikit-learn, SQLAlchemy/psycopg2): `src/ml`, the
|
||||
backfill clients/stores (`src/hii_collector.py`, `src/rid_reservoir.py`)
|
||||
and `src/ml/predict.py` as the reference implementation the Rust forecaster
|
||||
is tested against. PostgreSQL only; the Python database tests run against
|
||||
a throwaway schema (`TEST_DATABASE_URL`, as the Rust ones). One installer,
|
||||
`scripts/install.sh`.
|
||||
|
||||
Decided before phase 3 (2026-10-18): Matrix stays and behaves like ntfy;
|
||||
`/docs` stays; SQLite dev mode goes; the monthly retrain stays on the same
|
||||
|
||||
Reference in new issue
Block a user