B4L/Northern-Thailand-Ping-River-Monitor

0 Fork 0

Files

T

grabowski 7e642555b4 chore: remove the Python service; Python is the training stack only

Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.

2026-10-18 17:15:07 +07:00

2.5 KiB

Raw Blame History

Contributing

Issues and merge requests on git.b4l.co.th. The repository has two halves: the service in Rust (rust/, see rust/README.md) and the model training/evaluation stack in Python (src/ml, see docs/FLOOD_FORECASTING.md).

Setup

Python 3.11 and uv (3.12+ has no pandas 2.0.3 wheels)
Rust >= 1.88 from rustup, plus pkg-config and libssl-dev on Linux
Docker (or any PostgreSQL 16) for the database tests

git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
cd Northern-Thailand-Ping-River-Monitor
make install                     # uv sync + pre-commit hooks
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
export TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping
make check                       # format check, flake8, clippy, pytest, cargo test, Rust 1.88

Without TEST_DATABASE_URL the database tests are skipped, the rest still run.

Before a merge request

make format, then make check green. CI runs the same (.gitea/workflows/).
One logical change per commit, imperative subject (fix(collector): ..., feat(web): ..., docs: ...).
Dashboard changes (src/static/dashboard.html): every visible string in English and Thai (t() table); a screenshot in the merge request.
API changes: keep existing fields and formats; external clients read them. New parameters go into the route's declaration in rust/collector/src/web/params.rs, which also documents them at /docs.
Model changes: evaluate with scripts/evaluate_variants.py (rolling origin, judged on first-alert lead and false alarms) and record the result in docs/FLOOD_FORECASTING.md section 5, whether it helped or not. A feature change must be ported to rust/forecast in the same change (see CLAUDE.md for the fixture and config regeneration steps).
Never commit credentials, .env, trained models (models/*.joblib, models/*.json.gz) or the server's private addresses.

Useful commands

make build && make serve          # the website/API from your database on 127.0.0.1:8001
uv run python scripts/dev_proxy.py  # the working-copy dashboard against the live API
uv run python scripts/evaluate_variants.py --stations P.1
uv run python -m src.ml.export --config   # after changing thresholds or notifier settings

2.5 KiB Raw Blame History

Contributing

Setup

Before a merge request

Useful commands

2.5 KiB

Raw Blame History