chore: remove the Python service; Python is the training stack only

Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558

No files matched your search

+57 -168
View File
@@ -1,181 +1,70 @@
# Northern Thailand Ping River Monitor - Makefile
# Ping River Monitor. The service is Rust (rust/, ping-collector); Python is
# the model training/evaluation stack and the archive backfills (src/).
# Database tests (both sides) need TEST_DATABASE_URL, e.g.
# docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
# export TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping
.PHONY: help install install-dev test test-coverage lint format clean run run-api docker-build docker-run docs
PY_SRC := src/ scripts/ tests/ rust/forecast/testdata rust/collector/testdata
CARGO := cargo
MANIFEST := --manifest-path rust/Cargo.toml
.PHONY: help install format lint test test-py test-rust build check serve train evaluate export clean
# Default target
help:
@echo "Northern Thailand Ping River Monitor - Available Commands:"
@echo ""
@echo "Setup:"
@echo " install Install production dependencies"
@echo " install-dev Install development dependencies"
@echo ""
@echo "Development:"
@echo " test Run all tests"
@echo " test-cov Run tests with coverage report"
@echo " lint Run code linting"
@echo " format Format code with black and isort"
@echo " clean Clean up temporary files"
@echo ""
@echo "Running:"
@echo " run Run the monitor in continuous mode"
@echo " run-api Run the web API server"
@echo " run-test Run a single test cycle"
@echo " run-status Show system status"
@echo ""
@echo "Alerting:"
@echo " alert-check Check water levels and send alerts"
@echo " alert-test Send test Matrix message"
@echo ""
@echo "Distribution:"
@echo " build-exe Build standalone executable"
@echo " package Build and create distribution package"
@echo ""
@echo "Docker:"
@echo " docker-build Build Docker image"
@echo " docker-run Run with Docker Compose"
@echo " docker-stop Stop Docker services"
@echo ""
@echo "Database:"
@echo " setup-postgres Setup PostgreSQL database"
@echo " test-postgres Test PostgreSQL connection"
@echo " encode-password URL encode password for connection string"
@echo " migrate-sqlite Migrate SQLite data to PostgreSQL"
@echo " migrate-fast Fast migration with 10K batch size"
@echo " analyze-sqlite Analyze SQLite database structure (dry run)"
@echo ""
@echo "Documentation:"
@echo " docs Generate documentation"
@echo "make install uv sync (Python 3.11) + pre-commit hooks"
@echo "make format black + isort (Python), cargo fmt (Rust)"
@echo "make lint the CI checks without tests (format, flake8, clippy)"
@echo "make test pytest + cargo test"
@echo "make build release build of ping-collector (rust/target/release)"
@echo "make check lint + test + the Rust 1.88 minimum-version check"
@echo "make serve website/API from the local database on 127.0.0.1:8001"
@echo "make train retrain every station's model into models/"
@echo "make evaluate rolling-origin harness on P.1 (see scripts/evaluate_variants.py)"
@echo "make export export trained models + flood_config.json for the Rust forecaster"
# Installation
install:
uv sync --no-dev
install-dev:
uv sync
uv sync --python 3.11
uv run pre-commit install
# Testing
test:
format:
uv run black $(PY_SRC)
uv run isort src/ scripts/ tests/
$(CARGO) fmt --all $(MANIFEST)
lint:
uv run black --check $(PY_SRC)
uv run isort --check-only src/ scripts/ tests/
uv run flake8 src/ scripts/ tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
$(CARGO) fmt --all --check $(MANIFEST)
$(CARGO) clippy --all-targets --locked $(MANIFEST) -- -D warnings
test: test-py test-rust
test-py:
uv run pytest -q
test-cov:
uv run pytest --cov=src --cov-report=html --cov-report=term
test-rust:
$(CARGO) test --locked $(MANIFEST)
# Code quality
lint:
uv run flake8 src/ --max-line-length=100
uv run mypy src/
build:
$(CARGO) build --release --locked $(MANIFEST)
format:
uv run black src/ *.py
uv run isort src/ *.py
check: lint test
CARGO_TARGET_DIR=rust/target/msrv $(CARGO) +1.88 check --all-targets --locked $(MANIFEST)
serve:
rust/target/release/ping-collector serve --listen 127.0.0.1:8001 --no-analytics
train:
uv run python scripts/train_flood_model.py --stations all
evaluate:
uv run python scripts/evaluate_variants.py --stations P.1
export:
uv run python -m src.ml.export
uv run python -m src.ml.export --config
# Cleanup
clean:
find . -type f -name "*.pyc" -delete
find . -type d -name "__pycache__" -delete
find . -type f -name "*.log" -delete
rm -rf .pytest_cache/
rm -rf .mypy_cache/
rm -rf htmlcov/
rm -rf dist/
rm -rf build/
rm -rf *.egg-info/
# Running
run:
uv run python run.py
run-api:
uv run python run.py --web-api
run-test:
uv run python run.py --test
run-status:
uv run python run.py --status
# Alerting
alert-check:
uv run python run.py --alert-check
alert-test:
uv run python run.py --alert-test
# Docker
docker-build:
docker build -t ping-river-monitor .
docker-run:
docker-compose -f docker-compose.victoriametrics.yml up -d
docker-stop:
docker-compose -f docker-compose.victoriametrics.yml down
docker-logs:
docker-compose -f docker-compose.victoriametrics.yml logs -f
# Documentation
docs:
cd docs && make html
# Monitoring
health-check:
curl -f http://localhost:8000/health || exit 1
metrics:
curl -s http://localhost:8000/metrics | jq .
# Development helpers
dev-setup: install-dev
cp .env.example .env
@echo "Development environment set up!"
@echo "Edit .env file with your configuration"
# Production deployment
deploy-check:
uv run python run.py --test
@echo "Deployment check passed!"
# Database management
setup-postgres:
uv run python scripts/setup_postgres.py
test-postgres:
uv run python -c "from scripts.setup_postgres import test_postgres_connection; from src.config import Config; config = Config.get_database_config(); test_postgres_connection(config['connection_string'])"
migrate-sqlite:
uv run python scripts/migrate_sqlite_to_postgres.py
migrate-fast:
uv run python scripts/migrate_sqlite_to_postgres.py --fast
analyze-sqlite:
uv run python scripts/migrate_sqlite_to_postgres.py --dry-run
# Git helpers
git-setup:
git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
@echo "Git remote configured!"
# Quick start
quick-start: install dev-setup run-test
@echo "Quick start completed!"
@echo "Run 'make run-api' to start the web interface"
# Gitea Actions
validate-workflows:
@echo "Validating Gitea Actions workflows..."
@for file in .gitea/workflows/*.yml; do \
echo "Checking $$file..."; \
uv run python -c "import yaml; yaml.safe_load(open('$$file', encoding='utf-8'))" || exit 1; \
done
@echo "✅ All workflows are valid"
workflow-test:
@echo "Testing workflow components locally..."
make test
make lint
make format
@echo "✅ Workflow test completed"
find . -type d -name "__pycache__" -prune -exec rm -rf {} +
rm -rf .pytest_cache/ htmlcov/ dist/ build/ *.egg-info/