chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
@@ -1,181 +1,70 @@
|
||||
# Northern Thailand Ping River Monitor - Makefile
|
||||
# Ping River Monitor. The service is Rust (rust/, ping-collector); Python is
|
||||
# the model training/evaluation stack and the archive backfills (src/).
|
||||
# Database tests (both sides) need TEST_DATABASE_URL, e.g.
|
||||
# docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
|
||||
# export TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping
|
||||
|
||||
.PHONY: help install install-dev test test-coverage lint format clean run run-api docker-build docker-run docs
|
||||
PY_SRC := src/ scripts/ tests/ rust/forecast/testdata rust/collector/testdata
|
||||
CARGO := cargo
|
||||
MANIFEST := --manifest-path rust/Cargo.toml
|
||||
|
||||
.PHONY: help install format lint test test-py test-rust build check serve train evaluate export clean
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Northern Thailand Ping River Monitor - Available Commands:"
|
||||
@echo ""
|
||||
@echo "Setup:"
|
||||
@echo " install Install production dependencies"
|
||||
@echo " install-dev Install development dependencies"
|
||||
@echo ""
|
||||
@echo "Development:"
|
||||
@echo " test Run all tests"
|
||||
@echo " test-cov Run tests with coverage report"
|
||||
@echo " lint Run code linting"
|
||||
@echo " format Format code with black and isort"
|
||||
@echo " clean Clean up temporary files"
|
||||
@echo ""
|
||||
@echo "Running:"
|
||||
@echo " run Run the monitor in continuous mode"
|
||||
@echo " run-api Run the web API server"
|
||||
@echo " run-test Run a single test cycle"
|
||||
@echo " run-status Show system status"
|
||||
@echo ""
|
||||
@echo "Alerting:"
|
||||
@echo " alert-check Check water levels and send alerts"
|
||||
@echo " alert-test Send test Matrix message"
|
||||
@echo ""
|
||||
@echo "Distribution:"
|
||||
@echo " build-exe Build standalone executable"
|
||||
@echo " package Build and create distribution package"
|
||||
@echo ""
|
||||
@echo "Docker:"
|
||||
@echo " docker-build Build Docker image"
|
||||
@echo " docker-run Run with Docker Compose"
|
||||
@echo " docker-stop Stop Docker services"
|
||||
@echo ""
|
||||
@echo "Database:"
|
||||
@echo " setup-postgres Setup PostgreSQL database"
|
||||
@echo " test-postgres Test PostgreSQL connection"
|
||||
@echo " encode-password URL encode password for connection string"
|
||||
@echo " migrate-sqlite Migrate SQLite data to PostgreSQL"
|
||||
@echo " migrate-fast Fast migration with 10K batch size"
|
||||
@echo " analyze-sqlite Analyze SQLite database structure (dry run)"
|
||||
@echo ""
|
||||
@echo "Documentation:"
|
||||
@echo " docs Generate documentation"
|
||||
@echo "make install uv sync (Python 3.11) + pre-commit hooks"
|
||||
@echo "make format black + isort (Python), cargo fmt (Rust)"
|
||||
@echo "make lint the CI checks without tests (format, flake8, clippy)"
|
||||
@echo "make test pytest + cargo test"
|
||||
@echo "make build release build of ping-collector (rust/target/release)"
|
||||
@echo "make check lint + test + the Rust 1.88 minimum-version check"
|
||||
@echo "make serve website/API from the local database on 127.0.0.1:8001"
|
||||
@echo "make train retrain every station's model into models/"
|
||||
@echo "make evaluate rolling-origin harness on P.1 (see scripts/evaluate_variants.py)"
|
||||
@echo "make export export trained models + flood_config.json for the Rust forecaster"
|
||||
|
||||
# Installation
|
||||
install:
|
||||
uv sync --no-dev
|
||||
|
||||
install-dev:
|
||||
uv sync
|
||||
uv sync --python 3.11
|
||||
uv run pre-commit install
|
||||
|
||||
# Testing
|
||||
test:
|
||||
format:
|
||||
uv run black $(PY_SRC)
|
||||
uv run isort src/ scripts/ tests/
|
||||
$(CARGO) fmt --all $(MANIFEST)
|
||||
|
||||
lint:
|
||||
uv run black --check $(PY_SRC)
|
||||
uv run isort --check-only src/ scripts/ tests/
|
||||
uv run flake8 src/ scripts/ tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
|
||||
$(CARGO) fmt --all --check $(MANIFEST)
|
||||
$(CARGO) clippy --all-targets --locked $(MANIFEST) -- -D warnings
|
||||
|
||||
test: test-py test-rust
|
||||
|
||||
test-py:
|
||||
uv run pytest -q
|
||||
|
||||
test-cov:
|
||||
uv run pytest --cov=src --cov-report=html --cov-report=term
|
||||
test-rust:
|
||||
$(CARGO) test --locked $(MANIFEST)
|
||||
|
||||
# Code quality
|
||||
lint:
|
||||
uv run flake8 src/ --max-line-length=100
|
||||
uv run mypy src/
|
||||
build:
|
||||
$(CARGO) build --release --locked $(MANIFEST)
|
||||
|
||||
format:
|
||||
uv run black src/ *.py
|
||||
uv run isort src/ *.py
|
||||
check: lint test
|
||||
CARGO_TARGET_DIR=rust/target/msrv $(CARGO) +1.88 check --all-targets --locked $(MANIFEST)
|
||||
|
||||
serve:
|
||||
rust/target/release/ping-collector serve --listen 127.0.0.1:8001 --no-analytics
|
||||
|
||||
train:
|
||||
uv run python scripts/train_flood_model.py --stations all
|
||||
|
||||
evaluate:
|
||||
uv run python scripts/evaluate_variants.py --stations P.1
|
||||
|
||||
export:
|
||||
uv run python -m src.ml.export
|
||||
uv run python -m src.ml.export --config
|
||||
|
||||
# Cleanup
|
||||
clean:
|
||||
find . -type f -name "*.pyc" -delete
|
||||
find . -type d -name "__pycache__" -delete
|
||||
find . -type f -name "*.log" -delete
|
||||
rm -rf .pytest_cache/
|
||||
rm -rf .mypy_cache/
|
||||
rm -rf htmlcov/
|
||||
rm -rf dist/
|
||||
rm -rf build/
|
||||
rm -rf *.egg-info/
|
||||
|
||||
# Running
|
||||
run:
|
||||
uv run python run.py
|
||||
|
||||
run-api:
|
||||
uv run python run.py --web-api
|
||||
|
||||
run-test:
|
||||
uv run python run.py --test
|
||||
|
||||
run-status:
|
||||
uv run python run.py --status
|
||||
|
||||
# Alerting
|
||||
alert-check:
|
||||
uv run python run.py --alert-check
|
||||
|
||||
alert-test:
|
||||
uv run python run.py --alert-test
|
||||
|
||||
# Docker
|
||||
docker-build:
|
||||
docker build -t ping-river-monitor .
|
||||
|
||||
docker-run:
|
||||
docker-compose -f docker-compose.victoriametrics.yml up -d
|
||||
|
||||
docker-stop:
|
||||
docker-compose -f docker-compose.victoriametrics.yml down
|
||||
|
||||
docker-logs:
|
||||
docker-compose -f docker-compose.victoriametrics.yml logs -f
|
||||
|
||||
# Documentation
|
||||
docs:
|
||||
cd docs && make html
|
||||
|
||||
# Monitoring
|
||||
health-check:
|
||||
curl -f http://localhost:8000/health || exit 1
|
||||
|
||||
metrics:
|
||||
curl -s http://localhost:8000/metrics | jq .
|
||||
|
||||
# Development helpers
|
||||
dev-setup: install-dev
|
||||
cp .env.example .env
|
||||
@echo "Development environment set up!"
|
||||
@echo "Edit .env file with your configuration"
|
||||
|
||||
# Production deployment
|
||||
deploy-check:
|
||||
uv run python run.py --test
|
||||
@echo "Deployment check passed!"
|
||||
|
||||
# Database management
|
||||
setup-postgres:
|
||||
uv run python scripts/setup_postgres.py
|
||||
|
||||
test-postgres:
|
||||
uv run python -c "from scripts.setup_postgres import test_postgres_connection; from src.config import Config; config = Config.get_database_config(); test_postgres_connection(config['connection_string'])"
|
||||
|
||||
migrate-sqlite:
|
||||
uv run python scripts/migrate_sqlite_to_postgres.py
|
||||
|
||||
migrate-fast:
|
||||
uv run python scripts/migrate_sqlite_to_postgres.py --fast
|
||||
|
||||
analyze-sqlite:
|
||||
uv run python scripts/migrate_sqlite_to_postgres.py --dry-run
|
||||
|
||||
# Git helpers
|
||||
git-setup:
|
||||
git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
|
||||
@echo "Git remote configured!"
|
||||
|
||||
# Quick start
|
||||
quick-start: install dev-setup run-test
|
||||
@echo "Quick start completed!"
|
||||
@echo "Run 'make run-api' to start the web interface"
|
||||
|
||||
# Gitea Actions
|
||||
validate-workflows:
|
||||
@echo "Validating Gitea Actions workflows..."
|
||||
@for file in .gitea/workflows/*.yml; do \
|
||||
echo "Checking $$file..."; \
|
||||
uv run python -c "import yaml; yaml.safe_load(open('$$file', encoding='utf-8'))" || exit 1; \
|
||||
done
|
||||
@echo "✅ All workflows are valid"
|
||||
|
||||
workflow-test:
|
||||
@echo "Testing workflow components locally..."
|
||||
make test
|
||||
make lint
|
||||
make format
|
||||
@echo "✅ Workflow test completed"
|
||||
find . -type d -name "__pycache__" -prune -exec rm -rf {} +
|
||||
rm -rf .pytest_cache/ htmlcov/ dist/ build/ *.egg-info/
|
||||
Reference in new issue
Block a user