grabowski
7e642555b4
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
86 lines
2.6 KiB
YAML
86 lines
2.6 KiB
YAML
name: CI
|
|
|
|
# The Python side (src/ml training/evaluation stack, archive backfills), on
|
|
# every push and PR to master:
|
|
# 1. formatting contract (black + isort, config in pyproject.toml)
|
|
# 2. flake8 hard-error gate (syntax, undefined names)
|
|
# 3. the pytest suite: synthetic data, no network; the database tests run
|
|
# against a throwaway PostgreSQL service (TEST_DATABASE_URL, as in rust.yml)
|
|
# The service itself (ping-collector) is checked by rust.yml. Production is a
|
|
# systemd unit deployed by `git pull` + scripts/install.sh on the server.
|
|
|
|
on:
|
|
push:
|
|
branches: [master, develop]
|
|
pull_request:
|
|
branches: [master]
|
|
schedule:
|
|
# daily, catches dependency drift
|
|
- cron: "0 2 * * *"
|
|
workflow_dispatch:
|
|
|
|
env:
|
|
PYTHON_VERSION: "3.11" # pandas 2.0.3 ships no 3.12 wheels; psycopg2-binary 2.9.9 breaks on 3.13
|
|
|
|
jobs:
|
|
lint:
|
|
name: Format & lint
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
cache: pip
|
|
cache-dependency-path: requirements-dev.txt
|
|
|
|
- name: Install tools
|
|
run: |
|
|
python -m pip install --upgrade pip --root-user-action=ignore
|
|
pip install --root-user-action=ignore black==26.5.1 isort==5.12.0 flake8==6.1.0
|
|
|
|
- name: black
|
|
run: black --check --diff src/ scripts/ tests/ rust/forecast/testdata rust/collector/testdata
|
|
|
|
- name: isort
|
|
run: isort --check-only --diff src/ scripts/ tests/
|
|
|
|
- name: flake8 (errors only)
|
|
run: flake8 src/ scripts/ tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
|
|
|
|
test:
|
|
name: Test suite
|
|
runs-on: ubuntu-latest
|
|
services:
|
|
postgres:
|
|
image: postgres:16
|
|
env:
|
|
POSTGRES_PASSWORD: ping
|
|
POSTGRES_DB: ping
|
|
options: >-
|
|
--health-cmd pg_isready
|
|
--health-interval 5s
|
|
--health-timeout 5s
|
|
--health-retries 20
|
|
env:
|
|
TEST_DATABASE_URL: postgres://postgres:ping@postgres:5432/ping
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
cache: pip
|
|
cache-dependency-path: |
|
|
requirements.txt
|
|
requirements-dev.txt
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
python -m pip install --upgrade pip --root-user-action=ignore
|
|
pip install --root-user-action=ignore -r requirements.txt pytest==9.1.1
|
|
|
|
- name: pytest
|
|
run: pytest -q -p no:cacheprovider -rs
|