chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed
1 parent
330fdf7804
commit
7e642555b4
125 files changed
+1300
-19558
No files matched your search
+63
-119
@@ -1,141 +1,85 @@
|
||||
# Northern Thailand Ping River Monitor Configuration
|
||||
# Copy this file to .env and customize for your environment
|
||||
# Ping River Monitor configuration: copy to .env and fill in.
|
||||
#
|
||||
# Read by ping-collector (the service; rust/collector/src/config.rs) and by the
|
||||
# Python training tools (src/config.py). Under systemd the variables come from
|
||||
# EnvironmentFile= (systemd's parser); for manual runs both programs load
|
||||
# ./.env with python-dotenv's rules. Unset keys take the defaults shown.
|
||||
|
||||
# Database Configuration
|
||||
DB_TYPE=postgresql
|
||||
# Options: sqlite, mysql, postgresql, influxdb, victoriametrics
|
||||
|
||||
# SQLite Configuration (default)
|
||||
WATER_DB_PATH=water_levels.db
|
||||
|
||||
# VictoriaMetrics Configuration
|
||||
VM_HOST=localhost
|
||||
VM_PORT=8428
|
||||
VM_URL=
|
||||
|
||||
# InfluxDB Configuration
|
||||
INFLUX_HOST=localhost
|
||||
INFLUX_PORT=8086
|
||||
INFLUX_DATABASE=ping_river_monitoring
|
||||
INFLUX_USERNAME=
|
||||
INFLUX_PASSWORD=
|
||||
|
||||
# PostgreSQL Configuration (Remote Server)
|
||||
# Option 1: Full connection string (URL encode special characters in password)
|
||||
POSTGRES_CONNECTION_STRING=postgresql://username:url_encoded_password@your-postgres-host:5432/water_monitoring
|
||||
|
||||
# Option 2: Individual components (password will be automatically URL encoded)
|
||||
POSTGRES_HOST=your-postgres-host
|
||||
# --- Database (PostgreSQL only) ---------------------------------------------
|
||||
# Either a full connection string (URL-encode special characters in the
|
||||
# password: : %3A @ %40 # %23 ? %3F & %26 / %2F % %25) ...
|
||||
POSTGRES_CONNECTION_STRING=
|
||||
# ... or the parts (the password is encoded for you)
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_DB=water_monitoring
|
||||
POSTGRES_USER=username
|
||||
POSTGRES_PASSWORD=your:password@with!special#chars
|
||||
POSTGRES_USER=postgres
|
||||
POSTGRES_PASSWORD=
|
||||
|
||||
# Examples for connection string:
|
||||
# - Local: postgresql://postgres:password@localhost:5432/water_monitoring
|
||||
# - Remote: postgresql://user:pass@192.168.1.100:5432/water_monitoring
|
||||
# - With special chars: postgresql://user:my%3Apass%40word@host:5432/db
|
||||
# - With SSL: postgresql://user:pass@host:port/db?sslmode=require
|
||||
# - Connection pooling: postgresql://user:pass@host:port/db?pool_size=20&max_overflow=0
|
||||
# --- What the service runs ---------------------------------------------------
|
||||
# Website and API: the address to listen on (empty = no website)
|
||||
WEB_LISTEN=0.0.0.0:8000
|
||||
# Forecast + ntfy/Matrix notifications after each collection (false = collect only)
|
||||
FORECAST_IN_COLLECTOR=true
|
||||
# Trained models (models/flood_<station>.json.gz from src/ml/export.py)
|
||||
# MODELS_DIR=models
|
||||
|
||||
# Special character URL encoding:
|
||||
# : → %3A @ → %40 # → %23 ? → %3F & → %26 / → %2F % → %25
|
||||
|
||||
# MySQL Configuration
|
||||
MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/ping_river_monitoring
|
||||
|
||||
# API Configuration
|
||||
API_HOST=0.0.0.0
|
||||
API_PORT=8000
|
||||
API_WORKERS=1
|
||||
# Public ThaiWater API key used to add Ping-basin water-level sensors.
|
||||
THAIWATER_API_KEY=
|
||||
|
||||
# Data Collection Settings
|
||||
SCRAPING_INTERVAL_HOURS=1
|
||||
# RID poll minute past the hour; retry interval while RID is behind
|
||||
# --- Collection ----------------------------------------------------------------
|
||||
# RID is polled this many minutes past each hour and retried every
|
||||
# RID_RETRY_MINUTES while the current hour is not published yet
|
||||
RID_POLL_OFFSET_MINUTES=5
|
||||
RID_RETRY_MINUTES=10
|
||||
# Days of RID holes the leader re-fetches at startup (0 = off)
|
||||
# RID reports with holes re-fetched at startup (0 = off); older: `ping-collector fill-gaps all`
|
||||
GAP_FILL_DAYS_ON_START=14
|
||||
# false when the Rust ping-collector service collects (rust/collector);
|
||||
# share the status file between both services for /scraping/status
|
||||
COLLECTION_ENABLED=true
|
||||
# SCRAPING_STATUS_FILE=/opt/thailand-water-monitor/collector-status.json
|
||||
# true: ping-collector also forecasts (models/flood_*.json.gz, rust/forecast)
|
||||
# and sends the ntfy notifications; this app serves its stored forecast and
|
||||
# computes itself only when that is older than FORECAST_MAX_AGE_SECONDS.
|
||||
# Set by scripts/install_collector.sh --cutover-forecast.
|
||||
FORECAST_IN_COLLECTOR=false
|
||||
# FORECAST_MAX_AGE_SECONDS=7200
|
||||
# MODELS_DIR=models
|
||||
REQUEST_TIMEOUT=30
|
||||
MAX_RETRIES=3
|
||||
RETRY_DELAY_SECONDS=60
|
||||
|
||||
# Data Retention
|
||||
DATA_RETENTION_DAYS=365
|
||||
|
||||
# Logging Configuration
|
||||
# ENABLE_HII_COLLECTION=true
|
||||
# ENABLE_RESERVOIR_COLLECTION=true
|
||||
# ENABLE_RAIN_COLLECTION=true
|
||||
# HII_BASIN_CODE=6
|
||||
# REQUEST_TIMEOUT=30
|
||||
# Runtime station list written by the admin API; falls back to src/data/stations.json
|
||||
# STATION_CONFIG_PATH=stations.json
|
||||
LOG_LEVEL=INFO
|
||||
LOG_FILE=water_monitor.log
|
||||
|
||||
# Security (for production)
|
||||
SECRET_KEY=your-secret-key-here
|
||||
API_KEY=your-api-key-here
|
||||
|
||||
# Monitoring
|
||||
ENABLE_METRICS=true
|
||||
ENABLE_HEALTH_CHECKS=true
|
||||
|
||||
# Geographic Settings
|
||||
TIMEZONE=Asia/Bangkok
|
||||
DEFAULT_LATITUDE=18.7875
|
||||
DEFAULT_LONGITUDE=99.0045
|
||||
|
||||
# External Services
|
||||
NOTIFICATION_EMAIL=
|
||||
SMTP_SERVER=
|
||||
SMTP_PORT=587
|
||||
SMTP_USERNAME=
|
||||
SMTP_PASSWORD=
|
||||
|
||||
# Public push notifications via self-hosted ntfy (https://ntfy.sh, single binary).
|
||||
# Leave NTFY_SERVER empty to disable. Topics published: <prefix>-<station>-warning,
|
||||
# <prefix>-<station>-danger, <prefix>-warning, <prefix>-danger, <prefix>-p1-outlook,
|
||||
# <prefix>-status. See docs/NOTIFICATIONS.md.
|
||||
# --- Notifications (docs/NOTIFICATIONS.md) ------------------------------------
|
||||
# Public push alerts via self-hosted ntfy; empty NTFY_SERVER = off. Topics:
|
||||
# <prefix>-<station>-warning|danger, <prefix>-warning|danger,
|
||||
# <prefix>-p1-outlook, <prefix>-status.
|
||||
NTFY_SERVER=
|
||||
# Where the monitor POSTs (defaults to NTFY_SERVER). Use the local ntfy
|
||||
# address (loopback or Tailscale IP) so publishing does not depend on
|
||||
# DNS / the reverse proxy being up.
|
||||
# Where the service POSTs (default NTFY_SERVER): the local ntfy address, so
|
||||
# publishing does not depend on DNS or the reverse proxy
|
||||
NTFY_PUBLISH_URL=
|
||||
NTFY_TOPIC_PREFIX=ping
|
||||
NTFY_TOKEN=
|
||||
PUBLIC_URL=https://water.buildfor.life/
|
||||
# ping-collector notifier: report RID lagging this many hours while ThaiWater
|
||||
# telemetry is current (0 = never)
|
||||
# Report RID lagging this many hours while ThaiWater telemetry is current (0 = never)
|
||||
RID_STALE_ALERT_HOURS=18
|
||||
|
||||
# Matrix: ping-collector sends the ntfy transitions to this room too
|
||||
# (docs/MATRIX_QUICK_START.md); off while token or room is empty
|
||||
# Matrix room for a team: the same transitions (docs/MATRIX_QUICK_START.md);
|
||||
# off while token or room is empty
|
||||
MATRIX_HOMESERVER=https://matrix.org
|
||||
MATRIX_ACCESS_TOKEN=
|
||||
MATRIX_ROOM_ID=
|
||||
|
||||
# Website and API served by ping-collector (rust/collector/src/web): the
|
||||
# address to listen on; empty = the Python app serves (port 8000). Set by
|
||||
# scripts/install_collector.sh --cutover-api.
|
||||
WEB_LISTEN=
|
||||
# --- Website and API -------------------------------------------------------------
|
||||
# X-API-Key for station editing, /config and POST /scrape/trigger (empty = those are off)
|
||||
ADMIN_API_KEY=
|
||||
# Comma-separated origins allowed cross-site with credentials (empty: any origin, no credentials)
|
||||
CORS_ALLOW_ORIGINS=
|
||||
# ThaiWater API key for /sensors/thaiwater (extra Ping-basin sensors on the map)
|
||||
THAIWATER_API_KEY=
|
||||
# Self-hosted Umami: server-side tracking of API calls
|
||||
# UMAMI_API_URL=https://stats.buildfor.life/api/send
|
||||
# UMAMI_WEBSITE_ID=00b2be73-8f5f-4400-9029-3be852eb08f7
|
||||
# UMAMI_TRACK_API=true
|
||||
# Response cache TTLs (seconds)
|
||||
# LATEST_CACHE_TTL_SECONDS=45
|
||||
# HII_CACHE_TTL_SECONDS=120
|
||||
# HEALTH_CACHE_TTL_SECONDS=30
|
||||
# A stored forecast older than this is served marked X-Data-Stale: true
|
||||
# FORECAST_MAX_AGE_SECONDS=7200
|
||||
# STATIC_DIR=src/static
|
||||
# ADMIN_API_KEY= # X-API-Key for station editing, /config, /scrape/trigger
|
||||
# Only for `ping-collector serve` running separately from `run`: the status file
|
||||
# SCRAPING_STATUS_FILE=collector-status.json
|
||||
|
||||
# Grafana Integration
|
||||
GRAFANA_URL=http://localhost:3000
|
||||
|
||||
# Alert Configuration
|
||||
ALERT_MAX_AGE_HOURS=2
|
||||
ALERT_CHECK_INTERVAL_MINUTES=15
|
||||
|
||||
# Development Settings
|
||||
DEBUG=false
|
||||
DEVELOPMENT_MODE=false
|
||||
# --- Python tools ----------------------------------------------------------------
|
||||
# Training/evaluation read the database above; this overrides it for them only
|
||||
# FLOOD_ML_DB_URL=
|
||||
Reference in new issue
Block a user