10 Commits

Include renames

Author	SHA1	Message	Date
grabowski	779728e38b	feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions ... Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what the FastAPI app (src/web_api.py) served, so one process collects, forecasts, notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside `ping-collector run`; `ping-collector serve` runs it alone. Same routes, parameters, JSON fields and field order, status codes, {"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip (>500 B), CORS, Umami API tracking, the dashboard and static files, and the stale-while-revalidate response caches (now with byte budgets: a history response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI document is built from the route table, whose per-route parameter declarations also validate the requests, so docs and behaviour cannot drift. Python output conventions are kept: isoformat(), round(), numpy's pairwise sums (catchment means), Python's summation order (skill metrics). scripts/api_parity.py compares the two servers route by route on one database (53 cases incl. errors, admin and static files; query values vary per run so FastAPI's caches answer fresh). Locally: 53 identical; the station CRUD writes byte-identical stations.json files; the skill/trend logic matches on 9 station/horizon pairs with synthetic model versions. Intended differences: /health counts the ThaiWater telemetry of the same gauges while RID lags (as the notifier); /scraping/status adds the collector version and last forecast run; an old stored forecast is served marked X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute; .geojson is served as application/geo+json. Load (local, 16 clients): the dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s; 25 MB resident when serving only. Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets (per station, outlook, feed; not ntfy's per-topic summaries), each once, with its own matrix: keys in notification_state, seeded from ntfy's on first use so nothing already announced repeats. Checked against a fake homeserver. It replaces the `run.py --alert-check` cron job (repeated alerts every 15 min), which install_collector.sh now comments out (crontab backup kept). install_collector.sh: --check-api (ping-collector serve on :8001 next to FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets WEB_LISTEN, stops and disables water-monitor.service), --rollback-api; --rollback-forecast and --rollback bring the Python app back as server. ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route tests against a throwaway schema.	2026-10-18 16:08:06 +07:00
grabowski	6ae67dec63	feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night ... Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock).	2026-10-18 14:43:39 +07:00
grabowski	4e619aa2ae	feat(rust): ping-collector, phase 1 of the Rust rewrite ... rust/collector replaces the collection half of the Python web app: RID hourly gauges (with the late-report semantics: store everything, retry every 10 min while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain + water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and upsert semantics, one UNNEST statement per table, Bangkok wall-clock time regardless of the server time zone. Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and instead re-forecast + notify when the newest RID reading in the database advances (polled every FORECAST_POLL_SECONDS) and once per clock hour; /scraping/status then serves the collector's status file; /scrape/trigger answers 409. Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against PostgreSQL); live parity run of both collectors into two empty databases: all 9 tables row-for-row identical (scripts/collector_parity.py); the Python API served health, measurements, stats, status and model forecasts from a database filled only by ping-collector. 177 Python tests pass. reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which rustls refuses. Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service; CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service, release artifact).	2026-10-17 22:20:02 +07:00
grabowski	04ebdaa931	fix: RID collector keeps late readings instead of losing every evening ... RID's hourly report often lags 1-2 h in the evening. The cycle discarded the whole report unless it already held the current hour, the web-API loop then slept a full hour, and after 01:00 only today's report was fetched. Result: the 19:00-24:00 readings were dropped most nights (P.1 lost 00:00 on 23 of the last 60 days, 13 evening hours this week), the feed went stale from ~21:00 until morning and ntfy ping-status sent a stale/recovered pair daily. - run_scraping_cycle() upserts every reading the report has; "current hour present" now only decides the retry cadence (returned as before). - Yesterday's report is re-fetched while the store has holes in it (hours 22-24 are only reachable through that day's report). - Leader loop polls at :05 (RID_POLL_OFFSET_MINUTES) and retries every RID_RETRY_MINUTES (10) while RID is behind; HII/dams/rain once per clock hour; forecast precompute + ntfy when newer readings arrive or hourly. - Startup re-fetches holes from the last GAP_FILL_DAYS_ON_START (14) days. - /scraping/status: the leader shares its stats via a JSON file, so any worker reports them (was always 0 runs); adds feed_current, latest_reading, readings_saved.	2026-10-17 21:45:09 +07:00
grabowski	f4d42c90f4	fix: ntfy listens on the Tailscale address; monitor publishes to it directly ... The reverse proxy is a separate VPS on the tailnet, so a loopback-only ntfy was unreachable from it. install_ntfy.sh now binds the host's Tailscale IP (NTFY_LISTEN overrides). New NTFY_PUBLISH_URL: where the monitor POSTs, separate from the public NTFY_SERVER subscribers see, so an alert never waits on DNS or the proxy (first cycle logged 502s from Cloudflare while the domain was not yet proxied).	2026-09-12 00:28:29 +02:00
grabowski	777b230baf	feat: public flood notifications over self-hosted ntfy ... Anyone can now get push alerts on their phone without an account: the monitor publishes to an ntfy server (one Go binary, ~30 MB RSS) and subscribers pick topics in the free iOS/Android/web app. Semantics are transitions, never state. One message when a gauge crosses its warning or danger threshold, one all-clear when it drops back (0.10 m hysteresis), nothing while it sits above. A three-day flood is two messages; a quiet season is zero. Topics: ping-warning / ping-danger (basin digest), ping-<station>-warning / -danger, ping-p1-outlook (opt-in: model P(warning within 24 h) at P.1 rises through 50 %, clears below 25 %, message says it is experimental), ping-status (feed stale >= 3 h / recovered). Priority 5 on danger so it rings through Do Not Disturb. src/notify.py runs once per collection cycle in the API process (leader only, after the forecast precompute, same data the dashboard shows). Last-sent state lives in a notification_state table so a restart never re-sends; a failed publish leaves state untouched so the crossing is retried next cycle instead of lost. Off unless NTFY_SERVER is set. Dashboard: a "Get alerts" button (only when configured) opens a panel with the server, per-topic cards, ntfy:// deep links and web links, app store links and a disclaimer. EN + TH. GET /api/notifications feeds it. scripts/install_ntfy.sh: .deb install, server.yml (loopback listen, anonymous read, token-only write scoped to ping-*, 72 h cache, signup/ login/metrics off, tight visitor limits), systemd, user + token, .env. Verified against ntfy 2.28.0: anon publish 403, token publish 200, token on foreign topic 403, anon read 200, and a seeded crossing through the real _notify_transitions path arrived in the topic with priority, tags, click and action button. docs/NOTIFICATIONS.md has the deployment and reverse-proxy notes. Tests: 10 for the state machine (159 total).	2026-09-12 00:18:38 +02:00
grabowski	ae5d0a13d7	[verified] feat: add live river dashboard ... Add mapped river and ThaiWater sensor layers, PostgreSQL history charts, API endpoints, and dashboard tests.	2026-08-09 16:59:25 +07:00
grabowskiandClaude	ca730e484b	Add comprehensive Matrix alerting system with Grafana integration ... - Implement custom Python alerting system (src/alerting.py) with water level monitoring, data freshness checks, and Matrix notifications - Add complete Grafana Matrix alerting setup guide (docs/GRAFANA_MATRIX_SETUP.md) with webhook configuration, alert rules, and notification policies - Create Matrix quick start guide (docs/MATRIX_QUICK_START.md) for rapid deployment - Integrate alerting commands into main application (--alert-check, --alert-test) - Add Matrix configuration to environment variables (.env.example) - Update Makefile with alerting targets (alert-check, alert-test) - Enhance status command to show Matrix notification status - Support station-specific water level thresholds and escalation rules - Provide dual alerting approach: native Grafana alerts and custom Python system 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-26 16:18:02 +07:00
grabowskiandClaude	6c7c128b4d	Major refactor: Migrate to uv, add PostgreSQL support, and comprehensive tooling ... - **Migration to uv package manager**: Replace pip/requirements with modern pyproject.toml - Add pyproject.toml with complete dependency management - Update all scripts and Makefile to use uv commands - Maintain backward compatibility with existing workflows - **PostgreSQL integration and migration tools**: - Enhanced config.py with automatic password URL encoding - Complete PostgreSQL setup scripts and documentation - High-performance SQLite to PostgreSQL migration tool (91x speed improvement) - Support for both connection strings and individual components - **Executable distribution system**: - PyInstaller integration for standalone .exe creation - Automated build scripts with batch file generation - Complete packaging system for end-user distribution - **Enhanced data management**: - Fix --fill-gaps command with proper method implementation - Add gap detection and historical data backfill capabilities - Implement data update functionality for existing records - Add comprehensive database adapter methods - **Developer experience improvements**: - Password encoding tools for special characters - Interactive setup wizards for PostgreSQL configuration - Comprehensive documentation and migration guides - Automated testing and validation tools 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-26 15:10:10 +07:00
grabowski	af62cfef0b	Initial commit: Northern Thailand Ping River Monitor v3.1.0 ... Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!	2025-08-12 15:40:24 +07:00