15 Commits

Include renames

Author	SHA1	Message	Date
grabowski	160617e87b	feat: openmeteo_rain 2021+ backfill entry point ... rain.backfill_db pushes the full cached Open-Meteo archive into the openmeteo_rain table in 5k-row idempotent upsert chunks; scripts/backfill_rain_db.py is the thin CLI (DB from Config/.env or --db-url). Safe to re-run and safe alongside the hourly live writer.	2026-08-12 17:28:44 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	21e9d2e114	feat: hgb-v2 — regression heads predict rise, recovering flood warning lead ... Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.	2026-08-12 15:43:29 +07:00
grabowski	a0086086a2	feat: rolling-origin event-aware evaluation harness for model variants ... One fold per monsoon season (train <= 30 Apr, test Jun-Nov, 2021-2025) replaces the single fixed holdout that contained only ~4 warning events. Metrics are what matters operationally: sustained first-alert lead vs each observed 3.70m crossing (two consecutive alerting samples required; lookback floored at the previous event's end so multi-peak floods can't launder lead credit), peak error from the prediction actually issued 24h before the peak (3h match tolerance, null on outages), false-alarm episodes (12h gap tolerance), MAE / flood-regime MAE, and a Brier score on warning exceedance — included because sigma cancels algebraically in any p>=0.5 alert metric, so lead times compare predictors while Brier compares uncertainty models. Variants: baseline_abs (current), rise (target = future max - current level), rise_weighted (flood-regime sample weights 1x->5x), and rise_quantile (q50/q90 heads, spread-implied sigma). Harness verified by a 3-agent adversarial review (features bit-identical across fold cutoffs; three metric flaws found and fixed before first use). Also: features.build_labels/build_matrix gain stats_end so the rescue quantile is computed from pre-cutoff data only, closing the label- construction leak flagged in the earlier ML review.	2026-08-12 15:19:26 +07:00
grabowski	1ec5cfb4df	perf: gzip responses; multi-worker serving with single collection leader ... GZipMiddleware (min 500 bytes) compresses the dashboard HTML ~4x and station JSON up to ~100x, end-to-end through the Caddy TLS terminator — production load testing showed the deployment is bandwidth-bound once the response caches hit, so compression is the capacity lever. WEB_WORKERS (default 2) runs uvicorn multi-process via the app import string. Every worker executes the lifespan, so a localhost lock port (COLLECTION_LEADER_PORT, default 8901) elects exactly one background-collection leader per machine — RID/HII polling stays once-per-cycle instead of once-per-worker; the lock releases with the process. Locust clients now send Accept-Encoding so future runs measure compressed transfer, as browsers do.	2026-08-12 11:34:12 +07:00
grabowski	0005f7dce1	feat: codified backtests, honest docs, belt-and-braces serving, perf fixes ... Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).	2026-08-12 10:46:00 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00
grabowski	ce31a5254e	Harden install.sh per security review ... - .env now chmod 0600 and APP_DIR chmod 0750 after chown, so the Matrix token and DB credentials are not world-readable. - uv auto-install (curl | sh as root) is now opt-in via AUTO_INSTALL_UV=1 and pins a specific uv version; otherwise the script requires uv to be pre-installed and fails with instructions, avoiding unattended remote code execution as root.	2026-07-22 14:00:44 +07:00
grabowski	ab8a10dd75	Add install.sh and fix service unit placeholder ... - scripts/install.sh: one-command hardened deploy (creates the water-monitor system user, deploys to /opt, builds a uv-managed venv, installs and enables the systemd unit). Idempotent; excludes .env/*.db/stations.json from sync so runtime state is preserved. - Fix placeholder Documentation= URL in water-monitor.service. - README: document the script as the primary systemd install path, with manual steps kept as a fallback.	2026-07-22 12:55:17 +07:00
grabowskiandClaude	6c7c128b4d	Major refactor: Migrate to uv, add PostgreSQL support, and comprehensive tooling ... - **Migration to uv package manager**: Replace pip/requirements with modern pyproject.toml - Add pyproject.toml with complete dependency management - Update all scripts and Makefile to use uv commands - Maintain backward compatibility with existing workflows - **PostgreSQL integration and migration tools**: - Enhanced config.py with automatic password URL encoding - Complete PostgreSQL setup scripts and documentation - High-performance SQLite to PostgreSQL migration tool (91x speed improvement) - Support for both connection strings and individual components - **Executable distribution system**: - PyInstaller integration for standalone .exe creation - Automated build scripts with batch file generation - Complete packaging system for end-user distribution - **Enhanced data management**: - Fix --fill-gaps command with proper method implementation - Add gap detection and historical data backfill capabilities - Implement data update functionality for existing records - Add comprehensive database adapter methods - **Developer experience improvements**: - Password encoding tools for special characters - Interactive setup wizards for PostgreSQL configuration - Comprehensive documentation and migration guides - Automated testing and validation tools 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-26 15:10:10 +07:00
grabowski	17a716fcd0	Version bump: 3.1.2 3.1.3 (Force new build) ... Version Updates: - Core application: src/__init__.py, src/main.py, src/web_api.py - Package configuration: setup.py - Documentation: README.md, docs/GITEA_WORKFLOWS.md - Workflows: .gitea/workflows/docs.yml, .gitea/workflows/release.yml - Scripts: generate_badges.py, init_git scripts - Tests: test_integration.py - Deployment docs: GITEA_SETUP_SUMMARY.md, DEPLOYMENT_CHECKLIST.md Purpose: - Force new build process after workflow fixes - Test updated security.yml without YAML errors - Verify setup.py robustness improvements - Trigger clean CI/CD pipeline execution All version references synchronized at v3.1.3 Ready for new build and deployment testing	2025-08-12 17:47:26 +07:00
grabowski	40aef686af	Fix: Replace GitHub checkout with Gitea checkout + Version bump ... Checkout Action Migration: - Replace all 'actions/checkout@v4' with 'https://gitea.com/actions/checkout' - Fixes 'Bad credentials' errors when workflows try to access GitHub API - Native Gitea checkout action eliminates authentication issues - Applied across all 4 workflow files (CI, Security, Release, Docs) Version Increment: 3.1.1 3.1.2 - Core application version updates - Web API version synchronization - Documentation version alignment - Badge and release example updates Problem Solved: - Workflows no longer attempt GitHub API calls - Gitea-native checkout action handles repository access properly - Eliminates 'Retrieving the default branch name' failures - Cleaner workflow execution without authentication errors Files Updated: - 4 workflow files: checkout action replacement - 13 files: version number updates - Consistent v3.1.2 across all components Benefits: - Workflows will now run successfully in Gitea - No more GitHub API authentication failures - Native Gitea action compatibility - Ready for successful CI/CD pipeline execution	2025-08-12 17:06:20 +07:00
grabowski	19e182c53b	Version bump: 3.1.0 3.1.1 ... Version Updates: - Core application version (src/__init__.py) - Web API version (src/web_api.py) - Main application logging (src/main.py) - Package setup version (setup.py) - Documentation generation (docs workflow) - Release workflow example version - Badge generation script - Integration test version display - README.md badge version - Setup and deployment documentation - Git initialization scripts Patch Release (3.1.1): - Workflow token migration fixes (GITHUB_TOKEN GH_TOKEN) - Pip installation warning elimination - Improved workflow reliability and logging - Better Gitea compatibility - Enhanced error handling and validation Files Updated: - 13 files with version references updated - Consistent versioning across all components - Ready for release tagging and deployment	2025-08-12 16:52:39 +07:00
grabowski	af62cfef0b	Initial commit: Northern Thailand Ping River Monitor v3.1.0 ... Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!	2025-08-12 15:40:24 +07:00