28 Commits

Include renames

Author	SHA1	Message	Date
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	6af6fbe02c	fix: survive junk RID dam values — widen storage_pct, bound inserts ... Dam 100602 reports 87798% storage on some days, overflowing NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/ MySQL tables) and every measure column is bounds-checked before insert so out-of-capacity junk becomes NULL instead of a batch-killing error. Rerunning the backfill repairs the days the overflow skipped.	2026-08-13 10:50:58 +07:00
grabowski	ba781465a9	feat: in-memory HII gap-fill in the ML data loader ... load_measurements() (DB path) patches missing station-hours from the hii_waterlevel mirror telemetry: exact mirrors (P.1/P.103/P.20/P.4A/P.67/ P.75/P.82/P.84/P.92) plus bias-corrected P.81 (+9,340 h). Per-station MSL->gauge offset is derived from >=168 h of series overlap, which reproduces the published offsets for exact mirrors and absorbs P.81's bias; P.76/P.77/P.85/P.87 HII twins are different physical sensors and stay excluded. Training and serving share the loader, so both sides see identical filled series; water_measurements is never written.	2026-08-13 10:29:54 +07:00
grabowski	6eafb353b1	feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow ... POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain.	2026-08-13 10:29:34 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	98023243af	feat: forecast precompute + issued-forecast archive + dashboard overlay ... The collection-leader worker now precomputes forecasts after every scrape cycle: primes the /forecast cache (users never trigger the multi-second inference — its TTL rises to 4500s so the hourly refresh always wins) and persists every issued forecast to a new forecast_history table keyed by (as_of, station, horizon) with predicted max level, warn/danger probabilities, current level, and model_version. This is the operational record the backtests lacked — predicted-vs-actual becomes a simple join instead of retraining historical models. GET /api/forecast/history/{station} serves the archive (hours or start/end + horizon filters, 5-min edge cache), and the station history chart overlays 'Model 24 h peak (as issued)' as a dashed violet line once data accumulates.	2026-08-12 14:13:39 +07:00
grabowski	731f10910e	perf: stale-while-revalidate caching, bigger executor, cheap DB health probe ... Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine).	2026-08-12 12:32:38 +07:00
grabowski	039af8caac	perf: cache /measurements/latest; stale-on-error fallback for cached endpoints ... The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120).	2026-08-12 11:05:02 +07:00
grabowski	d27ca8bf40	perf: 120s TTL cache with single-flight on /api/hii/*/latest ... These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one.	2026-08-12 10:58:19 +07:00
grabowski	09c1c84153	feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta ... robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404).	2026-08-11 16:33:44 +07:00
grabowski	e27d418a1a	feat: station search in side panel; DB stats cover HII tables ... A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'.	2026-08-11 16:06:39 +07:00
grabowski	ba4710b243	feat: 'Last N' labels on history ranges; dropdown mirrors into date pickers ... Quick-range options read 'Last 24 hours' … 'Last 90 days'; choosing one fills the from/to date inputs with today and today-N (All time clears them). Dropdown-driven loads still fetch the precise trailing-hours window; hand-edited dates take over via state.useDates.	2026-08-11 15:58:28 +07:00
grabowski	9516857d44	feat: date-range picker on the station history panel; drop PostgreSQL naming ... /measurements/history/{code} accepts optional start/end date params that override the hours window (end date inclusive). The dashboard history card gains from/to date inputs beside the quick-range dropdown — explicit dates win, changing the dropdown clears them. All user-facing 'PostgreSQL history' labels renamed to 'Station history'.	2026-08-11 15:52:40 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00
grabowski	9f26b32c86	fix: bump history limits to 100k so all-time shows full dataset	2026-08-09 17:41:14 +07:00
grabowski	c00a26402a	fix: update test limit boundary to match new 10k ceiling	2026-08-09 17:13:51 +07:00
grabowski	ae5d0a13d7	[verified] feat: add live river dashboard ... Add mapped river and ThaiWater sensor layers, PostgreSQL history charts, API endpoints, and dashboard tests.	2026-08-09 16:59:25 +07:00
grabowski	08dc536c93	Add unit tests for RID API response parsing ... Cover the previously-untested, highest-risk parsing in fetch_water_data_for_date by mocking the HTTP call: - hour 1-23 map to the same day; hour 24 rolls to next-day midnight - qvalues "***" / None yield discharge None (no crash) - None water level is skipped - out-of-range (0, 25) and empty hourlytime rows are skipped - missing "rows" key returns an empty list This gives the scraper a safety net before its module is split.	2026-07-22 14:24:28 +07:00
grabowski	12b7f9f422	Add assert-based pytest coverage for recent fixes ... - tests/conftest.py: put repo root on sys.path so `import src...` resolves under pytest regardless of invocation directory. - test_matrix_formatting.py: lock in HTML formatted_body + plain-text fallback, URL linkification, HTML escaping, and send_alert field rendering. - test_station_persistence.py: cover default-load, save/reload round-trip (incl. Thai text), runtime-file precedence, and atomic-write cleanup. These are real assert-based tests (unlike the existing print-style scripts) so CI can gate on them. 13 tests, all passing.	2026-07-22 14:02:06 +07:00
grabowskiandClaude	cc007f0e0c	Add comprehensive alerting system tests ... - Created test suite for zone-based water level alerts (9 test cases) - Created test suite for rate-of-change alerts (5 test cases) - Created combined alert scenario test - Fixed rate-of-change detection to use station_code instead of station_id - All 3 test suites passing (14 total test cases) Test coverage: - Zone alerts: P.1 zones 1-8 with INFO/WARNING/CRITICAL/EMERGENCY levels - Rate-of-change: 0.15/0.25/0.40 m/h thresholds for WARNING/CRITICAL/EMERGENCY - Combined: Simultaneous zone and rate-of-change alert triggering Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:35:34 +07:00
grabowski	17a716fcd0	Version bump: 3.1.2 3.1.3 (Force new build) ... Version Updates: - Core application: src/__init__.py, src/main.py, src/web_api.py - Package configuration: setup.py - Documentation: README.md, docs/GITEA_WORKFLOWS.md - Workflows: .gitea/workflows/docs.yml, .gitea/workflows/release.yml - Scripts: generate_badges.py, init_git scripts - Tests: test_integration.py - Deployment docs: GITEA_SETUP_SUMMARY.md, DEPLOYMENT_CHECKLIST.md Purpose: - Force new build process after workflow fixes - Test updated security.yml without YAML errors - Verify setup.py robustness improvements - Trigger clean CI/CD pipeline execution All version references synchronized at v3.1.3 Ready for new build and deployment testing	2025-08-12 17:47:26 +07:00
grabowski	40aef686af	Fix: Replace GitHub checkout with Gitea checkout + Version bump ... Checkout Action Migration: - Replace all 'actions/checkout@v4' with 'https://gitea.com/actions/checkout' - Fixes 'Bad credentials' errors when workflows try to access GitHub API - Native Gitea checkout action eliminates authentication issues - Applied across all 4 workflow files (CI, Security, Release, Docs) Version Increment: 3.1.1 3.1.2 - Core application version updates - Web API version synchronization - Documentation version alignment - Badge and release example updates Problem Solved: - Workflows no longer attempt GitHub API calls - Gitea-native checkout action handles repository access properly - Eliminates 'Retrieving the default branch name' failures - Cleaner workflow execution without authentication errors Files Updated: - 4 workflow files: checkout action replacement - 13 files: version number updates - Consistent v3.1.2 across all components Benefits: - Workflows will now run successfully in Gitea - No more GitHub API authentication failures - Native Gitea action compatibility - Ready for successful CI/CD pipeline execution	2025-08-12 17:06:20 +07:00
grabowski	19e182c53b	Version bump: 3.1.0 3.1.1 ... Version Updates: - Core application version (src/__init__.py) - Web API version (src/web_api.py) - Main application logging (src/main.py) - Package setup version (setup.py) - Documentation generation (docs workflow) - Release workflow example version - Badge generation script - Integration test version display - README.md badge version - Setup and deployment documentation - Git initialization scripts Patch Release (3.1.1): - Workflow token migration fixes (GITHUB_TOKEN GH_TOKEN) - Pip installation warning elimination - Improved workflow reliability and logging - Better Gitea compatibility - Enhanced error handling and validation Files Updated: - 13 files with version references updated - Consistent versioning across all components - Ready for release tagging and deployment	2025-08-12 16:52:39 +07:00
grabowski	af62cfef0b	Initial commit: Northern Thailand Ping River Monitor v3.1.0 ... Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!	2025-08-12 15:40:24 +07:00