31 Commits

Include renames

Author	SHA1	Message	Date
grabowski	7e64e0cf18	fix: one current river level, not two ... The verdict banner and the P.1 outlook each wrote state.p1Now from a different feed — the banner from the latest measurement, the outlook from current_level on the forecast rows, which carries whatever the model saw at its as_of. Forecasts are precomputed hourly, so the two drifted apart: production showed 1.66 m in the banner and 1.52 m in the outlook directly below it. Harmless at low water; at flood stage two contradictory river levels on one screen undermine the warning. setP1Level() now arbitrates: freshest timestamp wins, and the replay and demo hooks pass force since they deliberately pin a level that is not the live one. The outlook renders the arbitrated value and clamps the shown peak to at least the current level, so a stale forecast can no longer predict a peak below where the river already is. endReplay drops the replayed level so live data re-arbitrates cleanly. Verified against a stub reproducing the exact production conditions (gauge 1.66 at 08:35 vs forecast 1.52 at 08:00): both now read 1.66; a 2.50 m rise against a stale 1.81 m peak renders 2.50/2.50; the 2024 replay still tracks its frames and returns to live on stop.	2026-08-14 09:25:39 +07:00
grabowski	382daa7d86	perf: backfill one dam per request via the api/dam range endpoint ... GET app.rid.go.th/reservoir/api/dam?dam_id&date_start&date_end returns a single dam's whole date range in one response — Mae Ngat's 2009-today archive is ~4 chunked requests instead of the ~2,900 one-day POSTs the all-dams path needs. Field names differ from api/dams and are mapped in parse_dam_range_records, verified equal on spot-checked dates; the range endpoint also carries DMD_ULevel, the reservoir level in m MSL that api/dams stopped publishing after ~2013. scripts/backfill_rid_reservoir.py defaults to the fast per-dam path (--all-dams keeps the full-fleet crawl, --refresh rewrites stored days). Already-stored dates are still skipped, junk values are still bounded, and the consecutive-failure abort still applies.	2026-08-13 23:10:36 +07:00
grabowski	b02e815d72	feat: Thai localisation and portrait-phone layout for the dashboard ... Thai is the default unless the browser prefers English, chosen by first match in navigator.languages order and remembered in localStorage. A STRINGS table carries both languages (interpolated strings as functions), applyTranslations() drives static markup via data-i18n attributes, and setLang() rebuilds everything the JS renders — including map layers, so popups render in the current language and sensor markers are replaced rather than stacked. Thai dates use the Buddhist era, matching the replay label; station names lead with the reader's language. Portrait phones: the header overflowed a 412 px Android viewport by 64 px, so the page scrolled sideways and the Refresh button sat off screen. The header now wraps into two deliberate rows (DOM order matches visual order, so focus order is unaffected), the map description box is hidden on phones, map height is capped by viewport — including a height-gated rule for landscape phones, whose 850-960 px widths never matched the width breakpoints — and the forecast grid goes single column. Verified in a real browser at 412x915, 360x800, 915x412 and 1440x900: zero horizontal overflow, no desktop change. Review-swarm fixes: a language switch no longer relabels a pinned SIMULATION or the 2024 replay as LIVE DATA (it kept the mode from state); Thai wording corrected where it asserted a rising trend the code never checks, labelled every gauge 'critical', or used a malformed compound; aria-labels, the Leaflet load failure and the flood-stage chips are translated; the language toggle states its action instead of an aria-pressed value that contradicted its label; and Thai font families sit after the Latin stack so they cannot restyle English text.	2026-08-13 23:10:34 +07:00
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	6af6fbe02c	fix: survive junk RID dam values — widen storage_pct, bound inserts ... Dam 100602 reports 87798% storage on some days, overflowing NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/ MySQL tables) and every measure column is bounds-checked before insert so out-of-capacity junk becomes NULL instead of a batch-killing error. Rerunning the backfill repairs the days the overflow skipped.	2026-08-13 10:50:58 +07:00
grabowski	ba781465a9	feat: in-memory HII gap-fill in the ML data loader ... load_measurements() (DB path) patches missing station-hours from the hii_waterlevel mirror telemetry: exact mirrors (P.1/P.103/P.20/P.4A/P.67/ P.75/P.82/P.84/P.92) plus bias-corrected P.81 (+9,340 h). Per-station MSL->gauge offset is derived from >=168 h of series overlap, which reproduces the published offsets for exact mirrors and absorbs P.81's bias; P.76/P.77/P.85/P.87 HII twins are different physical sensors and stay excluded. Training and serving share the loader, so both sides see identical filled series; water_measurements is never written.	2026-08-13 10:29:54 +07:00
grabowski	6eafb353b1	feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow ... POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain.	2026-08-13 10:29:34 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	98023243af	feat: forecast precompute + issued-forecast archive + dashboard overlay ... The collection-leader worker now precomputes forecasts after every scrape cycle: primes the /forecast cache (users never trigger the multi-second inference — its TTL rises to 4500s so the hourly refresh always wins) and persists every issued forecast to a new forecast_history table keyed by (as_of, station, horizon) with predicted max level, warn/danger probabilities, current level, and model_version. This is the operational record the backtests lacked — predicted-vs-actual becomes a simple join instead of retraining historical models. GET /api/forecast/history/{station} serves the archive (hours or start/end + horizon filters, 5-min edge cache), and the station history chart overlays 'Model 24 h peak (as issued)' as a dashed violet line once data accumulates.	2026-08-12 14:13:39 +07:00
grabowski	731f10910e	perf: stale-while-revalidate caching, bigger executor, cheap DB health probe ... Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine).	2026-08-12 12:32:38 +07:00
grabowski	039af8caac	perf: cache /measurements/latest; stale-on-error fallback for cached endpoints ... The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120).	2026-08-12 11:05:02 +07:00
grabowski	d27ca8bf40	perf: 120s TTL cache with single-flight on /api/hii/*/latest ... These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one.	2026-08-12 10:58:19 +07:00
grabowski	09c1c84153	feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta ... robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404).	2026-08-11 16:33:44 +07:00
grabowski	e27d418a1a	feat: station search in side panel; DB stats cover HII tables ... A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'.	2026-08-11 16:06:39 +07:00
grabowski	ba4710b243	feat: 'Last N' labels on history ranges; dropdown mirrors into date pickers ... Quick-range options read 'Last 24 hours' … 'Last 90 days'; choosing one fills the from/to date inputs with today and today-N (All time clears them). Dropdown-driven loads still fetch the precise trailing-hours window; hand-edited dates take over via state.useDates.	2026-08-11 15:58:28 +07:00
grabowski	9516857d44	feat: date-range picker on the station history panel; drop PostgreSQL naming ... /measurements/history/{code} accepts optional start/end date params that override the hours window (end date inclusive). The dashboard history card gains from/to date inputs beside the quick-range dropdown — explicit dates win, changing the dropdown clears them. All user-facing 'PostgreSQL history' labels renamed to 'Station history'.	2026-08-11 15:52:40 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00
grabowski	9f26b32c86	fix: bump history limits to 100k so all-time shows full dataset	2026-08-09 17:41:14 +07:00
grabowski	c00a26402a	fix: update test limit boundary to match new 10k ceiling	2026-08-09 17:13:51 +07:00
grabowski	ae5d0a13d7	[verified] feat: add live river dashboard ... Add mapped river and ThaiWater sensor layers, PostgreSQL history charts, API endpoints, and dashboard tests.	2026-08-09 16:59:25 +07:00
grabowski	08dc536c93	Add unit tests for RID API response parsing ... Cover the previously-untested, highest-risk parsing in fetch_water_data_for_date by mocking the HTTP call: - hour 1-23 map to the same day; hour 24 rolls to next-day midnight - qvalues "***" / None yield discharge None (no crash) - None water level is skipped - out-of-range (0, 25) and empty hourlytime rows are skipped - missing "rows" key returns an empty list This gives the scraper a safety net before its module is split.	2026-07-22 14:24:28 +07:00
grabowski	12b7f9f422	Add assert-based pytest coverage for recent fixes ... - tests/conftest.py: put repo root on sys.path so `import src...` resolves under pytest regardless of invocation directory. - test_matrix_formatting.py: lock in HTML formatted_body + plain-text fallback, URL linkification, HTML escaping, and send_alert field rendering. - test_station_persistence.py: cover default-load, save/reload round-trip (incl. Thai text), runtime-file precedence, and atomic-write cleanup. These are real assert-based tests (unlike the existing print-style scripts) so CI can gate on them. 13 tests, all passing.	2026-07-22 14:02:06 +07:00
grabowskiandClaude	cc007f0e0c	Add comprehensive alerting system tests ... - Created test suite for zone-based water level alerts (9 test cases) - Created test suite for rate-of-change alerts (5 test cases) - Created combined alert scenario test - Fixed rate-of-change detection to use station_code instead of station_id - All 3 test suites passing (14 total test cases) Test coverage: - Zone alerts: P.1 zones 1-8 with INFO/WARNING/CRITICAL/EMERGENCY levels - Rate-of-change: 0.15/0.25/0.40 m/h thresholds for WARNING/CRITICAL/EMERGENCY - Combined: Simultaneous zone and rate-of-change alert triggering Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:35:34 +07:00
grabowski	17a716fcd0	Version bump: 3.1.2 3.1.3 (Force new build) ... Version Updates: - Core application: src/__init__.py, src/main.py, src/web_api.py - Package configuration: setup.py - Documentation: README.md, docs/GITEA_WORKFLOWS.md - Workflows: .gitea/workflows/docs.yml, .gitea/workflows/release.yml - Scripts: generate_badges.py, init_git scripts - Tests: test_integration.py - Deployment docs: GITEA_SETUP_SUMMARY.md, DEPLOYMENT_CHECKLIST.md Purpose: - Force new build process after workflow fixes - Test updated security.yml without YAML errors - Verify setup.py robustness improvements - Trigger clean CI/CD pipeline execution All version references synchronized at v3.1.3 Ready for new build and deployment testing	2025-08-12 17:47:26 +07:00
grabowski	40aef686af	Fix: Replace GitHub checkout with Gitea checkout + Version bump ... Checkout Action Migration: - Replace all 'actions/checkout@v4' with 'https://gitea.com/actions/checkout' - Fixes 'Bad credentials' errors when workflows try to access GitHub API - Native Gitea checkout action eliminates authentication issues - Applied across all 4 workflow files (CI, Security, Release, Docs) Version Increment: 3.1.1 3.1.2 - Core application version updates - Web API version synchronization - Documentation version alignment - Badge and release example updates Problem Solved: - Workflows no longer attempt GitHub API calls - Gitea-native checkout action handles repository access properly - Eliminates 'Retrieving the default branch name' failures - Cleaner workflow execution without authentication errors Files Updated: - 4 workflow files: checkout action replacement - 13 files: version number updates - Consistent v3.1.2 across all components Benefits: - Workflows will now run successfully in Gitea - No more GitHub API authentication failures - Native Gitea action compatibility - Ready for successful CI/CD pipeline execution	2025-08-12 17:06:20 +07:00
grabowski	19e182c53b	Version bump: 3.1.0 3.1.1 ... Version Updates: - Core application version (src/__init__.py) - Web API version (src/web_api.py) - Main application logging (src/main.py) - Package setup version (setup.py) - Documentation generation (docs workflow) - Release workflow example version - Badge generation script - Integration test version display - README.md badge version - Setup and deployment documentation - Git initialization scripts Patch Release (3.1.1): - Workflow token migration fixes (GITHUB_TOKEN GH_TOKEN) - Pip installation warning elimination - Improved workflow reliability and logging - Better Gitea compatibility - Enhanced error handling and validation Files Updated: - 13 files with version references updated - Consistent versioning across all components - Ready for release tagging and deployment	2025-08-12 16:52:39 +07:00
grabowski	af62cfef0b	Initial commit: Northern Thailand Ping River Monitor v3.1.0 ... Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!	2025-08-12 15:40:24 +07:00