100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
grabowski	2e19974fad	docs: README describes the project that exists; CLAUDE.md for agents ... The README was the original template: P.1 "in Nakhon Sawan", P.103 "in Bangkok", VictoriaMetrics as the recommended database, Docker/Grafana sections, github.com/your-username support links. Rewritten around what runs: sources, gap fill, the forecast and its 13 h result, the live API table, systemd deployment with the retrain timer, repository layout, real docs links, data-source credits. Other DB adapters are mentioned as supported-but-not-production. CLAUDE.md replaces the untracked Ruflo boilerplate with project rules: Python 3.11, format/test gates, Bangkok timestamps, harness-first model changes judged on lead, the RainUnavailableError guard, no git add -A.	2026-09-11 23:44:43 +02:00
grabowski	b03318210c	security: pip-audit + bandit gates that can fail; patch 29 known CVEs ... security.yml previously ran safety/bandit/semgrep with `|| true` and could not go red. Now: pip-audit on requirements.txt is a hard gate (dev deps reported only), bandit HIGH fails (B104 bind-all skipped: intended behind Cloudflare/Caddy), pip-licenses uploaded as a report. Weekly + on dependency/source changes. Running it locally found 29 advisories, all in pinned-and-forgotten runtime deps: starlette 0.27 (7, incl. Host-header path confusion and form DoS), fastapi 0.104, requests 2.31 (3), pymysql 1.1. Bumped to current: fastapi 0.141.1 / starlette 1.6.0, pydantic 2.13.5, uvicorn 0.52.4, requests 2.34.2, pymysql 1.2.0; dev: pytest 9.1.1, black 26.5.1. pip-audit is now clean. requires-python narrowed to 3.11 (the truth: psycopg2-binary 2.9.9 fails on 3.13; pandas 2.0.3 has no 3.12 wheels). Full suite passes; API smoke-tested (health, stations, forecast, history, stats, docs, openapi) on the new stack. black 26 reformatted 8 files.	2026-09-11 23:44:43 +02:00
grabowski	97a6694ab2	feat(dashboard): light/dark theme ... Sun/moon button next to the language switch. Follows the OS preference until the user picks one (persisted in localStorage, tracks OS changes only while unpinned). Every colour that was a literal white / grey is now a token with a dark counterpart; flood-verdict, live-pill and lang-toggle states got their own bg/ink/border tokens. OSM tiles are inverted with a hue rotate so roads and labels stay legible while the river network, markers and rain dots (SVG, unfiltered) keep their data colours. Chart.js reads grid/tick/legend colours from the tokens and the open chart is rebuilt on toggle. Leaflet popups and controls follow the theme.	2026-09-11 23:16:30 +02:00
grabowski	5ad8e4eac3	ci: green pipelines that check what exists; one formatting contract ... The Test Suite job failed on every push since the black check was added because the tree had never been formatted, and pre-commit said 120 columns while CI ran black's default 88. pyproject.toml now carries [tool.black] / [tool.isort] (88, black profile) as the single source; pre-commit reads it; `make format` applied it (13 files, whitespace only, 146 insertions / 128 deletions, tests unchanged at 146 passed). ci.yml: lint (black, isort, flake8 hard errors) + pytest. The Docker registry push, VictoriaMetrics integration test, staging/production deploy and Apache-Bench jobs were template scaffolding for hosts and registries that do not exist; production is a systemd unit updated by git pull. Removed rather than left permanently skipped. docs.yml: the "Check markdown links" step curl'd every URL in every .md and failed on localhost examples and the Tailscale IP, and the Sphinx jobs built artifacts nobody read. Replaced by two checks that mean something: relative links/images in README, CONTRIBUTING and docs/ resolve inside the repo, and the FastAPI OpenAPI schema exports with the documented endpoints present (uploaded as an artifact).	2026-09-11 23:05:37 +02:00
grabowski	6f4a86edbb	fix(dashboard): timestamps are Asia/Bangkok everywhere; stale feed says so ... The API emits naive ICT timestamps ("2026-09-12T02:00:00"). The page fed them to new Date(), which applies the BROWSER's zone: a viewer in Europe parsed a 02:00 ICT reading as 02:00 CEST, five hours in the future, so "Last updated" clamped to "0 min ago" forever; a viewer in the Americas saw thousands of minutes. parseTs() now pins +07:00 on naive strings and every display formats with timeZone: Asia/Bangkok, so the site shows river time regardless of where it is opened. Daily/hourly chart buckets key on the Bangkok calendar day instead of UTC getters. The tile also gets a real stale state: past 3 h (RID is hourly) it turns red, reads "Feed stale · last reading <day>, N h ago", and the header pill switches from LIVE DATA to STALE FEED. Age shows hours past 2 h. scripts/dev_proxy.py serves the working-copy dashboard with API calls proxied to water.buildfor.life so browser-side changes can be checked against live data (and any browser timezone) before deploying.	2026-09-11 23:00:39 +02:00
grabowski	ce08312c0f	docs: public dashboard URL (water.buildfor.life) replaces the Tailscale IP everywhere	2026-09-11 22:38:13 +02:00
grabowski	d621aa9ce7	eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate ... Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_* caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.	2026-09-11 21:55:37 +02:00
grabowski	764764e07e	feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote ... train_all() now raises RainUnavailableError when use_rain=True and the Open-Meteo history cannot be loaded, instead of logging a warning and writing gauge-only (v2) bundles over the deployed v3 set -- which is what the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit way to get v2. CLI exits 2 with a one-line error. Three tests cover the guard, the opt-out, and the v3 happy path. scripts/retrain.sh trains into models/.staging, refuses to promote unless metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles into place (previous generation kept in models/.previous). No API restart: predict.py reloads by mtime on the hourly precompute. water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent, OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now does `uv sync` into .venv (one env rule; removes a stale venv/) and enables the timer. water-monitor.service in the repo matched neither the deployed unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).	2026-09-11 21:37:11 +02:00
grabowski	0a4bf843ff	chore: remove empty shell-accident files committed at repo root	2026-09-11 20:54:48 +02:00
grabowski	f6570ac10f	chore: declutter the repo ... Removes 26 tracked files that no longer describe or serve the running system, verified one by one against the whole repo (source, tests, docs, README, Makefile, Dockerfile, .gitea workflows, pyproject, packaging spec) plus dynamic-reference paths, before deletion. Root (11): one-off launch/setup write-ups from the project's first weeks that document events which never happened the way they describe — a github.com publication (the remote is self-hosted Gitea) and a 15-minute scheduler (the service runs hourly). Also .gitlab-ci.yml (unused, CI is .gitea/), .env.postgres and setup.py.backup (a placeholder env file and a backup in version control), and the PyInstaller packaging trio build_executable.py / build_simple.py / ping-river-monitor.spec, which bundled docs that no longer exist and is not how this deploys. docs (9): stale guides superseded by DATABASE_DEPLOYMENT_GUIDE, GITEA_WORKFLOWS, FLOOD_FORECASTING and DATA_SOURCES, plus two snapshots (PROJECT_STATUS, PROJECT_STRUCTURE) describing a 4-file src/ that is now 39. scripts (5): one-shot bootstrap tools already run — init_git.sh/.bat, generate_badges.py, migrate_geolocation.py, encode_password.py. Every inbound reference was fixed rather than left dangling: README doc index and migration section, three Makefile targets, the docs.yml summary step, and the GITEA_WORKFLOWS resource list. src/ is deliberately untouched. The audit proposed removing several live modules; verification showed those proposals were mis-scoped and would have broken production. .gitignore now covers the agent tooling dirs, model eval output and editor/shell leftovers — the working tree had collected 56 zero-byte files named after fragments of shell commands. 136 tests pass; production modules import clean.	2026-08-14 13:45:03 +07:00
grabowski	7e64e0cf18	fix: one current river level, not two ... The verdict banner and the P.1 outlook each wrote state.p1Now from a different feed — the banner from the latest measurement, the outlook from current_level on the forecast rows, which carries whatever the model saw at its as_of. Forecasts are precomputed hourly, so the two drifted apart: production showed 1.66 m in the banner and 1.52 m in the outlook directly below it. Harmless at low water; at flood stage two contradictory river levels on one screen undermine the warning. setP1Level() now arbitrates: freshest timestamp wins, and the replay and demo hooks pass force since they deliberately pin a level that is not the live one. The outlook renders the arbitrated value and clamps the shown peak to at least the current level, so a stale forecast can no longer predict a peak below where the river already is. endReplay drops the replayed level so live data re-arbitrates cleanly. Verified against a stub reproducing the exact production conditions (gauge 1.66 at 08:35 vs forecast 1.52 at 08:00): both now read 1.66; a 2.50 m rise against a stale 1.81 m peak renders 2.50/2.50; the 2024 replay still tracks its frames and returns to live on stop.	2026-08-14 09:25:39 +07:00
grabowski	382daa7d86	perf: backfill one dam per request via the api/dam range endpoint ... GET app.rid.go.th/reservoir/api/dam?dam_id&date_start&date_end returns a single dam's whole date range in one response — Mae Ngat's 2009-today archive is ~4 chunked requests instead of the ~2,900 one-day POSTs the all-dams path needs. Field names differ from api/dams and are mapped in parse_dam_range_records, verified equal on spot-checked dates; the range endpoint also carries DMD_ULevel, the reservoir level in m MSL that api/dams stopped publishing after ~2013. scripts/backfill_rid_reservoir.py defaults to the fast per-dam path (--all-dams keeps the full-fleet crawl, --refresh rewrites stored days). Already-stored dates are still skipped, junk values are still bounded, and the consecutive-failure abort still applies.	2026-08-13 23:10:36 +07:00
grabowski	b02e815d72	feat: Thai localisation and portrait-phone layout for the dashboard ... Thai is the default unless the browser prefers English, chosen by first match in navigator.languages order and remembered in localStorage. A STRINGS table carries both languages (interpolated strings as functions), applyTranslations() drives static markup via data-i18n attributes, and setLang() rebuilds everything the JS renders — including map layers, so popups render in the current language and sensor markers are replaced rather than stacked. Thai dates use the Buddhist era, matching the replay label; station names lead with the reader's language. Portrait phones: the header overflowed a 412 px Android viewport by 64 px, so the page scrolled sideways and the Refresh button sat off screen. The header now wraps into two deliberate rows (DOM order matches visual order, so focus order is unaffected), the map description box is hidden on phones, map height is capped by viewport — including a height-gated rule for landscape phones, whose 850-960 px widths never matched the width breakpoints — and the forecast grid goes single column. Verified in a real browser at 412x915, 360x800, 915x412 and 1440x900: zero horizontal overflow, no desktop change. Review-swarm fixes: a language switch no longer relabels a pinned SIMULATION or the 2024 replay as LIVE DATA (it kept the mode from state); Thai wording corrected where it asserted a rising trend the code never checks, labelled every gauge 'critical', or used a malformed compound; aria-labels, the Leaflet load failure and the flood-stage chips are translated; the language toggle states its action instead of an aria-pressed value that contradicted its label; and Thai font families sit after the Latin stack so they cannot restyle English text.	2026-08-13 23:10:34 +07:00
grabowski	5dc5850df6	docs: source sweep — P.75 already is the Mae Ngat release signal ... Verified from a Thai ISP that lsim.rid.go.th is unreachable (not geo-blocked), then swept for any better-than-daily Mae Ngat source. Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has been a model feature since v1 — the model has always read the dam's actual outflow, hourly and directly. That is the likelier reason the daily reservoir table adds nothing, beyond its publication lag. Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with no archive, so they cannot retrain against past events; the HII collector accumulates them from 2026-08-11 for a post-monsoon revisit. Also documents api/dam (whole per-dam range in one request, vs the ~2,900-request per-day loop) and several cross-check mirrors.	2026-08-13 21:58:47 +07:00
grabowski	70e4da07a0	docs: lsim.rid.go.th is unreachable, not geo-blocked ... Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Correct the earlier note that assumed the timeout was a foreign-network block, and stop pointing the dam-feature follow-up at a host that cannot be reached.	2026-08-13 21:00:24 +07:00
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	6af6fbe02c	fix: survive junk RID dam values — widen storage_pct, bound inserts ... Dam 100602 reports 87798% storage on some days, overflowing NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/ MySQL tables) and every measure column is bounds-checked before insert so out-of-capacity junk becomes NULL instead of a batch-killing error. Rerunning the backfill repairs the days the overflow skipped.	2026-08-13 10:50:58 +07:00
grabowski	ba781465a9	feat: in-memory HII gap-fill in the ML data loader ... load_measurements() (DB path) patches missing station-hours from the hii_waterlevel mirror telemetry: exact mirrors (P.1/P.103/P.20/P.4A/P.67/ P.75/P.82/P.84/P.92) plus bias-corrected P.81 (+9,340 h). Per-station MSL->gauge offset is derived from >=168 h of series overlap, which reproduces the published offsets for exact mirrors and absorbs P.81's bias; P.76/P.77/P.85/P.87 HII twins are different physical sensors and stay excluded. Training and serving share the loader, so both sides see identical filled series; water_measurements is never written.	2026-08-13 10:29:54 +07:00
grabowski	6eafb353b1	feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow ... POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain.	2026-08-13 10:29:34 +07:00
grabowski	811af1625b	feat: dashboard shows the live model version; DB stats count openmeteo_rain ... The flood-outlook subtitle now reads '... · model hgb-v3+<sha> (trained <date>)' from the forecast rows, so the deployed model is always visible on the page. /api/stats adds openmeteo_rain_measurements (guarded — the table may not exist on older DBs) to the whole-DB total, and the Total-datapoints tile note gains a 'forecast rain' entry.	2026-08-12 17:35:23 +07:00
grabowski	160617e87b	feat: openmeteo_rain 2021+ backfill entry point ... rain.backfill_db pushes the full cached Open-Meteo archive into the openmeteo_rain table in 5k-row idempotent upsert chunks; scripts/backfill_rain_db.py is the thin CLI (DB from Config/.env or --db-url). Safe to re-run and safe alongside the hourly live writer.	2026-08-12 17:28:44 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	cbb3bf7369	feat: Open-Meteo catchment rainfall series + rain features + harness variant ... src/ml/rain.py fetches hourly precipitation for five upper-Ping catchment points (Chiang Dao, Mae Taeng, Mae Ngat, Mae Rim, city) from the Open-Meteo forecast-model archive (2021-03 onward, no API key, Bangkok-local timestamps, year-chunked local cache) plus the live forecast endpoint for serving (trailing days + next 48h). features.build_features/build_matrix accept the catchment-mean series and add rain_6h/24h/72h trailing sums and rain_fc24 — the forward 24h sum, a genuine forecast feature (archived forecasts at training time, a real weather forecast at serving; never contains river data). Columns exist only when a series is provided: HGB rejects all-NaN columns at fit, so no-rain training omits them and serving passes an empty series for alignment. evaluate.py gains the rise_rain variant (and --no-rain) so the harness can judge whether rain beats the deployed rise baseline.	2026-08-12 16:39:36 +07:00
grabowski	21e9d2e114	feat: hgb-v2 — regression heads predict rise, recovering flood warning lead ... Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.	2026-08-12 15:43:29 +07:00
grabowski	a0086086a2	feat: rolling-origin event-aware evaluation harness for model variants ... One fold per monsoon season (train <= 30 Apr, test Jun-Nov, 2021-2025) replaces the single fixed holdout that contained only ~4 warning events. Metrics are what matters operationally: sustained first-alert lead vs each observed 3.70m crossing (two consecutive alerting samples required; lookback floored at the previous event's end so multi-peak floods can't launder lead credit), peak error from the prediction actually issued 24h before the peak (3h match tolerance, null on outages), false-alarm episodes (12h gap tolerance), MAE / flood-regime MAE, and a Brier score on warning exceedance — included because sigma cancels algebraically in any p>=0.5 alert metric, so lead times compare predictors while Brier compares uncertainty models. Variants: baseline_abs (current), rise (target = future max - current level), rise_weighted (flood-regime sample weights 1x->5x), and rise_quantile (q50/q90 heads, spread-implied sigma). Harness verified by a 3-agent adversarial review (features bit-identical across fold cutoffs; three metric flaws found and fixed before first use). Also: features.build_labels/build_matrix gain stats_end so the rescue quantile is computed from pre-cutoff data only, closing the label- construction leak flagged in the earlier ML review.	2026-08-12 15:19:26 +07:00
grabowski	98023243af	feat: forecast precompute + issued-forecast archive + dashboard overlay ... The collection-leader worker now precomputes forecasts after every scrape cycle: primes the /forecast cache (users never trigger the multi-second inference — its TTL rises to 4500s so the hourly refresh always wins) and persists every issued forecast to a new forecast_history table keyed by (as_of, station, horizon) with predicted max level, warn/danger probabilities, current level, and model_version. This is the operational record the backtests lacked — predicted-vs-actual becomes a simple join instead of retraining historical models. GET /api/forecast/history/{station} serves the archive (hours or start/end + horizon filters, 5-min edge cache), and the station history chart overlays 'Model 24 h peak (as issued)' as a dashed violet line once data accumulates.	2026-08-12 14:13:39 +07:00
grabowski	731f10910e	perf: stale-while-revalidate caching, bigger executor, cheap DB health probe ... Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine).	2026-08-12 12:32:38 +07:00
grabowski	96fedb3991	perf: single-flight /api/stats; fast-fail health probe ... The on-box rerun after the inline fast path showed the cached endpoints healthy (latest p50 72-160ms, HII ~100ms) but /api/stats at 81% timeouts and /health at 33%: stats had a cache but NO single-flight, so every concurrent miss ran the heavy whole-DB counts (~1.7M rows) in parallel, re-jamming Postgres and the executor — which also dragged uncached history windows into 60s timeouts. /api/stats now computes through _ttl_cached_stale (one computation per 5min TTL, stale served on failure). The health API probe fails fast (5s instead of 30s) and its cache TTL rises to 30s, so a slow upstream can no longer pin executor threads longer than the cache lifetime.	2026-08-12 11:55:49 +07:00
grabowski	0b14e394ad	perf: Cache-Control headers so browsers and edge caches absorb traffic ... Bandwidth diagnosis: iperf shows the proxy<->API VPN link at 65-78 Mbit/s and a fast client pulls 15.7 MB/s from a CDN but only ~66 KB/s from the site — the Caddy host's public uplink is the scarce resource. Max-ages mirror the server cache TTLs (latest 30s, HII 60s, forecast 120s, history/stats/stations 300s, static 1h, dashboard HTML 120s), so repeat views come from browser cache and an edge proxy (e.g. Cloudflare free tier in front of Caddy) can serve most traffic without touching the uplink.	2026-08-12 11:51:43 +07:00
grabowski	d9c65bcf0c	perf: inline cache fast path; cache /health checks ... The on-box load test exposed the real stall: cache HITS were dispatched through asyncio.to_thread, so under load a microsecond lookup queued ~11s behind slow work in the ~8-thread default executor (endpoints answered inline — /forecast 4ms, /api/stats 2ms — while every to_thread endpoint sat at p50 8-17s). Handlers now check TTL caches inline in the async path via _cache_fresh() and only pay for a thread on a miss. /health results are cached for HEALTH_CACHE_TTL_SECONDS (10): its external RID-API probe plus DB query were occupying executor threads on every hit, which is what jammed the pool in the first place.	2026-08-12 11:48:05 +07:00
grabowski	1ec5cfb4df	perf: gzip responses; multi-worker serving with single collection leader ... GZipMiddleware (min 500 bytes) compresses the dashboard HTML ~4x and station JSON up to ~100x, end-to-end through the Caddy TLS terminator — production load testing showed the deployment is bandwidth-bound once the response caches hit, so compression is the capacity lever. WEB_WORKERS (default 2) runs uvicorn multi-process via the app import string. Every worker executes the lifespan, so a localhost lock port (COLLECTION_LEADER_PORT, default 8901) elects exactly one background-collection leader per machine — RID/HII polling stays once-per-cycle instead of once-per-worker; the lock releases with the process. Locust clients now send Accept-Encoding so future runs measure compressed transfer, as browsers do.	2026-08-12 11:34:12 +07:00
grabowski	039af8caac	perf: cache /measurements/latest; stale-on-error fallback for cached endpoints ... The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120).	2026-08-12 11:05:02 +07:00
grabowski	d27ca8bf40	perf: 120s TTL cache with single-flight on /api/hii/*/latest ... These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one.	2026-08-12 10:58:19 +07:00
grabowski	0005f7dce1	feat: codified backtests, honest docs, belt-and-braces serving, perf fixes ... Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).	2026-08-12 10:46:00 +07:00
grabowski	d2d0e655aa	fix: All-time range fills the date pickers with the DB's first record date ... Uses first_timestamp from /api/stats (2018-08-01 fallback) as the start date and today as the end, instead of clearing the pickers.	2026-08-11 17:02:31 +07:00
grabowski	b89c7e1915	feat: flood-verdict banner, violet rain palette, per-station bands, chip contrast, Umami ... UI/UX review backlog round 2 plus analytics: - at-a-glance verdict banner (role=status, aria-live) above the tiles: ok/watch/danger from P.1 level vs the official 3.70 m stage and the 24 h forecast, with a not-an-official-warning disclaimer + ThaiWater link (also added to the forecast card) - rainfall dots switch to a violet sequential ramp so they can never be confused with flow-status colors; legend gains 'Other markers' rows (grey no-data gauges, + bank-height sensors) - history-chart flood bands now use each station's own /forecast thresholds (P.1 official fallback) instead of hardcoded 3.0/4.5 - chipStyle(): contrast-safe text on all code/risk/stage chips (dark ink on amber/ochre, darkened fills for white ink); risk/stage ramps drop the blue step for a monotonic green-ochre-amber-red escalation - plain-language tooltips on the flow and capacity tiles - Umami: client script tag plus opt-out server-side middleware posting api-request events (fire-and-forget, 3 s timeout, never blocks)	2026-08-11 16:56:53 +07:00
grabowski	5848621c77	fix: UI/UX review round 1 — mobile reachability, legend toggle, replay correctness ... From the multi-lens review (27 confirmed findings; this lands the quick high-impact set): - mobile: side-card cap 400px->70vh so the basin-stations expander is reachable; live-pill kept as a dot (title carries the label); replay button no longer wraps; legend returns behind a Legend toggle button (was display:none, which also removed the only rain toggle) - replay: auto-refresh skips while replaying (was clobbering state); Chiang Mai flow tile shows P.1 replay discharge instead of a basin-wide sum mislabelled as P.1 - history: placeholder text in the empty chart, dropdown flips to 'Custom range' when dates are hand-edited, controls hint when no station is selected - a11y/copy: aria-expanded/aria-controls on all three disclosure controls, sensor rows say 'of bank height' vs tile 'channel capacity', P.1 tile note deduped	2026-08-11 16:39:53 +07:00
grabowski	09c1c84153	feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta ... robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404).	2026-08-11 16:33:44 +07:00
grabowski	bd3353e2dc	feat: buildfor.life link in dashboard footer	2026-08-11 16:30:25 +07:00
grabowski	d0018d6529	feat: physically meaningful stat tiles + footer links ... 'Combined discharge' summed sequential mainstem gauges, counting the same water several times — replaced with Chiang Mai river flow (P.1 discharge + % of channel capacity). 'Strongest flow' becomes 'Most stressed gauge' (max discharge_percent basin-wide). Reporting-stations note shows the ThaiWater/HII station count. Footer links to the source repo and /docs API reference.	2026-08-11 16:25:41 +07:00
grabowski	df31de092b	fix: collapse additional-stations list by default; no scroll-jump on station click ... The 97-row ThaiWater/HII list was starving the RID flow list down to one visible row. The section header is now a click/keyboard toggle (collapsed by default, arrow indicator); searching auto-expands it so hidden stations stay findable, and the flow list keeps a 280px minimum when both are open. Selecting a station no longer scrollIntoViews the history card.	2026-08-11 16:15:34 +07:00
grabowski	e27d418a1a	feat: station search in side panel; DB stats cover HII tables ... A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'.	2026-08-11 16:06:39 +07:00
grabowski	ba4710b243	feat: 'Last N' labels on history ranges; dropdown mirrors into date pickers ... Quick-range options read 'Last 24 hours' … 'Last 90 days'; choosing one fills the from/to date inputs with today and today-N (All time clears them). Dropdown-driven loads still fetch the precise trailing-hours window; hand-edited dates take over via state.useDates.	2026-08-11 15:58:28 +07:00
grabowski	9516857d44	feat: date-range picker on the station history panel; drop PostgreSQL naming ... /measurements/history/{code} accepts optional start/end date params that override the hours window (end date inclusive). The dashboard history card gains from/to date inputs beside the quick-range dropdown — explicit dates win, changing the dropdown clears them. All user-facing 'PostgreSQL history' labels renamed to 'Station history'.	2026-08-11 15:52:40 +07:00
grabowski	d29d49eac7	feat: collapsible flood outlook (P.1 only by default) + dashboard fixes ... The flood-risk card now shows just the Chiang Mai P.1 outlook; the per-station forecast grid collapses behind a Show/Hide expander. Fixes: the rain-layer toggle was unclickable (.map-overlay pointer-events:none now re-enabled on the legend), long HII station codes (MOU302, FOP015) wrap inside their chips instead of clipping, and the sensor panel sorts by bank percentage with no-data stations last.	2026-08-11 15:48:20 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	33d8baa8dd	fix: composite (station_id, timestamp) PK on hii_* measurement tables ... TimescaleDB create_hypertable rejects tables whose primary key omits the partitioning column; the surrogate id PK served no purpose, so use the natural key directly.	2026-08-11 14:50:27 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00
grabowski	7befc82ff5	feat: database stats on the dashboard via GET /api/stats ... New SQLAdapter.get_database_stats() aggregates totals, station count, date range, and hourly-slot coverage in one query per dialect. The endpoint follows the existing 503-guard/to_thread/TTL-cache pattern; the dashboard gains a five-tile stats strip on the existing refresh cadence. Coverage denominator is hour-truncated so off-hour endpoints cannot push it past 100%; MySQL slot expression avoids % characters that would break under pyformat bind interpolation.	2026-08-10 23:27:36 +07:00
grabowski	5e62ea529d	feat: --fill-gaps all repairs missing data across the whole DB range ... Gap detection is now hour-granular: days with partial data (missing hourly slots) are re-fetched, not just days with no rows at all. The API's hour-24-is-next-midnight quirk is handled by re-fetching day D-1 when day D is missing its 00:00 slot. SQL adapters gain single-query range and recorded-hours lookups; non-SQL backends fall back to the old day-granular check.	2026-08-10 22:13:58 +07:00
grabowski	410faeddd5	fix: protect admin API endpoints; stop rejecting flood-peak measurements ... Security: POST/PUT/DELETE /stations, POST /scrape/trigger and GET /config now require an X-API-Key header matching ADMIN_API_KEY. Secure by default - with no key configured those endpoints return 503 instead of being open. Comparison via secrets.compare_digest. Dashboard and read endpoints stay public. Data: the validator rejected any measurement whose discharge_percent exceeded 200 - which silently deleted the Oct 2024 record-flood peaks (the river genuinely ran at 201-226% of channel capacity). The cap is now 500%, and an out-of-range percent nulls that auxiliary field instead of discarding the whole row (water level and discharge are the data that matter). Surfaced by the user's historical backfill log.	2026-08-10 18:47:08 +07:00
grabowski	27fa292e09	docs: September 2025 flood render - deployed config, 24 h warning, peak within 7 cm ... Hour-by-hour chart of the 25-28 Sep 2025 event as forecast by the exact deployed configuration (trained through 2024, event unseen): first alert 26 Sep 18:00, flooding began 27 Sep 18:00 (24 h lead), predicted peak 4.00 m vs actual 3.93 m. The near-miss 3.51 m crest on 26 Sep correctly never alerted.	2026-08-10 18:35:55 +07:00
grabowski	f0183faa62	feat: discharge-driven river animation speed, replay stat tiles, hourly doc render ... - river dash animation now flows at a speed continuously derived from each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via inline per-path animation-duration, so it updates live and per-frame during the replay (CSS speed classes removed - setStyle cannot change classes) - the replay drives the Combined discharge and Strongest flow stat tiles each frame (marked '2024 replay'), restored on finish - docs: hour-by-hour detection detail render (22-28 Sep 2024) showing the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the 24 h warning between them; embedded with commentary - Matrix alerts now link to https://water.buildfor.life/ (override via ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link	2026-08-10 18:29:01 +07:00
grabowski	6112c681a7	docs: render of actual vs predicted through the 2024 flood; adaptive replay speed ... docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead predicted peak issued at each hour by a model trained only on pre-flood data, with the warning-probability panel below (first alert 24 Sep 01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's headline-validation section with an honest reading, including the ~0.4 m peak under-prediction. Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when risk is elevated, 1 h (hour-by-hour) while the model is alerting or the river is near/above flood stage - so viewers can watch the detection sequence unfold.	2026-08-10 18:10:48 +07:00
grabowski	199b0e3483	feat: replay shows when the model would have detected and warned ... The 2024 snapshot now carries a model track: p_warning and predicted 24 h peak from HGB heads trained ONLY on pre-flood data (an honest backtest, cutoff 2024-08-31), plus its calibration sigma. During replay: - the map clock shows the system's state at each moment: 'model: no flood expected' -> 'elevated risk' -> amber 'MODEL ALERT - flooding within 24 h likely (predicted peak X m)' -> red 'FLOODING' once the river actually crosses stage 1. First alert fires 24 Sep 01:00, a full day before the water crossed the flood line on 25 Sep. - the flood-risk outlook stage chips re-render each frame from the historic model prediction (sigmoid over the stage levels), so the whole outlook panel time-travels with the map.	2026-08-10 18:07:57 +07:00
grabowski	d015420b66	feat: full-basin 2024 flood replay with clock and demo indicator ... The replay now drives the entire map from a 120 kB all-station snapshot (static/flood-2024-all.json: 649 hourly frames x 16 stations of level + discharge, forward-filled on an aligned grid): - station markers recolor/resize per frame from historic discharge - river segments restyle per frame (color/width from the same nearest-gauge grading as live mode) - flood zones flood/recede from P.1's historic level - a large clock overlay on the map shows replay date/time, P.1 level and total basin flow - the LIVE DATA pill switches to an amber '2024 REPLAY' (or 'SIMULATION' for the demo_level/demo_rise hooks) and back on finish - replay end or stop restores the live view via a full dashboard reload Replaces the P.1-only snapshot (flood-2024-p1.json removed).	2026-08-10 17:59:06 +07:00
grabowski	0b885e572d	chore: gitignore .playwright-mcp browser artifacts	2026-08-10 17:45:21 +07:00
grabowski	0672ea5ffa	feat: Oct-2024 flood replay, simulation hooks, and static replay data ... - header button 'Replay Oct 2024 flood': animates the real Sep 15 - Oct 12 2024 P.1 hourly series (~45 s) through the flood-zone display; zones flood blue as the river climbs to the 5.30 m record and recede as it falls; click again to stop; restores live state when done - replay reads a pre-extracted 15 kB static snapshot (static/flood-2024-p1.json, 335 frames) instead of pulling the 4.6 MB all-time history on every click; falls back to the history API if the snapshot is missing - demo hooks for testing/presentations: ?demo_level=4.4 pins a simulated P.1 level, ?demo_rise=1 animates rising water to 5.30 m; both label the outlook line as SIMULATION with the real level alongside	2026-08-10 17:44:36 +07:00
grabowski	c62a03e778	feat: flooded zones render as water and auto-surface ... When P.1's current level reaches a zone's trigger level the zone fills water-blue (solid border, .62 opacity) instead of the amber risk ramp, and the zone layer adds itself to the map automatically - no toggle needed during an actual flood. A manual hide sets a session flag so auto-show does not fight the user.	2026-08-10 17:22:27 +07:00
grabowski	711629682d	feat: per-marker icon field for custom markers ... custom-markers.json entries accept an optional "icon" (any emoji, default pin); seeded homes/office icons for the existing markers.	2026-08-10 17:20:12 +07:00
grabowski	7e7d244efa	feat: add baan boe marker	2026-08-10 17:19:31 +07:00
grabowski	786b8514ea	feat: custom map markers with per-location flood-zone risk ... src/static/custom-markers.json holds user points of interest ({name, lat, lon, optional note}); the dashboard renders them as pin markers whose popups show which inundation zone the point sits in, its P.1 trigger level, and the live 24 h exceedance probability (ray-cast point-in-polygon against the zone geojson; smallest matching zone wins). Seeded with the owner's four properties.	2026-08-10 17:16:28 +07:00
grabowski	a17509f3ea	feat: replace digitized flood zones with owner-traced polygons ... The 7 Chiang Mai inundation zones are now hand-traced by the project owner against the real basemap (cnx_flood.geojson), replacing the scan-digitized approximation and its georeferencing error entirely. Features are ordered zone 7 -> 1 so the earlier-flooding (smaller) zones render on top of the wider extents in Leaflet.	2026-08-10 17:12:23 +07:00
grabowski	4f12360960	fix: shift flood zones 1.05 km south, anchored to Nawarat Bridge ... The P.1 marker (Nawarat Bridge, 18.7875) sat inside zone 2 near its top edge, but the bridge IS Chang Khlan's northern boundary - the layer was ~0.0095 deg too far north (the district-centroid correction in c4e0fb6 overshot). Zone 2's north edge now lands at the bridge.	2026-08-10 16:47:14 +07:00
grabowski	c4e0fb6bae	fix: re-georeference flood zones and make them clearly visible ... The previous affine fit slid along the mostly north-south river (an ill-conditioned direction) leaving the zones ~1 km south-west of their true position. New approach: extract the scanned river band by color, match it to the OSM Ping mainstem centerline by arc length (which pins the along-river position), then correct residual translation using known district locations cross-checked against the river residual (533 m -> ~300 m median; Chang Khlan zone centroid now within 200 m). Mountain-area artifacts are clipped away via the municipality hull. Zones were also nearly invisible at fillOpacity 0.16 - base opacity is now 0.38, rising with the live 24 h exceedance probability, and 0.72 with a solid red border once the river is at or above a zone's level.	2026-08-10 16:38:09 +07:00
grabowski	f05289e628	fix: CI matrix to Python 3.11 only until pandas is upgraded ... pandas 2.0.3 has no cp312 wheels, so the 3.12 matrix leg fails during dependency install. The deployment runs 3.11.	2026-08-10 16:15:32 +07:00
grabowski	3b919adc56	feat: vector flood-zone polygons replace the scanned-map overlay ... Digitize the 7 Chiang Mai inundation zones from the municipal map into geojson polygons: pixels classified by legend color, vectorized via marching squares, and georeferenced by a 6-parameter affine fitted least-squares to the OSM Ping centerline (109 m median residual after outlier-filtered refits - the scanned image overlay could never scale correctly and is removed). The dashboard renders the zones as a Leaflet geoJSON layer styled live from the forecast: fill opacity scales with each zone's 24 h exceedance probability, zones whose trigger level the river has already reached get a solid red border, and each polygon's popup shows its trigger level and live probability. Zone styles refresh with every forecast load.	2026-08-10 16:14:34 +07:00
grabowski	6c0bb024a8	chore: remove stray zero-byte files accidentally committed	2026-08-10 15:57:19 +07:00
grabowski	ecd34177bb	fix: backtest/review findings in the flood-ML package ... From the adversarial review and threshold backtest (swarm verification): - predict.py: when a bundle's trained thresholds differ from the current config (deploy before retrain), skip its stale classifier heads and derive p_warning/p_danger from the regression + sigma against the CURRENT thresholds - the dashboard can no longer show contradictory old-threshold classifier output next to new-threshold stages - features.py: decouple the low-coverage regression-label rescue from the warning threshold (now the station's own p97.5 level); the old coupling silently dropped 34% of P.5's regression training rows and cost +46% MAE when its threshold rose - features.py: P.82 danger 3.80 -> 3.75 (3.80 was above the station's 8-year maximum of 3.78, so danger could never train or fire) - data.py / predict.py: anchor models/cache paths to the repo root; the relative paths silently returned zero rows when run from another CWD - annotate P.4A thresholds as low-confidence (11 supporting readings) 47 tests pass. Retrain required for the label-rescue and P.82 changes to reach the classifier heads.	2026-08-10 15:57:01 +07:00
grabowski	9cac9c4d2a	style: apply black/isort across the repo; make CI mypy advisory ... The push-CI gates (black/isort/mypy) had never actually run before the branch-trigger fix, and the codebase predates them. Formatting is now black/isort clean repo-wide. mypy keeps running but non-blocking: 86 pre-existing errors are a separate cleanup, not a gate to hold hostage.	2026-08-10 15:57:00 +07:00
grabowski	300c0e0b6f	fix: trigger CI on master (repo default branch), drop unsupported Python matrix entries ... Workflows listened on 'main'/'develop' but the repo's default branch is master, so push events never started a job (every historical run is a schedule event). Point push/PR triggers and the deploy-gate refs at master, and trim the test matrix to 3.11/3.12 to match requires-python >=3.11.	2026-08-10 15:46:24 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	29f4b5818d	fix: require Python >=3.11 and lock scikit-learn deps for uv ... scikit-learn 1.9.0 supports only Python >=3.11, so uv could not resolve the >=3.9 range. The deployment runs 3.11. Also pins numpy back to 1.26.4 in the lockfile (pandas 2.0.3 ABI).	2026-08-10 14:44:54 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00
grabowski	49a3de0087	fix: restore station telemetry and make river flow visualization data-driven ... Station selection showed no history since 21ca844: Chart.js v4 datasets had parsing:false with plain number arrays, drawing empty axes. Remove the flag so the chart parses values again. Backend hardening for the same flow: - /measurements/history/{code} no longer 503s on non-Postgres configs; it falls back to the configured adapter (reversed to ascending order) - DB_TYPE defaults to postgresql when POSTGRES_CONNECTION_STRING is set and DB_TYPE is unset, so the .env psql wins over the sqlite default - zero readings (0.0) are no longer coerced to None, which would fail MeasurementResponse validation and 500 /measurements/latest Map visualization: - river segments are now colored, widened and dash-speed-animated by the discharge at the nearest gauge (same scale as the marker legend) - fix z-order bug that drew the animated flow line behind its casing - legend entries for river lines, reduced-motion fallback River geometry: rebuild ping-river-network.geojson from Overpass (110 -> 202 features), restoring missing Ping mainstem reaches through the Bhumibol reservoir and the Tak-Kamphaeng Phet braided section (unnamed waterway=river ways in OSM), with short synthetic connectors (connector: true) bridging remaining sub-8 km holes.	2026-08-10 10:30:05 +07:00
grabowski	af1909db73	fix: restore history chart initialization after flood bands change	2026-08-10 09:46:34 +07:00
grabowski	76c934e475	feat: add _calculate_discharge to estimate from water level when DB discharge is NULL	2026-08-09 18:09:27 +07:00
grabowski	32e455783a	fix: harden loadHistory against race conditions and canvas reuse	2026-08-09 18:03:41 +07:00
grabowski	b3ea340bbd	fix: fix chart reuse error and harden flood-bands plugin	2026-08-09 17:56:30 +07:00
grabowski	ba0348b580	feat: add flood-zone bands (normal/warning/danger) to history chart	2026-08-09 17:54:11 +07:00
grabowski	21ca84444d	perf: downsample history to daily averages and fix chart overflow	2026-08-09 17:48:31 +07:00
grabowski	9f26b32c86	fix: bump history limits to 100k so all-time shows full dataset	2026-08-09 17:41:14 +07:00
grabowski	bbbf548d66	fix: align web API limit cap with 10k and clean up postgres_history imports	2026-08-09 17:36:58 +07:00
grabowski	ef106fce8d	fix: include year in history chart labels	2026-08-09 17:21:20 +07:00
grabowski	33f2dd45f4	fix: render history chart timestamps in ICT (Asia/Bangkok)	2026-08-09 17:19:28 +07:00
grabowski	2fe1dcf4da	feat: add 5-minute TTL cache for PostgreSQL history queries	2026-08-09 17:15:47 +07:00
grabowski	c00a26402a	fix: update test limit boundary to match new 10k ceiling	2026-08-09 17:13:51 +07:00
grabowski	abfac1d3bb	fix: correct all-time hours value and update backend limit	2026-08-09 17:12:22 +07:00
grabowski	6f2a8a0d8f	feat: add all-time history option and increase limit to 10k	2026-08-09 17:11:26 +07:00
grabowski	9ef7798e00	feat: load history when clicking station markers on map	2026-08-09 17:04:21 +07:00
grabowski	ae5d0a13d7	[verified] feat: add live river dashboard ... Add mapped river and ThaiWater sensor layers, PostgreSQL history charts, API endpoints, and dashboard tests.	2026-08-09 16:59:25 +07:00
grabowski	e5936d5717	Move dashboard HTML out of web_api into a static file ... Extract the inline root() dashboard markup into src/static/dashboard.html, loaded once at import. Keeps HTML out of the Python module (web_api 616 -> 576 lines) with a defensive fallback if the file is missing. Full <500 compliance for web_api still needs the endpoints split into APIRouter modules; tracked as remaining #4 work.	2026-07-22 14:28:34 +07:00
grabowski	08dc536c93	Add unit tests for RID API response parsing ... Cover the previously-untested, highest-risk parsing in fetch_water_data_for_date by mocking the HTTP call: - hour 1-23 map to the same day; hour 24 rolls to next-day midnight - qvalues "***" / None yield discharge None (no crash) - None water level is skipped - out-of-range (0, 25) and empty hourlytime rows are skipped - missing "rows" key returns an empty list This gives the scraper a safety net before its module is split.	2026-07-22 14:24:28 +07:00
grabowski	ad7f7b8c76	Extract API Pydantic models into schemas.py ... Move the seven request/response models out of web_api.py into a dedicated src/schemas.py (separation of concerns; first step of the file-size cleanup). web_api.py imports them back, so behaviour is unchanged. Note: web_api.py is still over the 500-line guideline; the remaining bulk is the inline HTML dashboard in root(), to be extracted in a follow-up.	2026-07-22 14:07:31 +07:00
grabowski	12b7f9f422	Add assert-based pytest coverage for recent fixes ... - tests/conftest.py: put repo root on sys.path so `import src...` resolves under pytest regardless of invocation directory. - test_matrix_formatting.py: lock in HTML formatted_body + plain-text fallback, URL linkification, HTML escaping, and send_alert field rendering. - test_station_persistence.py: cover default-load, save/reload round-trip (incl. Thai text), runtime-file precedence, and atomic-write cleanup. These are real assert-based tests (unlike the existing print-style scripts) so CI can gate on them. 13 tests, all passing.	2026-07-22 14:02:06 +07:00
grabowski	ce31a5254e	Harden install.sh per security review ... - .env now chmod 0600 and APP_DIR chmod 0750 after chown, so the Matrix token and DB credentials are not world-readable. - uv auto-install (curl | sh as root) is now opt-in via AUTO_INSTALL_UV=1 and pins a specific uv version; otherwise the script requires uv to be pre-installed and fails with instructions, avoiding unattended remote code execution as root.	2026-07-22 14:00:44 +07:00
grabowski	ab8a10dd75	Add install.sh and fix service unit placeholder ... - scripts/install.sh: one-command hardened deploy (creates the water-monitor system user, deploys to /opt, builds a uv-managed venv, installs and enables the systemd unit). Idempotent; excludes .env/*.db/stations.json from sync so runtime state is preserved. - Fix placeholder Documentation= URL in water-monitor.service. - README: document the script as the primary systemd install path, with manual steps kept as a fallback.	2026-07-22 12:55:17 +07:00
grabowski	4bc3d82773	Persist station CRUD across restarts via JSON config ... Station CRUD via the API previously mutated the scraper's in-memory station_mapping only, so changes were lost on restart (and the systemd service auto-restarts). - Extract the 130-line hardcoded station_mapping into bundled defaults at src/data/stations.json; the scraper loads from a runtime-writable config file (STATION_CONFIG_PATH, default stations.json) and falls back to the bundled defaults to seed it. - Add scraper.save_stations() with an atomic temp-file + os.replace write. - create/update/delete station endpoints now persist and roll back the in-memory change if the write fails; re-raise HTTPException so persistence errors surface as real 500s instead of being swallowed. - Backend-agnostic (works for the VictoriaMetrics deployment, which has no relational stations table). Runtime stations.json is gitignored. Also clears pre-existing flake8 debt in water_scraper_v3.py (unused imports, long lines, duplicate logging import) and dedupes the User-Agent to Config.USER_AGENT.	2026-07-22 12:51:29 +07:00
grabowski	6e78225d00	Fix optional-discharge crashes and dedupe measurement mapping ... - MeasurementResponse.discharge is now Optional[float]; measurements with a null discharge no longer raise a Pydantic ValidationError (HTTP 500) on the /measurements/latest and /measurements/station endpoints. - InfluxDB save_measurements guards float(discharge) against None instead of crashing with TypeError. - Extract the duplicated measurement->response mapping into a single _to_measurement_response helper used by both measurement endpoints.	2026-07-22 12:34:46 +07:00