155 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
grabowski	764764e07e	feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote ... train_all() now raises RainUnavailableError when use_rain=True and the Open-Meteo history cannot be loaded, instead of logging a warning and writing gauge-only (v2) bundles over the deployed v3 set -- which is what the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit way to get v2. CLI exits 2 with a one-line error. Three tests cover the guard, the opt-out, and the v3 happy path. scripts/retrain.sh trains into models/.staging, refuses to promote unless metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles into place (previous generation kept in models/.previous). No API restart: predict.py reloads by mtime on the hourly precompute. water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent, OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now does `uv sync` into .venv (one env rule; removes a stale venv/) and enables the timer. water-monitor.service in the repo matched neither the deployed unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).	2026-09-11 21:37:11 +02:00
grabowski	0a4bf843ff	chore: remove empty shell-accident files committed at repo root	2026-09-11 20:54:48 +02:00
grabowski	f6570ac10f	chore: declutter the repo ... Removes 26 tracked files that no longer describe or serve the running system, verified one by one against the whole repo (source, tests, docs, README, Makefile, Dockerfile, .gitea workflows, pyproject, packaging spec) plus dynamic-reference paths, before deletion. Root (11): one-off launch/setup write-ups from the project's first weeks that document events which never happened the way they describe — a github.com publication (the remote is self-hosted Gitea) and a 15-minute scheduler (the service runs hourly). Also .gitlab-ci.yml (unused, CI is .gitea/), .env.postgres and setup.py.backup (a placeholder env file and a backup in version control), and the PyInstaller packaging trio build_executable.py / build_simple.py / ping-river-monitor.spec, which bundled docs that no longer exist and is not how this deploys. docs (9): stale guides superseded by DATABASE_DEPLOYMENT_GUIDE, GITEA_WORKFLOWS, FLOOD_FORECASTING and DATA_SOURCES, plus two snapshots (PROJECT_STATUS, PROJECT_STRUCTURE) describing a 4-file src/ that is now 39. scripts (5): one-shot bootstrap tools already run — init_git.sh/.bat, generate_badges.py, migrate_geolocation.py, encode_password.py. Every inbound reference was fixed rather than left dangling: README doc index and migration section, three Makefile targets, the docs.yml summary step, and the GITEA_WORKFLOWS resource list. src/ is deliberately untouched. The audit proposed removing several live modules; verification showed those proposals were mis-scoped and would have broken production. .gitignore now covers the agent tooling dirs, model eval output and editor/shell leftovers — the working tree had collected 56 zero-byte files named after fragments of shell commands. 136 tests pass; production modules import clean.	2026-08-14 13:45:03 +07:00
grabowski	7e64e0cf18	fix: one current river level, not two ... The verdict banner and the P.1 outlook each wrote state.p1Now from a different feed — the banner from the latest measurement, the outlook from current_level on the forecast rows, which carries whatever the model saw at its as_of. Forecasts are precomputed hourly, so the two drifted apart: production showed 1.66 m in the banner and 1.52 m in the outlook directly below it. Harmless at low water; at flood stage two contradictory river levels on one screen undermine the warning. setP1Level() now arbitrates: freshest timestamp wins, and the replay and demo hooks pass force since they deliberately pin a level that is not the live one. The outlook renders the arbitrated value and clamps the shown peak to at least the current level, so a stale forecast can no longer predict a peak below where the river already is. endReplay drops the replayed level so live data re-arbitrates cleanly. Verified against a stub reproducing the exact production conditions (gauge 1.66 at 08:35 vs forecast 1.52 at 08:00): both now read 1.66; a 2.50 m rise against a stale 1.81 m peak renders 2.50/2.50; the 2024 replay still tracks its frames and returns to live on stop.	2026-08-14 09:25:39 +07:00
grabowski	382daa7d86	perf: backfill one dam per request via the api/dam range endpoint ... GET app.rid.go.th/reservoir/api/dam?dam_id&date_start&date_end returns a single dam's whole date range in one response — Mae Ngat's 2009-today archive is ~4 chunked requests instead of the ~2,900 one-day POSTs the all-dams path needs. Field names differ from api/dams and are mapped in parse_dam_range_records, verified equal on spot-checked dates; the range endpoint also carries DMD_ULevel, the reservoir level in m MSL that api/dams stopped publishing after ~2013. scripts/backfill_rid_reservoir.py defaults to the fast per-dam path (--all-dams keeps the full-fleet crawl, --refresh rewrites stored days). Already-stored dates are still skipped, junk values are still bounded, and the consecutive-failure abort still applies.	2026-08-13 23:10:36 +07:00
grabowski	b02e815d72	feat: Thai localisation and portrait-phone layout for the dashboard ... Thai is the default unless the browser prefers English, chosen by first match in navigator.languages order and remembered in localStorage. A STRINGS table carries both languages (interpolated strings as functions), applyTranslations() drives static markup via data-i18n attributes, and setLang() rebuilds everything the JS renders — including map layers, so popups render in the current language and sensor markers are replaced rather than stacked. Thai dates use the Buddhist era, matching the replay label; station names lead with the reader's language. Portrait phones: the header overflowed a 412 px Android viewport by 64 px, so the page scrolled sideways and the Refresh button sat off screen. The header now wraps into two deliberate rows (DOM order matches visual order, so focus order is unaffected), the map description box is hidden on phones, map height is capped by viewport — including a height-gated rule for landscape phones, whose 850-960 px widths never matched the width breakpoints — and the forecast grid goes single column. Verified in a real browser at 412x915, 360x800, 915x412 and 1440x900: zero horizontal overflow, no desktop change. Review-swarm fixes: a language switch no longer relabels a pinned SIMULATION or the 2024 replay as LIVE DATA (it kept the mode from state); Thai wording corrected where it asserted a rising trend the code never checks, labelled every gauge 'critical', or used a malformed compound; aria-labels, the Leaflet load failure and the flood-stage chips are translated; the language toggle states its action instead of an aria-pressed value that contradicted its label; and Thai font families sit after the Latin stack so they cannot restyle English text.	2026-08-13 23:10:34 +07:00
grabowski	5dc5850df6	docs: source sweep — P.75 already is the Mae Ngat release signal ... Verified from a Thai ISP that lsim.rid.go.th is unreachable (not geo-blocked), then swept for any better-than-daily Mae Ngat source. Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has been a model feature since v1 — the model has always read the dam's actual outflow, hourly and directly. That is the likelier reason the daily reservoir table adds nothing, beyond its publication lag. Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with no archive, so they cannot retrain against past events; the HII collector accumulates them from 2026-08-11 for a post-monsoon revisit. Also documents api/dam (whole per-dam range in one request, vs the ~2,900-request per-day loop) and several cross-check mirrors.	2026-08-13 21:58:47 +07:00
grabowski	70e4da07a0	docs: lsim.rid.go.th is unreachable, not geo-blocked ... Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Correct the earlier note that assumed the timeout was a foreign-network block, and stop pointing the dam-feature follow-up at a host that cannot be reached.	2026-08-13 21:00:24 +07:00
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	6af6fbe02c	fix: survive junk RID dam values — widen storage_pct, bound inserts ... Dam 100602 reports 87798% storage on some days, overflowing NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/ MySQL tables) and every measure column is bounds-checked before insert so out-of-capacity junk becomes NULL instead of a batch-killing error. Rerunning the backfill repairs the days the overflow skipped.	2026-08-13 10:50:58 +07:00
grabowski	ba781465a9	feat: in-memory HII gap-fill in the ML data loader ... load_measurements() (DB path) patches missing station-hours from the hii_waterlevel mirror telemetry: exact mirrors (P.1/P.103/P.20/P.4A/P.67/ P.75/P.82/P.84/P.92) plus bias-corrected P.81 (+9,340 h). Per-station MSL->gauge offset is derived from >=168 h of series overlap, which reproduces the published offsets for exact mirrors and absorbs P.81's bias; P.76/P.77/P.85/P.87 HII twins are different physical sensors and stay excluded. Training and serving share the loader, so both sides see identical filled series; water_measurements is never written.	2026-08-13 10:29:54 +07:00
grabowski	6eafb353b1	feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow ... POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain.	2026-08-13 10:29:34 +07:00
grabowski	811af1625b	feat: dashboard shows the live model version; DB stats count openmeteo_rain ... The flood-outlook subtitle now reads '... · model hgb-v3+<sha> (trained <date>)' from the forecast rows, so the deployed model is always visible on the page. /api/stats adds openmeteo_rain_measurements (guarded — the table may not exist on older DBs) to the whole-DB total, and the Total-datapoints tile note gains a 'forecast rain' entry.	2026-08-12 17:35:23 +07:00
grabowski	160617e87b	feat: openmeteo_rain 2021+ backfill entry point ... rain.backfill_db pushes the full cached Open-Meteo archive into the openmeteo_rain table in 5k-row idempotent upsert chunks; scripts/backfill_rain_db.py is the thin CLI (DB from Config/.env or --db-url). Safe to re-run and safe alongside the hourly live writer.	2026-08-12 17:28:44 +07:00
grabowski	df0ae8cda3	feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate ... The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.	2026-08-12 17:05:01 +07:00
grabowski	cbb3bf7369	feat: Open-Meteo catchment rainfall series + rain features + harness variant ... src/ml/rain.py fetches hourly precipitation for five upper-Ping catchment points (Chiang Dao, Mae Taeng, Mae Ngat, Mae Rim, city) from the Open-Meteo forecast-model archive (2021-03 onward, no API key, Bangkok-local timestamps, year-chunked local cache) plus the live forecast endpoint for serving (trailing days + next 48h). features.build_features/build_matrix accept the catchment-mean series and add rain_6h/24h/72h trailing sums and rain_fc24 — the forward 24h sum, a genuine forecast feature (archived forecasts at training time, a real weather forecast at serving; never contains river data). Columns exist only when a series is provided: HGB rejects all-NaN columns at fit, so no-rain training omits them and serving passes an empty series for alignment. evaluate.py gains the rise_rain variant (and --no-rain) so the harness can judge whether rain beats the deployed rise baseline.	2026-08-12 16:39:36 +07:00
grabowski	21e9d2e114	feat: hgb-v2 — regression heads predict rise, recovering flood warning lead ... Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.	2026-08-12 15:43:29 +07:00
grabowski	a0086086a2	feat: rolling-origin event-aware evaluation harness for model variants ... One fold per monsoon season (train <= 30 Apr, test Jun-Nov, 2021-2025) replaces the single fixed holdout that contained only ~4 warning events. Metrics are what matters operationally: sustained first-alert lead vs each observed 3.70m crossing (two consecutive alerting samples required; lookback floored at the previous event's end so multi-peak floods can't launder lead credit), peak error from the prediction actually issued 24h before the peak (3h match tolerance, null on outages), false-alarm episodes (12h gap tolerance), MAE / flood-regime MAE, and a Brier score on warning exceedance — included because sigma cancels algebraically in any p>=0.5 alert metric, so lead times compare predictors while Brier compares uncertainty models. Variants: baseline_abs (current), rise (target = future max - current level), rise_weighted (flood-regime sample weights 1x->5x), and rise_quantile (q50/q90 heads, spread-implied sigma). Harness verified by a 3-agent adversarial review (features bit-identical across fold cutoffs; three metric flaws found and fixed before first use). Also: features.build_labels/build_matrix gain stats_end so the rescue quantile is computed from pre-cutoff data only, closing the label- construction leak flagged in the earlier ML review.	2026-08-12 15:19:26 +07:00
grabowski	98023243af	feat: forecast precompute + issued-forecast archive + dashboard overlay ... The collection-leader worker now precomputes forecasts after every scrape cycle: primes the /forecast cache (users never trigger the multi-second inference — its TTL rises to 4500s so the hourly refresh always wins) and persists every issued forecast to a new forecast_history table keyed by (as_of, station, horizon) with predicted max level, warn/danger probabilities, current level, and model_version. This is the operational record the backtests lacked — predicted-vs-actual becomes a simple join instead of retraining historical models. GET /api/forecast/history/{station} serves the archive (hours or start/end + horizon filters, 5-min edge cache), and the station history chart overlays 'Model 24 h peak (as issued)' as a dashed violet line once data accumulates.	2026-08-12 14:13:39 +07:00
grabowski	731f10910e	perf: stale-while-revalidate caching, bigger executor, cheap DB health probe ... Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine).	2026-08-12 12:32:38 +07:00
grabowski	96fedb3991	perf: single-flight /api/stats; fast-fail health probe ... The on-box rerun after the inline fast path showed the cached endpoints healthy (latest p50 72-160ms, HII ~100ms) but /api/stats at 81% timeouts and /health at 33%: stats had a cache but NO single-flight, so every concurrent miss ran the heavy whole-DB counts (~1.7M rows) in parallel, re-jamming Postgres and the executor — which also dragged uncached history windows into 60s timeouts. /api/stats now computes through _ttl_cached_stale (one computation per 5min TTL, stale served on failure). The health API probe fails fast (5s instead of 30s) and its cache TTL rises to 30s, so a slow upstream can no longer pin executor threads longer than the cache lifetime.	2026-08-12 11:55:49 +07:00
grabowski	0b14e394ad	perf: Cache-Control headers so browsers and edge caches absorb traffic ... Bandwidth diagnosis: iperf shows the proxy<->API VPN link at 65-78 Mbit/s and a fast client pulls 15.7 MB/s from a CDN but only ~66 KB/s from the site — the Caddy host's public uplink is the scarce resource. Max-ages mirror the server cache TTLs (latest 30s, HII 60s, forecast 120s, history/stats/stations 300s, static 1h, dashboard HTML 120s), so repeat views come from browser cache and an edge proxy (e.g. Cloudflare free tier in front of Caddy) can serve most traffic without touching the uplink.	2026-08-12 11:51:43 +07:00
grabowski	d9c65bcf0c	perf: inline cache fast path; cache /health checks ... The on-box load test exposed the real stall: cache HITS were dispatched through asyncio.to_thread, so under load a microsecond lookup queued ~11s behind slow work in the ~8-thread default executor (endpoints answered inline — /forecast 4ms, /api/stats 2ms — while every to_thread endpoint sat at p50 8-17s). Handlers now check TTL caches inline in the async path via _cache_fresh() and only pay for a thread on a miss. /health results are cached for HEALTH_CACHE_TTL_SECONDS (10): its external RID-API probe plus DB query were occupying executor threads on every hit, which is what jammed the pool in the first place.	2026-08-12 11:48:05 +07:00
grabowski	1ec5cfb4df	perf: gzip responses; multi-worker serving with single collection leader ... GZipMiddleware (min 500 bytes) compresses the dashboard HTML ~4x and station JSON up to ~100x, end-to-end through the Caddy TLS terminator — production load testing showed the deployment is bandwidth-bound once the response caches hit, so compression is the capacity lever. WEB_WORKERS (default 2) runs uvicorn multi-process via the app import string. Every worker executes the lifespan, so a localhost lock port (COLLECTION_LEADER_PORT, default 8901) elects exactly one background-collection leader per machine — RID/HII polling stays once-per-cycle instead of once-per-worker; the lock releases with the process. Locust clients now send Accept-Encoding so future runs measure compressed transfer, as browsers do.	2026-08-12 11:34:12 +07:00
grabowski	039af8caac	perf: cache /measurements/latest; stale-on-error fallback for cached endpoints ... The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120).	2026-08-12 11:05:02 +07:00
grabowski	d27ca8bf40	perf: 120s TTL cache with single-flight on /api/hii/*/latest ... These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one.	2026-08-12 10:58:19 +07:00
grabowski	0005f7dce1	feat: codified backtests, honest docs, belt-and-braces serving, perf fixes ... Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).	2026-08-12 10:46:00 +07:00
grabowski	d2d0e655aa	fix: All-time range fills the date pickers with the DB's first record date ... Uses first_timestamp from /api/stats (2018-08-01 fallback) as the start date and today as the end, instead of clearing the pickers.	2026-08-11 17:02:31 +07:00
grabowski	b89c7e1915	feat: flood-verdict banner, violet rain palette, per-station bands, chip contrast, Umami ... UI/UX review backlog round 2 plus analytics: - at-a-glance verdict banner (role=status, aria-live) above the tiles: ok/watch/danger from P.1 level vs the official 3.70 m stage and the 24 h forecast, with a not-an-official-warning disclaimer + ThaiWater link (also added to the forecast card) - rainfall dots switch to a violet sequential ramp so they can never be confused with flow-status colors; legend gains 'Other markers' rows (grey no-data gauges, + bank-height sensors) - history-chart flood bands now use each station's own /forecast thresholds (P.1 official fallback) instead of hardcoded 3.0/4.5 - chipStyle(): contrast-safe text on all code/risk/stage chips (dark ink on amber/ochre, darkened fills for white ink); risk/stage ramps drop the blue step for a monotonic green-ochre-amber-red escalation - plain-language tooltips on the flow and capacity tiles - Umami: client script tag plus opt-out server-side middleware posting api-request events (fire-and-forget, 3 s timeout, never blocks)	2026-08-11 16:56:53 +07:00
grabowski	5848621c77	fix: UI/UX review round 1 — mobile reachability, legend toggle, replay correctness ... From the multi-lens review (27 confirmed findings; this lands the quick high-impact set): - mobile: side-card cap 400px->70vh so the basin-stations expander is reachable; live-pill kept as a dot (title carries the label); replay button no longer wraps; legend returns behind a Legend toggle button (was display:none, which also removed the only rain toggle) - replay: auto-refresh skips while replaying (was clobbering state); Chiang Mai flow tile shows P.1 replay discharge instead of a basin-wide sum mislabelled as P.1 - history: placeholder text in the empty chart, dropdown flips to 'Custom range' when dates are hand-edited, controls hint when no station is selected - a11y/copy: aria-expanded/aria-controls on all three disclosure controls, sensor rows say 'of bank height' vs tile 'channel capacity', P.1 tile note deduped	2026-08-11 16:39:53 +07:00
grabowski	09c1c84153	feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta ... robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404).	2026-08-11 16:33:44 +07:00
grabowski	bd3353e2dc	feat: buildfor.life link in dashboard footer	2026-08-11 16:30:25 +07:00
grabowski	d0018d6529	feat: physically meaningful stat tiles + footer links ... 'Combined discharge' summed sequential mainstem gauges, counting the same water several times — replaced with Chiang Mai river flow (P.1 discharge + % of channel capacity). 'Strongest flow' becomes 'Most stressed gauge' (max discharge_percent basin-wide). Reporting-stations note shows the ThaiWater/HII station count. Footer links to the source repo and /docs API reference.	2026-08-11 16:25:41 +07:00
grabowski	df31de092b	fix: collapse additional-stations list by default; no scroll-jump on station click ... The 97-row ThaiWater/HII list was starving the RID flow list down to one visible row. The section header is now a click/keyboard toggle (collapsed by default, arrow indicator); searching auto-expands it so hidden stations stay findable, and the flow list keeps a 280px minimum when both are open. Selecting a station no longer scrollIntoViews the history card.	2026-08-11 16:15:34 +07:00
grabowski	e27d418a1a	feat: station search in side panel; DB stats cover HII tables ... A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'.	2026-08-11 16:06:39 +07:00
grabowski	ba4710b243	feat: 'Last N' labels on history ranges; dropdown mirrors into date pickers ... Quick-range options read 'Last 24 hours' … 'Last 90 days'; choosing one fills the from/to date inputs with today and today-N (All time clears them). Dropdown-driven loads still fetch the precise trailing-hours window; hand-edited dates take over via state.useDates.	2026-08-11 15:58:28 +07:00
grabowski	9516857d44	feat: date-range picker on the station history panel; drop PostgreSQL naming ... /measurements/history/{code} accepts optional start/end date params that override the hours window (end date inclusive). The dashboard history card gains from/to date inputs beside the quick-range dropdown — explicit dates win, changing the dropdown clears them. All user-facing 'PostgreSQL history' labels renamed to 'Station history'.	2026-08-11 15:52:40 +07:00
grabowski	d29d49eac7	feat: collapsible flood outlook (P.1 only by default) + dashboard fixes ... The flood-risk card now shows just the Chiang Mai P.1 outlook; the per-station forecast grid collapses behind a Show/Hide expander. Fixes: the rain-layer toggle was unclickable (.map-overlay pointer-events:none now re-enabled on the legend), long HII station codes (MOU302, FOP015) wrap inside their chips instead of clipping, and the sensor panel sorts by bank percentage with no-data stations last.	2026-08-11 15:48:20 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	33d8baa8dd	fix: composite (station_id, timestamp) PK on hii_* measurement tables ... TimescaleDB create_hypertable rejects tables whose primary key omits the partitioning column; the surrogate id PK served no purpose, so use the natural key directly.	2026-08-11 14:50:27 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00
grabowski	7befc82ff5	feat: database stats on the dashboard via GET /api/stats ... New SQLAdapter.get_database_stats() aggregates totals, station count, date range, and hourly-slot coverage in one query per dialect. The endpoint follows the existing 503-guard/to_thread/TTL-cache pattern; the dashboard gains a five-tile stats strip on the existing refresh cadence. Coverage denominator is hour-truncated so off-hour endpoints cannot push it past 100%; MySQL slot expression avoids % characters that would break under pyformat bind interpolation.	2026-08-10 23:27:36 +07:00
grabowski	5e62ea529d	feat: --fill-gaps all repairs missing data across the whole DB range ... Gap detection is now hour-granular: days with partial data (missing hourly slots) are re-fetched, not just days with no rows at all. The API's hour-24-is-next-midnight quirk is handled by re-fetching day D-1 when day D is missing its 00:00 slot. SQL adapters gain single-query range and recorded-hours lookups; non-SQL backends fall back to the old day-granular check.	2026-08-10 22:13:58 +07:00
grabowski	410faeddd5	fix: protect admin API endpoints; stop rejecting flood-peak measurements ... Security: POST/PUT/DELETE /stations, POST /scrape/trigger and GET /config now require an X-API-Key header matching ADMIN_API_KEY. Secure by default - with no key configured those endpoints return 503 instead of being open. Comparison via secrets.compare_digest. Dashboard and read endpoints stay public. Data: the validator rejected any measurement whose discharge_percent exceeded 200 - which silently deleted the Oct 2024 record-flood peaks (the river genuinely ran at 201-226% of channel capacity). The cap is now 500%, and an out-of-range percent nulls that auxiliary field instead of discarding the whole row (water level and discharge are the data that matter). Surfaced by the user's historical backfill log.	2026-08-10 18:47:08 +07:00
grabowski	27fa292e09	docs: September 2025 flood render - deployed config, 24 h warning, peak within 7 cm ... Hour-by-hour chart of the 25-28 Sep 2025 event as forecast by the exact deployed configuration (trained through 2024, event unseen): first alert 26 Sep 18:00, flooding began 27 Sep 18:00 (24 h lead), predicted peak 4.00 m vs actual 3.93 m. The near-miss 3.51 m crest on 26 Sep correctly never alerted.	2026-08-10 18:35:55 +07:00
grabowski	f0183faa62	feat: discharge-driven river animation speed, replay stat tiles, hourly doc render ... - river dash animation now flows at a speed continuously derived from each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via inline per-path animation-duration, so it updates live and per-frame during the replay (CSS speed classes removed - setStyle cannot change classes) - the replay drives the Combined discharge and Strongest flow stat tiles each frame (marked '2024 replay'), restored on finish - docs: hour-by-hour detection detail render (22-28 Sep 2024) showing the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the 24 h warning between them; embedded with commentary - Matrix alerts now link to https://water.buildfor.life/ (override via ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link	2026-08-10 18:29:01 +07:00
grabowski	6112c681a7	docs: render of actual vs predicted through the 2024 flood; adaptive replay speed ... docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead predicted peak issued at each hour by a model trained only on pre-flood data, with the warning-probability panel below (first alert 24 Sep 01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's headline-validation section with an honest reading, including the ~0.4 m peak under-prediction. Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when risk is elevated, 1 h (hour-by-hour) while the model is alerting or the river is near/above flood stage - so viewers can watch the detection sequence unfold.	2026-08-10 18:10:48 +07:00
grabowski	199b0e3483	feat: replay shows when the model would have detected and warned ... The 2024 snapshot now carries a model track: p_warning and predicted 24 h peak from HGB heads trained ONLY on pre-flood data (an honest backtest, cutoff 2024-08-31), plus its calibration sigma. During replay: - the map clock shows the system's state at each moment: 'model: no flood expected' -> 'elevated risk' -> amber 'MODEL ALERT - flooding within 24 h likely (predicted peak X m)' -> red 'FLOODING' once the river actually crosses stage 1. First alert fires 24 Sep 01:00, a full day before the water crossed the flood line on 25 Sep. - the flood-risk outlook stage chips re-render each frame from the historic model prediction (sigmoid over the stage levels), so the whole outlook panel time-travels with the map.	2026-08-10 18:07:57 +07:00
grabowski	d015420b66	feat: full-basin 2024 flood replay with clock and demo indicator ... The replay now drives the entire map from a 120 kB all-station snapshot (static/flood-2024-all.json: 649 hourly frames x 16 stations of level + discharge, forward-filled on an aligned grid): - station markers recolor/resize per frame from historic discharge - river segments restyle per frame (color/width from the same nearest-gauge grading as live mode) - flood zones flood/recede from P.1's historic level - a large clock overlay on the map shows replay date/time, P.1 level and total basin flow - the LIVE DATA pill switches to an amber '2024 REPLAY' (or 'SIMULATION' for the demo_level/demo_rise hooks) and back on finish - replay end or stop restores the live view via a full dashboard reload Replaces the P.1-only snapshot (flood-2024-p1.json removed).	2026-08-10 17:59:06 +07:00