126 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
grabowski	5848621c77	fix: UI/UX review round 1 — mobile reachability, legend toggle, replay correctness ... From the multi-lens review (27 confirmed findings; this lands the quick high-impact set): - mobile: side-card cap 400px->70vh so the basin-stations expander is reachable; live-pill kept as a dot (title carries the label); replay button no longer wraps; legend returns behind a Legend toggle button (was display:none, which also removed the only rain toggle) - replay: auto-refresh skips while replaying (was clobbering state); Chiang Mai flow tile shows P.1 replay discharge instead of a basin-wide sum mislabelled as P.1 - history: placeholder text in the empty chart, dropdown flips to 'Custom range' when dates are hand-edited, controls hint when no station is selected - a11y/copy: aria-expanded/aria-controls on all three disclosure controls, sensor rows say 'of bank height' vs tile 'channel capacity', P.1 tile note deduped	2026-08-11 16:39:53 +07:00
grabowski	09c1c84153	feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta ... robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404).	2026-08-11 16:33:44 +07:00
grabowski	bd3353e2dc	feat: buildfor.life link in dashboard footer	2026-08-11 16:30:25 +07:00
grabowski	d0018d6529	feat: physically meaningful stat tiles + footer links ... 'Combined discharge' summed sequential mainstem gauges, counting the same water several times — replaced with Chiang Mai river flow (P.1 discharge + % of channel capacity). 'Strongest flow' becomes 'Most stressed gauge' (max discharge_percent basin-wide). Reporting-stations note shows the ThaiWater/HII station count. Footer links to the source repo and /docs API reference.	2026-08-11 16:25:41 +07:00
grabowski	df31de092b	fix: collapse additional-stations list by default; no scroll-jump on station click ... The 97-row ThaiWater/HII list was starving the RID flow list down to one visible row. The section header is now a click/keyboard toggle (collapsed by default, arrow indicator); searching auto-expands it so hidden stations stay findable, and the flow list keeps a 280px minimum when both are open. Selecting a station no longer scrollIntoViews the history card.	2026-08-11 16:15:34 +07:00
grabowski	e27d418a1a	feat: station search in side panel; DB stats cover HII tables ... A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'.	2026-08-11 16:06:39 +07:00
grabowski	ba4710b243	feat: 'Last N' labels on history ranges; dropdown mirrors into date pickers ... Quick-range options read 'Last 24 hours' … 'Last 90 days'; choosing one fills the from/to date inputs with today and today-N (All time clears them). Dropdown-driven loads still fetch the precise trailing-hours window; hand-edited dates take over via state.useDates.	2026-08-11 15:58:28 +07:00
grabowski	9516857d44	feat: date-range picker on the station history panel; drop PostgreSQL naming ... /measurements/history/{code} accepts optional start/end date params that override the hours window (end date inclusive). The dashboard history card gains from/to date inputs beside the quick-range dropdown — explicit dates win, changing the dropdown clears them. All user-facing 'PostgreSQL history' labels renamed to 'Station history'.	2026-08-11 15:52:40 +07:00
grabowski	d29d49eac7	feat: collapsible flood outlook (P.1 only by default) + dashboard fixes ... The flood-risk card now shows just the Chiang Mai P.1 outlook; the per-station forecast grid collapses behind a Show/Hide expander. Fixes: the rain-layer toggle was unclickable (.map-overlay pointer-events:none now re-enabled on the legend), long HII station codes (MOU302, FOP015) wrap inside their chips instead of clipping, and the sensor panel sorts by bank percentage with no-data stations last.	2026-08-11 15:48:20 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	33d8baa8dd	fix: composite (station_id, timestamp) PK on hii_* measurement tables ... TimescaleDB create_hypertable rejects tables whose primary key omits the partitioning column; the surrogate id PK served no purpose, so use the natural key directly.	2026-08-11 14:50:27 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00
grabowski	7befc82ff5	feat: database stats on the dashboard via GET /api/stats ... New SQLAdapter.get_database_stats() aggregates totals, station count, date range, and hourly-slot coverage in one query per dialect. The endpoint follows the existing 503-guard/to_thread/TTL-cache pattern; the dashboard gains a five-tile stats strip on the existing refresh cadence. Coverage denominator is hour-truncated so off-hour endpoints cannot push it past 100%; MySQL slot expression avoids % characters that would break under pyformat bind interpolation.	2026-08-10 23:27:36 +07:00
grabowski	5e62ea529d	feat: --fill-gaps all repairs missing data across the whole DB range ... Gap detection is now hour-granular: days with partial data (missing hourly slots) are re-fetched, not just days with no rows at all. The API's hour-24-is-next-midnight quirk is handled by re-fetching day D-1 when day D is missing its 00:00 slot. SQL adapters gain single-query range and recorded-hours lookups; non-SQL backends fall back to the old day-granular check.	2026-08-10 22:13:58 +07:00
grabowski	410faeddd5	fix: protect admin API endpoints; stop rejecting flood-peak measurements ... Security: POST/PUT/DELETE /stations, POST /scrape/trigger and GET /config now require an X-API-Key header matching ADMIN_API_KEY. Secure by default - with no key configured those endpoints return 503 instead of being open. Comparison via secrets.compare_digest. Dashboard and read endpoints stay public. Data: the validator rejected any measurement whose discharge_percent exceeded 200 - which silently deleted the Oct 2024 record-flood peaks (the river genuinely ran at 201-226% of channel capacity). The cap is now 500%, and an out-of-range percent nulls that auxiliary field instead of discarding the whole row (water level and discharge are the data that matter). Surfaced by the user's historical backfill log.	2026-08-10 18:47:08 +07:00
grabowski	27fa292e09	docs: September 2025 flood render - deployed config, 24 h warning, peak within 7 cm ... Hour-by-hour chart of the 25-28 Sep 2025 event as forecast by the exact deployed configuration (trained through 2024, event unseen): first alert 26 Sep 18:00, flooding began 27 Sep 18:00 (24 h lead), predicted peak 4.00 m vs actual 3.93 m. The near-miss 3.51 m crest on 26 Sep correctly never alerted.	2026-08-10 18:35:55 +07:00
grabowski	f0183faa62	feat: discharge-driven river animation speed, replay stat tiles, hourly doc render ... - river dash animation now flows at a speed continuously derived from each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via inline per-path animation-duration, so it updates live and per-frame during the replay (CSS speed classes removed - setStyle cannot change classes) - the replay drives the Combined discharge and Strongest flow stat tiles each frame (marked '2024 replay'), restored on finish - docs: hour-by-hour detection detail render (22-28 Sep 2024) showing the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the 24 h warning between them; embedded with commentary - Matrix alerts now link to https://water.buildfor.life/ (override via ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link	2026-08-10 18:29:01 +07:00
grabowski	6112c681a7	docs: render of actual vs predicted through the 2024 flood; adaptive replay speed ... docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead predicted peak issued at each hour by a model trained only on pre-flood data, with the warning-probability panel below (first alert 24 Sep 01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's headline-validation section with an honest reading, including the ~0.4 m peak under-prediction. Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when risk is elevated, 1 h (hour-by-hour) while the model is alerting or the river is near/above flood stage - so viewers can watch the detection sequence unfold.	2026-08-10 18:10:48 +07:00
grabowski	199b0e3483	feat: replay shows when the model would have detected and warned ... The 2024 snapshot now carries a model track: p_warning and predicted 24 h peak from HGB heads trained ONLY on pre-flood data (an honest backtest, cutoff 2024-08-31), plus its calibration sigma. During replay: - the map clock shows the system's state at each moment: 'model: no flood expected' -> 'elevated risk' -> amber 'MODEL ALERT - flooding within 24 h likely (predicted peak X m)' -> red 'FLOODING' once the river actually crosses stage 1. First alert fires 24 Sep 01:00, a full day before the water crossed the flood line on 25 Sep. - the flood-risk outlook stage chips re-render each frame from the historic model prediction (sigmoid over the stage levels), so the whole outlook panel time-travels with the map.	2026-08-10 18:07:57 +07:00
grabowski	d015420b66	feat: full-basin 2024 flood replay with clock and demo indicator ... The replay now drives the entire map from a 120 kB all-station snapshot (static/flood-2024-all.json: 649 hourly frames x 16 stations of level + discharge, forward-filled on an aligned grid): - station markers recolor/resize per frame from historic discharge - river segments restyle per frame (color/width from the same nearest-gauge grading as live mode) - flood zones flood/recede from P.1's historic level - a large clock overlay on the map shows replay date/time, P.1 level and total basin flow - the LIVE DATA pill switches to an amber '2024 REPLAY' (or 'SIMULATION' for the demo_level/demo_rise hooks) and back on finish - replay end or stop restores the live view via a full dashboard reload Replaces the P.1-only snapshot (flood-2024-p1.json removed).	2026-08-10 17:59:06 +07:00
grabowski	0b885e572d	chore: gitignore .playwright-mcp browser artifacts	2026-08-10 17:45:21 +07:00
grabowski	0672ea5ffa	feat: Oct-2024 flood replay, simulation hooks, and static replay data ... - header button 'Replay Oct 2024 flood': animates the real Sep 15 - Oct 12 2024 P.1 hourly series (~45 s) through the flood-zone display; zones flood blue as the river climbs to the 5.30 m record and recede as it falls; click again to stop; restores live state when done - replay reads a pre-extracted 15 kB static snapshot (static/flood-2024-p1.json, 335 frames) instead of pulling the 4.6 MB all-time history on every click; falls back to the history API if the snapshot is missing - demo hooks for testing/presentations: ?demo_level=4.4 pins a simulated P.1 level, ?demo_rise=1 animates rising water to 5.30 m; both label the outlook line as SIMULATION with the real level alongside	2026-08-10 17:44:36 +07:00
grabowski	c62a03e778	feat: flooded zones render as water and auto-surface ... When P.1's current level reaches a zone's trigger level the zone fills water-blue (solid border, .62 opacity) instead of the amber risk ramp, and the zone layer adds itself to the map automatically - no toggle needed during an actual flood. A manual hide sets a session flag so auto-show does not fight the user.	2026-08-10 17:22:27 +07:00
grabowski	711629682d	feat: per-marker icon field for custom markers ... custom-markers.json entries accept an optional "icon" (any emoji, default pin); seeded homes/office icons for the existing markers.	2026-08-10 17:20:12 +07:00
grabowski	7e7d244efa	feat: add baan boe marker	2026-08-10 17:19:31 +07:00
grabowski	786b8514ea	feat: custom map markers with per-location flood-zone risk ... src/static/custom-markers.json holds user points of interest ({name, lat, lon, optional note}); the dashboard renders them as pin markers whose popups show which inundation zone the point sits in, its P.1 trigger level, and the live 24 h exceedance probability (ray-cast point-in-polygon against the zone geojson; smallest matching zone wins). Seeded with the owner's four properties.	2026-08-10 17:16:28 +07:00
grabowski	a17509f3ea	feat: replace digitized flood zones with owner-traced polygons ... The 7 Chiang Mai inundation zones are now hand-traced by the project owner against the real basemap (cnx_flood.geojson), replacing the scan-digitized approximation and its georeferencing error entirely. Features are ordered zone 7 -> 1 so the earlier-flooding (smaller) zones render on top of the wider extents in Leaflet.	2026-08-10 17:12:23 +07:00
grabowski	4f12360960	fix: shift flood zones 1.05 km south, anchored to Nawarat Bridge ... The P.1 marker (Nawarat Bridge, 18.7875) sat inside zone 2 near its top edge, but the bridge IS Chang Khlan's northern boundary - the layer was ~0.0095 deg too far north (the district-centroid correction in c4e0fb6 overshot). Zone 2's north edge now lands at the bridge.	2026-08-10 16:47:14 +07:00
grabowski	c4e0fb6bae	fix: re-georeference flood zones and make them clearly visible ... The previous affine fit slid along the mostly north-south river (an ill-conditioned direction) leaving the zones ~1 km south-west of their true position. New approach: extract the scanned river band by color, match it to the OSM Ping mainstem centerline by arc length (which pins the along-river position), then correct residual translation using known district locations cross-checked against the river residual (533 m -> ~300 m median; Chang Khlan zone centroid now within 200 m). Mountain-area artifacts are clipped away via the municipality hull. Zones were also nearly invisible at fillOpacity 0.16 - base opacity is now 0.38, rising with the live 24 h exceedance probability, and 0.72 with a solid red border once the river is at or above a zone's level.	2026-08-10 16:38:09 +07:00
grabowski	f05289e628	fix: CI matrix to Python 3.11 only until pandas is upgraded ... pandas 2.0.3 has no cp312 wheels, so the 3.12 matrix leg fails during dependency install. The deployment runs 3.11.	2026-08-10 16:15:32 +07:00
grabowski	3b919adc56	feat: vector flood-zone polygons replace the scanned-map overlay ... Digitize the 7 Chiang Mai inundation zones from the municipal map into geojson polygons: pixels classified by legend color, vectorized via marching squares, and georeferenced by a 6-parameter affine fitted least-squares to the OSM Ping centerline (109 m median residual after outlier-filtered refits - the scanned image overlay could never scale correctly and is removed). The dashboard renders the zones as a Leaflet geoJSON layer styled live from the forecast: fill opacity scales with each zone's 24 h exceedance probability, zones whose trigger level the river has already reached get a solid red border, and each polygon's popup shows its trigger level and live probability. Zone styles refresh with every forecast load.	2026-08-10 16:14:34 +07:00
grabowski	6c0bb024a8	chore: remove stray zero-byte files accidentally committed	2026-08-10 15:57:19 +07:00
grabowski	ecd34177bb	fix: backtest/review findings in the flood-ML package ... From the adversarial review and threshold backtest (swarm verification): - predict.py: when a bundle's trained thresholds differ from the current config (deploy before retrain), skip its stale classifier heads and derive p_warning/p_danger from the regression + sigma against the CURRENT thresholds - the dashboard can no longer show contradictory old-threshold classifier output next to new-threshold stages - features.py: decouple the low-coverage regression-label rescue from the warning threshold (now the station's own p97.5 level); the old coupling silently dropped 34% of P.5's regression training rows and cost +46% MAE when its threshold rose - features.py: P.82 danger 3.80 -> 3.75 (3.80 was above the station's 8-year maximum of 3.78, so danger could never train or fire) - data.py / predict.py: anchor models/cache paths to the repo root; the relative paths silently returned zero rows when run from another CWD - annotate P.4A thresholds as low-confidence (11 supporting readings) 47 tests pass. Retrain required for the label-rescue and P.82 changes to reach the classifier heads.	2026-08-10 15:57:01 +07:00
grabowski	9cac9c4d2a	style: apply black/isort across the repo; make CI mypy advisory ... The push-CI gates (black/isort/mypy) had never actually run before the branch-trigger fix, and the codebase predates them. Formatting is now black/isort clean repo-wide. mypy keeps running but non-blocking: 86 pre-existing errors are a separate cleanup, not a gate to hold hostage.	2026-08-10 15:57:00 +07:00
grabowski	300c0e0b6f	fix: trigger CI on master (repo default branch), drop unsupported Python matrix entries ... Workflows listened on 'main'/'develop' but the repo's default branch is master, so push events never started a job (every historical run is a schedule event). Point push/PR triggers and the deploy-gate refs at master, and trim the test matrix to 3.11/3.12 to match requires-python >=3.11.	2026-08-10 15:46:24 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	29f4b5818d	fix: require Python >=3.11 and lock scikit-learn deps for uv ... scikit-learn 1.9.0 supports only Python >=3.11, so uv could not resolve the >=3.9 range. The deployment runs 3.11. Also pins numpy back to 1.26.4 in the lockfile (pandas 2.0.3 ABI).	2026-08-10 14:44:54 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00
grabowski	49a3de0087	fix: restore station telemetry and make river flow visualization data-driven ... Station selection showed no history since 21ca844: Chart.js v4 datasets had parsing:false with plain number arrays, drawing empty axes. Remove the flag so the chart parses values again. Backend hardening for the same flow: - /measurements/history/{code} no longer 503s on non-Postgres configs; it falls back to the configured adapter (reversed to ascending order) - DB_TYPE defaults to postgresql when POSTGRES_CONNECTION_STRING is set and DB_TYPE is unset, so the .env psql wins over the sqlite default - zero readings (0.0) are no longer coerced to None, which would fail MeasurementResponse validation and 500 /measurements/latest Map visualization: - river segments are now colored, widened and dash-speed-animated by the discharge at the nearest gauge (same scale as the marker legend) - fix z-order bug that drew the animated flow line behind its casing - legend entries for river lines, reduced-motion fallback River geometry: rebuild ping-river-network.geojson from Overpass (110 -> 202 features), restoring missing Ping mainstem reaches through the Bhumibol reservoir and the Tak-Kamphaeng Phet braided section (unnamed waterway=river ways in OSM), with short synthetic connectors (connector: true) bridging remaining sub-8 km holes.	2026-08-10 10:30:05 +07:00
grabowski	af1909db73	fix: restore history chart initialization after flood bands change	2026-08-10 09:46:34 +07:00
grabowski	76c934e475	feat: add _calculate_discharge to estimate from water level when DB discharge is NULL	2026-08-09 18:09:27 +07:00
grabowski	32e455783a	fix: harden loadHistory against race conditions and canvas reuse	2026-08-09 18:03:41 +07:00
grabowski	b3ea340bbd	fix: fix chart reuse error and harden flood-bands plugin	2026-08-09 17:56:30 +07:00
grabowski	ba0348b580	feat: add flood-zone bands (normal/warning/danger) to history chart	2026-08-09 17:54:11 +07:00
grabowski	21ca84444d	perf: downsample history to daily averages and fix chart overflow	2026-08-09 17:48:31 +07:00
grabowski	9f26b32c86	fix: bump history limits to 100k so all-time shows full dataset	2026-08-09 17:41:14 +07:00
grabowski	bbbf548d66	fix: align web API limit cap with 10k and clean up postgres_history imports	2026-08-09 17:36:58 +07:00
grabowski	ef106fce8d	fix: include year in history chart labels	2026-08-09 17:21:20 +07:00
grabowski	33f2dd45f4	fix: render history chart timestamps in ICT (Asia/Bangkok)	2026-08-09 17:19:28 +07:00