100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
grabowski	cbb3bf7369	feat: Open-Meteo catchment rainfall series + rain features + harness variant ... src/ml/rain.py fetches hourly precipitation for five upper-Ping catchment points (Chiang Dao, Mae Taeng, Mae Ngat, Mae Rim, city) from the Open-Meteo forecast-model archive (2021-03 onward, no API key, Bangkok-local timestamps, year-chunked local cache) plus the live forecast endpoint for serving (trailing days + next 48h). features.build_features/build_matrix accept the catchment-mean series and add rain_6h/24h/72h trailing sums and rain_fc24 — the forward 24h sum, a genuine forecast feature (archived forecasts at training time, a real weather forecast at serving; never contains river data). Columns exist only when a series is provided: HGB rejects all-NaN columns at fit, so no-rain training omits them and serving passes an empty series for alignment. evaluate.py gains the rise_rain variant (and --no-rain) so the harness can judge whether rain beats the deployed rise baseline.	2026-08-12 16:39:36 +07:00
grabowski	21e9d2e114	feat: hgb-v2 — regression heads predict rise, recovering flood warning lead ... Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103; results in models/eval_variants.json) showed the absolute-level target alerting AT the crossing on essentially every event, while the rise target (future max - current level, level added back at serving) gives +6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than weighted/quantile variants, and ~11% better MAE. Weighted and quantile variants rejected: more false alarms, no Brier-score calibration gain. Ported to production: train.py fits rise in both eval and refit passes (sigma/metrics computed in absolute space), bundles stamped hgb-v2 with regression_target='rise', predict.py adds the level back for v2 and stays compatible with v1 bundles, backtest_render.py mirrors the same math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and the record-peak underprediction is gone (rise models can exceed the training max). The >=12h acceptance gate still fails honestly at +6h — closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at 6/12/24h; docs updated throughout.	2026-08-12 15:43:29 +07:00
grabowski	a0086086a2	feat: rolling-origin event-aware evaluation harness for model variants ... One fold per monsoon season (train <= 30 Apr, test Jun-Nov, 2021-2025) replaces the single fixed holdout that contained only ~4 warning events. Metrics are what matters operationally: sustained first-alert lead vs each observed 3.70m crossing (two consecutive alerting samples required; lookback floored at the previous event's end so multi-peak floods can't launder lead credit), peak error from the prediction actually issued 24h before the peak (3h match tolerance, null on outages), false-alarm episodes (12h gap tolerance), MAE / flood-regime MAE, and a Brier score on warning exceedance — included because sigma cancels algebraically in any p>=0.5 alert metric, so lead times compare predictors while Brier compares uncertainty models. Variants: baseline_abs (current), rise (target = future max - current level), rise_weighted (flood-regime sample weights 1x->5x), and rise_quantile (q50/q90 heads, spread-implied sigma). Harness verified by a 3-agent adversarial review (features bit-identical across fold cutoffs; three metric flaws found and fixed before first use). Also: features.build_labels/build_matrix gain stats_end so the rescue quantile is computed from pre-cutoff data only, closing the label- construction leak flagged in the earlier ML review.	2026-08-12 15:19:26 +07:00
grabowski	98023243af	feat: forecast precompute + issued-forecast archive + dashboard overlay ... The collection-leader worker now precomputes forecasts after every scrape cycle: primes the /forecast cache (users never trigger the multi-second inference — its TTL rises to 4500s so the hourly refresh always wins) and persists every issued forecast to a new forecast_history table keyed by (as_of, station, horizon) with predicted max level, warn/danger probabilities, current level, and model_version. This is the operational record the backtests lacked — predicted-vs-actual becomes a simple join instead of retraining historical models. GET /api/forecast/history/{station} serves the archive (hours or start/end + horizon filters, 5-min edge cache), and the station history chart overlays 'Model 24 h peak (as issued)' as a dashed violet line once data accumulates.	2026-08-12 14:13:39 +07:00
grabowski	731f10910e	perf: stale-while-revalidate caching, bigger executor, cheap DB health probe ... Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine).	2026-08-12 12:32:38 +07:00
grabowski	96fedb3991	perf: single-flight /api/stats; fast-fail health probe ... The on-box rerun after the inline fast path showed the cached endpoints healthy (latest p50 72-160ms, HII ~100ms) but /api/stats at 81% timeouts and /health at 33%: stats had a cache but NO single-flight, so every concurrent miss ran the heavy whole-DB counts (~1.7M rows) in parallel, re-jamming Postgres and the executor — which also dragged uncached history windows into 60s timeouts. /api/stats now computes through _ttl_cached_stale (one computation per 5min TTL, stale served on failure). The health API probe fails fast (5s instead of 30s) and its cache TTL rises to 30s, so a slow upstream can no longer pin executor threads longer than the cache lifetime.	2026-08-12 11:55:49 +07:00
grabowski	0b14e394ad	perf: Cache-Control headers so browsers and edge caches absorb traffic ... Bandwidth diagnosis: iperf shows the proxy<->API VPN link at 65-78 Mbit/s and a fast client pulls 15.7 MB/s from a CDN but only ~66 KB/s from the site — the Caddy host's public uplink is the scarce resource. Max-ages mirror the server cache TTLs (latest 30s, HII 60s, forecast 120s, history/stats/stations 300s, static 1h, dashboard HTML 120s), so repeat views come from browser cache and an edge proxy (e.g. Cloudflare free tier in front of Caddy) can serve most traffic without touching the uplink.	2026-08-12 11:51:43 +07:00
grabowski	d9c65bcf0c	perf: inline cache fast path; cache /health checks ... The on-box load test exposed the real stall: cache HITS were dispatched through asyncio.to_thread, so under load a microsecond lookup queued ~11s behind slow work in the ~8-thread default executor (endpoints answered inline — /forecast 4ms, /api/stats 2ms — while every to_thread endpoint sat at p50 8-17s). Handlers now check TTL caches inline in the async path via _cache_fresh() and only pay for a thread on a miss. /health results are cached for HEALTH_CACHE_TTL_SECONDS (10): its external RID-API probe plus DB query were occupying executor threads on every hit, which is what jammed the pool in the first place.	2026-08-12 11:48:05 +07:00
grabowski	1ec5cfb4df	perf: gzip responses; multi-worker serving with single collection leader ... GZipMiddleware (min 500 bytes) compresses the dashboard HTML ~4x and station JSON up to ~100x, end-to-end through the Caddy TLS terminator — production load testing showed the deployment is bandwidth-bound once the response caches hit, so compression is the capacity lever. WEB_WORKERS (default 2) runs uvicorn multi-process via the app import string. Every worker executes the lifespan, so a localhost lock port (COLLECTION_LEADER_PORT, default 8901) elects exactly one background-collection leader per machine — RID/HII polling stays once-per-cycle instead of once-per-worker; the lock releases with the process. Locust clients now send Accept-Encoding so future runs measure compressed transfer, as browsers do.	2026-08-12 11:34:12 +07:00
grabowski	039af8caac	perf: cache /measurements/latest; stale-on-error fallback for cached endpoints ... The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120).	2026-08-12 11:05:02 +07:00
grabowski	d27ca8bf40	perf: 120s TTL cache with single-flight on /api/hii/*/latest ... These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one.	2026-08-12 10:58:19 +07:00
grabowski	0005f7dce1	feat: codified backtests, honest docs, belt-and-braces serving, perf fixes ... Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load).	2026-08-12 10:46:00 +07:00
grabowski	d2d0e655aa	fix: All-time range fills the date pickers with the DB's first record date ... Uses first_timestamp from /api/stats (2018-08-01 fallback) as the start date and today as the end, instead of clearing the pickers.	2026-08-11 17:02:31 +07:00
grabowski	b89c7e1915	feat: flood-verdict banner, violet rain palette, per-station bands, chip contrast, Umami ... UI/UX review backlog round 2 plus analytics: - at-a-glance verdict banner (role=status, aria-live) above the tiles: ok/watch/danger from P.1 level vs the official 3.70 m stage and the 24 h forecast, with a not-an-official-warning disclaimer + ThaiWater link (also added to the forecast card) - rainfall dots switch to a violet sequential ramp so they can never be confused with flow-status colors; legend gains 'Other markers' rows (grey no-data gauges, + bank-height sensors) - history-chart flood bands now use each station's own /forecast thresholds (P.1 official fallback) instead of hardcoded 3.0/4.5 - chipStyle(): contrast-safe text on all code/risk/stage chips (dark ink on amber/ochre, darkened fills for white ink); risk/stage ramps drop the blue step for a monotonic green-ochre-amber-red escalation - plain-language tooltips on the flow and capacity tiles - Umami: client script tag plus opt-out server-side middleware posting api-request events (fire-and-forget, 3 s timeout, never blocks)	2026-08-11 16:56:53 +07:00
grabowski	5848621c77	fix: UI/UX review round 1 — mobile reachability, legend toggle, replay correctness ... From the multi-lens review (27 confirmed findings; this lands the quick high-impact set): - mobile: side-card cap 400px->70vh so the basin-stations expander is reachable; live-pill kept as a dot (title carries the label); replay button no longer wraps; legend returns behind a Legend toggle button (was display:none, which also removed the only rain toggle) - replay: auto-refresh skips while replaying (was clobbering state); Chiang Mai flow tile shows P.1 replay discharge instead of a basin-wide sum mislabelled as P.1 - history: placeholder text in the empty chart, dropdown flips to 'Custom range' when dates are hand-edited, controls hint when no station is selected - a11y/copy: aria-expanded/aria-controls on all three disclosure controls, sensor rows say 'of bank height' vs tile 'channel capacity', P.1 tile note deduped	2026-08-11 16:39:53 +07:00
grabowski	09c1c84153	feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta ... robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404).	2026-08-11 16:33:44 +07:00
grabowski	bd3353e2dc	feat: buildfor.life link in dashboard footer	2026-08-11 16:30:25 +07:00
grabowski	d0018d6529	feat: physically meaningful stat tiles + footer links ... 'Combined discharge' summed sequential mainstem gauges, counting the same water several times — replaced with Chiang Mai river flow (P.1 discharge + % of channel capacity). 'Strongest flow' becomes 'Most stressed gauge' (max discharge_percent basin-wide). Reporting-stations note shows the ThaiWater/HII station count. Footer links to the source repo and /docs API reference.	2026-08-11 16:25:41 +07:00
grabowski	df31de092b	fix: collapse additional-stations list by default; no scroll-jump on station click ... The 97-row ThaiWater/HII list was starving the RID flow list down to one visible row. The section header is now a click/keyboard toggle (collapsed by default, arrow indicator); searching auto-expands it so hidden stations stay findable, and the flow list keeps a 280px minimum when both are open. Selecting a station no longer scrollIntoViews the history card.	2026-08-11 16:15:34 +07:00
grabowski	e27d418a1a	feat: station search in side panel; DB stats cover HII tables ... A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'.	2026-08-11 16:06:39 +07:00
grabowski	ba4710b243	feat: 'Last N' labels on history ranges; dropdown mirrors into date pickers ... Quick-range options read 'Last 24 hours' … 'Last 90 days'; choosing one fills the from/to date inputs with today and today-N (All time clears them). Dropdown-driven loads still fetch the precise trailing-hours window; hand-edited dates take over via state.useDates.	2026-08-11 15:58:28 +07:00
grabowski	9516857d44	feat: date-range picker on the station history panel; drop PostgreSQL naming ... /measurements/history/{code} accepts optional start/end date params that override the hours window (end date inclusive). The dashboard history card gains from/to date inputs beside the quick-range dropdown — explicit dates win, changing the dropdown clears them. All user-facing 'PostgreSQL history' labels renamed to 'Station history'.	2026-08-11 15:52:40 +07:00
grabowski	d29d49eac7	feat: collapsible flood outlook (P.1 only by default) + dashboard fixes ... The flood-risk card now shows just the Chiang Mai P.1 outlook; the per-station forecast grid collapses behind a Show/Hide expander. Fixes: the rain-layer toggle was unclickable (.map-overlay pointer-events:none now re-enabled on the legend), long HII station codes (MOU302, FOP015) wrap inside their chips instead of clipping, and the sensor panel sorts by bank percentage with no-data stations last.	2026-08-11 15:48:20 +07:00
grabowski	4f3f19f6db	feat: rainfall + HII water-level layers on the dashboard ... New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough.	2026-08-11 15:26:58 +07:00
grabowski	d72496f404	feat: backfill hii_waterlevel from the HII waterlevel_graph archive ... scripts/backfill_hii_waterlevel.py walks the api-v3 waterlevel_graph endpoint (hourly wl_msl + discharge, archive back to ~2019) in full-year windows per station and upserts into hii_waterlevel. Defaults to the RID-mirror and key stations; --stations/--all/--start/--end/--chunk-days override. History upserts touch only wl_msl and discharge so colliding live-snapshot rows keep storage_percent/situation_level. Idempotent and safe to re-run.	2026-08-11 15:11:08 +07:00
grabowski	33d8baa8dd	fix: composite (station_id, timestamp) PK on hii_* measurement tables ... TimescaleDB create_hypertable rejects tables whose primary key omits the partitioning column; the surrogate id PK served no purpose, so use the natural key directly.	2026-08-11 14:50:27 +07:00
grabowski	1845ef7203	feat: hourly HII/ThaiWater rainfall + backup water-level collection ... Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints.	2026-08-11 14:44:53 +07:00
grabowski	7befc82ff5	feat: database stats on the dashboard via GET /api/stats ... New SQLAdapter.get_database_stats() aggregates totals, station count, date range, and hourly-slot coverage in one query per dialect. The endpoint follows the existing 503-guard/to_thread/TTL-cache pattern; the dashboard gains a five-tile stats strip on the existing refresh cadence. Coverage denominator is hour-truncated so off-hour endpoints cannot push it past 100%; MySQL slot expression avoids % characters that would break under pyformat bind interpolation.	2026-08-10 23:27:36 +07:00
grabowski	5e62ea529d	feat: --fill-gaps all repairs missing data across the whole DB range ... Gap detection is now hour-granular: days with partial data (missing hourly slots) are re-fetched, not just days with no rows at all. The API's hour-24-is-next-midnight quirk is handled by re-fetching day D-1 when day D is missing its 00:00 slot. SQL adapters gain single-query range and recorded-hours lookups; non-SQL backends fall back to the old day-granular check.	2026-08-10 22:13:58 +07:00
grabowski	410faeddd5	fix: protect admin API endpoints; stop rejecting flood-peak measurements ... Security: POST/PUT/DELETE /stations, POST /scrape/trigger and GET /config now require an X-API-Key header matching ADMIN_API_KEY. Secure by default - with no key configured those endpoints return 503 instead of being open. Comparison via secrets.compare_digest. Dashboard and read endpoints stay public. Data: the validator rejected any measurement whose discharge_percent exceeded 200 - which silently deleted the Oct 2024 record-flood peaks (the river genuinely ran at 201-226% of channel capacity). The cap is now 500%, and an out-of-range percent nulls that auxiliary field instead of discarding the whole row (water level and discharge are the data that matter). Surfaced by the user's historical backfill log.	2026-08-10 18:47:08 +07:00
grabowski	27fa292e09	docs: September 2025 flood render - deployed config, 24 h warning, peak within 7 cm ... Hour-by-hour chart of the 25-28 Sep 2025 event as forecast by the exact deployed configuration (trained through 2024, event unseen): first alert 26 Sep 18:00, flooding began 27 Sep 18:00 (24 h lead), predicted peak 4.00 m vs actual 3.93 m. The near-miss 3.51 m crest on 26 Sep correctly never alerted.	2026-08-10 18:35:55 +07:00
grabowski	f0183faa62	feat: discharge-driven river animation speed, replay stat tiles, hourly doc render ... - river dash animation now flows at a speed continuously derived from each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via inline per-path animation-duration, so it updates live and per-frame during the replay (CSS speed classes removed - setStyle cannot change classes) - the replay drives the Combined discharge and Strongest flow stat tiles each frame (marked '2024 replay'), restored on finish - docs: hour-by-hour detection detail render (22-28 Sep 2024) showing the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the 24 h warning between them; embedded with commentary - Matrix alerts now link to https://water.buildfor.life/ (override via ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link	2026-08-10 18:29:01 +07:00
grabowski	6112c681a7	docs: render of actual vs predicted through the 2024 flood; adaptive replay speed ... docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead predicted peak issued at each hour by a model trained only on pre-flood data, with the warning-probability panel below (first alert 24 Sep 01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's headline-validation section with an honest reading, including the ~0.4 m peak under-prediction. Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when risk is elevated, 1 h (hour-by-hour) while the model is alerting or the river is near/above flood stage - so viewers can watch the detection sequence unfold.	2026-08-10 18:10:48 +07:00
grabowski	199b0e3483	feat: replay shows when the model would have detected and warned ... The 2024 snapshot now carries a model track: p_warning and predicted 24 h peak from HGB heads trained ONLY on pre-flood data (an honest backtest, cutoff 2024-08-31), plus its calibration sigma. During replay: - the map clock shows the system's state at each moment: 'model: no flood expected' -> 'elevated risk' -> amber 'MODEL ALERT - flooding within 24 h likely (predicted peak X m)' -> red 'FLOODING' once the river actually crosses stage 1. First alert fires 24 Sep 01:00, a full day before the water crossed the flood line on 25 Sep. - the flood-risk outlook stage chips re-render each frame from the historic model prediction (sigmoid over the stage levels), so the whole outlook panel time-travels with the map.	2026-08-10 18:07:57 +07:00
grabowski	d015420b66	feat: full-basin 2024 flood replay with clock and demo indicator ... The replay now drives the entire map from a 120 kB all-station snapshot (static/flood-2024-all.json: 649 hourly frames x 16 stations of level + discharge, forward-filled on an aligned grid): - station markers recolor/resize per frame from historic discharge - river segments restyle per frame (color/width from the same nearest-gauge grading as live mode) - flood zones flood/recede from P.1's historic level - a large clock overlay on the map shows replay date/time, P.1 level and total basin flow - the LIVE DATA pill switches to an amber '2024 REPLAY' (or 'SIMULATION' for the demo_level/demo_rise hooks) and back on finish - replay end or stop restores the live view via a full dashboard reload Replaces the P.1-only snapshot (flood-2024-p1.json removed).	2026-08-10 17:59:06 +07:00
grabowski	0b885e572d	chore: gitignore .playwright-mcp browser artifacts	2026-08-10 17:45:21 +07:00
grabowski	0672ea5ffa	feat: Oct-2024 flood replay, simulation hooks, and static replay data ... - header button 'Replay Oct 2024 flood': animates the real Sep 15 - Oct 12 2024 P.1 hourly series (~45 s) through the flood-zone display; zones flood blue as the river climbs to the 5.30 m record and recede as it falls; click again to stop; restores live state when done - replay reads a pre-extracted 15 kB static snapshot (static/flood-2024-p1.json, 335 frames) instead of pulling the 4.6 MB all-time history on every click; falls back to the history API if the snapshot is missing - demo hooks for testing/presentations: ?demo_level=4.4 pins a simulated P.1 level, ?demo_rise=1 animates rising water to 5.30 m; both label the outlook line as SIMULATION with the real level alongside	2026-08-10 17:44:36 +07:00
grabowski	c62a03e778	feat: flooded zones render as water and auto-surface ... When P.1's current level reaches a zone's trigger level the zone fills water-blue (solid border, .62 opacity) instead of the amber risk ramp, and the zone layer adds itself to the map automatically - no toggle needed during an actual flood. A manual hide sets a session flag so auto-show does not fight the user.	2026-08-10 17:22:27 +07:00
grabowski	711629682d	feat: per-marker icon field for custom markers ... custom-markers.json entries accept an optional "icon" (any emoji, default pin); seeded homes/office icons for the existing markers.	2026-08-10 17:20:12 +07:00
grabowski	7e7d244efa	feat: add baan boe marker	2026-08-10 17:19:31 +07:00
grabowski	786b8514ea	feat: custom map markers with per-location flood-zone risk ... src/static/custom-markers.json holds user points of interest ({name, lat, lon, optional note}); the dashboard renders them as pin markers whose popups show which inundation zone the point sits in, its P.1 trigger level, and the live 24 h exceedance probability (ray-cast point-in-polygon against the zone geojson; smallest matching zone wins). Seeded with the owner's four properties.	2026-08-10 17:16:28 +07:00
grabowski	a17509f3ea	feat: replace digitized flood zones with owner-traced polygons ... The 7 Chiang Mai inundation zones are now hand-traced by the project owner against the real basemap (cnx_flood.geojson), replacing the scan-digitized approximation and its georeferencing error entirely. Features are ordered zone 7 -> 1 so the earlier-flooding (smaller) zones render on top of the wider extents in Leaflet.	2026-08-10 17:12:23 +07:00
grabowski	4f12360960	fix: shift flood zones 1.05 km south, anchored to Nawarat Bridge ... The P.1 marker (Nawarat Bridge, 18.7875) sat inside zone 2 near its top edge, but the bridge IS Chang Khlan's northern boundary - the layer was ~0.0095 deg too far north (the district-centroid correction in c4e0fb6 overshot). Zone 2's north edge now lands at the bridge.	2026-08-10 16:47:14 +07:00
grabowski	c4e0fb6bae	fix: re-georeference flood zones and make them clearly visible ... The previous affine fit slid along the mostly north-south river (an ill-conditioned direction) leaving the zones ~1 km south-west of their true position. New approach: extract the scanned river band by color, match it to the OSM Ping mainstem centerline by arc length (which pins the along-river position), then correct residual translation using known district locations cross-checked against the river residual (533 m -> ~300 m median; Chang Khlan zone centroid now within 200 m). Mountain-area artifacts are clipped away via the municipality hull. Zones were also nearly invisible at fillOpacity 0.16 - base opacity is now 0.38, rising with the live 24 h exceedance probability, and 0.72 with a solid red border once the river is at or above a zone's level.	2026-08-10 16:38:09 +07:00
grabowski	f05289e628	fix: CI matrix to Python 3.11 only until pandas is upgraded ... pandas 2.0.3 has no cp312 wheels, so the 3.12 matrix leg fails during dependency install. The deployment runs 3.11.	2026-08-10 16:15:32 +07:00
grabowski	3b919adc56	feat: vector flood-zone polygons replace the scanned-map overlay ... Digitize the 7 Chiang Mai inundation zones from the municipal map into geojson polygons: pixels classified by legend color, vectorized via marching squares, and georeferenced by a 6-parameter affine fitted least-squares to the OSM Ping centerline (109 m median residual after outlier-filtered refits - the scanned image overlay could never scale correctly and is removed). The dashboard renders the zones as a Leaflet geoJSON layer styled live from the forecast: fill opacity scales with each zone's 24 h exceedance probability, zones whose trigger level the river has already reached get a solid red border, and each polygon's popup shows its trigger level and live probability. Zone styles refresh with every forecast load.	2026-08-10 16:14:34 +07:00
grabowski	6c0bb024a8	chore: remove stray zero-byte files accidentally committed	2026-08-10 15:57:19 +07:00
grabowski	ecd34177bb	fix: backtest/review findings in the flood-ML package ... From the adversarial review and threshold backtest (swarm verification): - predict.py: when a bundle's trained thresholds differ from the current config (deploy before retrain), skip its stale classifier heads and derive p_warning/p_danger from the regression + sigma against the CURRENT thresholds - the dashboard can no longer show contradictory old-threshold classifier output next to new-threshold stages - features.py: decouple the low-coverage regression-label rescue from the warning threshold (now the station's own p97.5 level); the old coupling silently dropped 34% of P.5's regression training rows and cost +46% MAE when its threshold rose - features.py: P.82 danger 3.80 -> 3.75 (3.80 was above the station's 8-year maximum of 3.78, so danger could never train or fire) - data.py / predict.py: anchor models/cache paths to the repo root; the relative paths silently returned zero rows when run from another CWD - annotate P.4A thresholds as low-confidence (11 supporting readings) 47 tests pass. Retrain required for the label-rescue and P.82 changes to reach the classifier heads.	2026-08-10 15:57:01 +07:00
grabowski	9cac9c4d2a	style: apply black/isort across the repo; make CI mypy advisory ... The push-CI gates (black/isort/mypy) had never actually run before the branch-trigger fix, and the codebase predates them. Formatting is now black/isort clean repo-wide. mypy keeps running but non-blocking: 86 pre-existing errors are a separate cleanup, not a gate to hold hostage.	2026-08-10 15:57:00 +07:00
grabowski	300c0e0b6f	fix: trigger CI on master (repo default branch), drop unsupported Python matrix entries ... Workflows listened on 'main'/'develop' but the repo's default branch is master, so push events never started a job (every historical run is a schedule event). Point push/PR triggers and the deploy-gate refs at master, and trim the test matrix to 3.11/3.12 to match requires-python >=3.11.	2026-08-10 15:46:24 +07:00
grabowski	e4d5d274f0	feat: per-station flood thresholds and Chiang Mai inundation stages for P.1 ... Replace the network-wide (3.0, 4.5) m thresholds with per-station values calibrated from the DB's discharge_percent (RID % of channel capacity): warning = median level at 75-85% capacity, danger = median at 95-105%. Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67 under-alerting (overflow ~2.9 m). Requires a retrain to take effect in the classifier heads. P.1 uses the official Chiang Mai municipal inundation map instead: warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5), with the full 7-stage table (3.70-4.60 m + discharge) in features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage exceedance probabilities computed from the regression head + calibration sigma - available immediately without retraining. Dashboard: "Chiang Mai city flood outlook" block above the forecast grid (predicted peak + 7 stage-probability chips) and a toggleable georeferenced overlay of the official flood-zone map (static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).	2026-08-10 15:35:00 +07:00
grabowski	29f4b5818d	fix: require Python >=3.11 and lock scikit-learn deps for uv ... scikit-learn 1.9.0 supports only Python >=3.11, so uv could not resolve the >=3.9 range. The deployment runs 3.11. Also pins numpy back to 1.26.4 in the lockfile (pandas 2.0.3 ABI).	2026-08-10 14:44:54 +07:00
grabowski	4358d52d55	feat: ML flood-event forecasting from 8 years of gauge history ... Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py).	2026-08-10 12:49:47 +07:00
grabowski	49a3de0087	fix: restore station telemetry and make river flow visualization data-driven ... Station selection showed no history since 21ca844: Chart.js v4 datasets had parsing:false with plain number arrays, drawing empty axes. Remove the flag so the chart parses values again. Backend hardening for the same flow: - /measurements/history/{code} no longer 503s on non-Postgres configs; it falls back to the configured adapter (reversed to ascending order) - DB_TYPE defaults to postgresql when POSTGRES_CONNECTION_STRING is set and DB_TYPE is unset, so the .env psql wins over the sqlite default - zero readings (0.0) are no longer coerced to None, which would fail MeasurementResponse validation and 500 /measurements/latest Map visualization: - river segments are now colored, widened and dash-speed-animated by the discharge at the nearest gauge (same scale as the marker legend) - fix z-order bug that drew the animated flow line behind its casing - legend entries for river lines, reduced-motion fallback River geometry: rebuild ping-river-network.geojson from Overpass (110 -> 202 features), restoring missing Ping mainstem reaches through the Bhumibol reservoir and the Tak-Kamphaeng Phet braided section (unnamed waterway=river ways in OSM), with short synthetic connectors (connector: true) bridging remaining sub-8 km holes.	2026-08-10 10:30:05 +07:00
grabowski	af1909db73	fix: restore history chart initialization after flood bands change	2026-08-10 09:46:34 +07:00
grabowski	76c934e475	feat: add _calculate_discharge to estimate from water level when DB discharge is NULL	2026-08-09 18:09:27 +07:00
grabowski	32e455783a	fix: harden loadHistory against race conditions and canvas reuse	2026-08-09 18:03:41 +07:00
grabowski	b3ea340bbd	fix: fix chart reuse error and harden flood-bands plugin	2026-08-09 17:56:30 +07:00
grabowski	ba0348b580	feat: add flood-zone bands (normal/warning/danger) to history chart	2026-08-09 17:54:11 +07:00
grabowski	21ca84444d	perf: downsample history to daily averages and fix chart overflow	2026-08-09 17:48:31 +07:00
grabowski	9f26b32c86	fix: bump history limits to 100k so all-time shows full dataset	2026-08-09 17:41:14 +07:00
grabowski	bbbf548d66	fix: align web API limit cap with 10k and clean up postgres_history imports	2026-08-09 17:36:58 +07:00
grabowski	ef106fce8d	fix: include year in history chart labels	2026-08-09 17:21:20 +07:00
grabowski	33f2dd45f4	fix: render history chart timestamps in ICT (Asia/Bangkok)	2026-08-09 17:19:28 +07:00
grabowski	2fe1dcf4da	feat: add 5-minute TTL cache for PostgreSQL history queries	2026-08-09 17:15:47 +07:00
grabowski	c00a26402a	fix: update test limit boundary to match new 10k ceiling	2026-08-09 17:13:51 +07:00
grabowski	abfac1d3bb	fix: correct all-time hours value and update backend limit	2026-08-09 17:12:22 +07:00
grabowski	6f2a8a0d8f	feat: add all-time history option and increase limit to 10k	2026-08-09 17:11:26 +07:00
grabowski	9ef7798e00	feat: load history when clicking station markers on map	2026-08-09 17:04:21 +07:00
grabowski	ae5d0a13d7	[verified] feat: add live river dashboard ... Add mapped river and ThaiWater sensor layers, PostgreSQL history charts, API endpoints, and dashboard tests.	2026-08-09 16:59:25 +07:00
grabowski	e5936d5717	Move dashboard HTML out of web_api into a static file ... Extract the inline root() dashboard markup into src/static/dashboard.html, loaded once at import. Keeps HTML out of the Python module (web_api 616 -> 576 lines) with a defensive fallback if the file is missing. Full <500 compliance for web_api still needs the endpoints split into APIRouter modules; tracked as remaining #4 work.	2026-07-22 14:28:34 +07:00
grabowski	08dc536c93	Add unit tests for RID API response parsing ... Cover the previously-untested, highest-risk parsing in fetch_water_data_for_date by mocking the HTTP call: - hour 1-23 map to the same day; hour 24 rolls to next-day midnight - qvalues "***" / None yield discharge None (no crash) - None water level is skipped - out-of-range (0, 25) and empty hourlytime rows are skipped - missing "rows" key returns an empty list This gives the scraper a safety net before its module is split.	2026-07-22 14:24:28 +07:00
grabowski	ad7f7b8c76	Extract API Pydantic models into schemas.py ... Move the seven request/response models out of web_api.py into a dedicated src/schemas.py (separation of concerns; first step of the file-size cleanup). web_api.py imports them back, so behaviour is unchanged. Note: web_api.py is still over the 500-line guideline; the remaining bulk is the inline HTML dashboard in root(), to be extracted in a follow-up.	2026-07-22 14:07:31 +07:00
grabowski	12b7f9f422	Add assert-based pytest coverage for recent fixes ... - tests/conftest.py: put repo root on sys.path so `import src...` resolves under pytest regardless of invocation directory. - test_matrix_formatting.py: lock in HTML formatted_body + plain-text fallback, URL linkification, HTML escaping, and send_alert field rendering. - test_station_persistence.py: cover default-load, save/reload round-trip (incl. Thai text), runtime-file precedence, and atomic-write cleanup. These are real assert-based tests (unlike the existing print-style scripts) so CI can gate on them. 13 tests, all passing.	2026-07-22 14:02:06 +07:00
grabowski	ce31a5254e	Harden install.sh per security review ... - .env now chmod 0600 and APP_DIR chmod 0750 after chown, so the Matrix token and DB credentials are not world-readable. - uv auto-install (curl | sh as root) is now opt-in via AUTO_INSTALL_UV=1 and pins a specific uv version; otherwise the script requires uv to be pre-installed and fails with instructions, avoiding unattended remote code execution as root.	2026-07-22 14:00:44 +07:00
grabowski	ab8a10dd75	Add install.sh and fix service unit placeholder ... - scripts/install.sh: one-command hardened deploy (creates the water-monitor system user, deploys to /opt, builds a uv-managed venv, installs and enables the systemd unit). Idempotent; excludes .env/*.db/stations.json from sync so runtime state is preserved. - Fix placeholder Documentation= URL in water-monitor.service. - README: document the script as the primary systemd install path, with manual steps kept as a fallback.	2026-07-22 12:55:17 +07:00
grabowski	4bc3d82773	Persist station CRUD across restarts via JSON config ... Station CRUD via the API previously mutated the scraper's in-memory station_mapping only, so changes were lost on restart (and the systemd service auto-restarts). - Extract the 130-line hardcoded station_mapping into bundled defaults at src/data/stations.json; the scraper loads from a runtime-writable config file (STATION_CONFIG_PATH, default stations.json) and falls back to the bundled defaults to seed it. - Add scraper.save_stations() with an atomic temp-file + os.replace write. - create/update/delete station endpoints now persist and roll back the in-memory change if the write fails; re-raise HTTPException so persistence errors surface as real 500s instead of being swallowed. - Backend-agnostic (works for the VictoriaMetrics deployment, which has no relational stations table). Runtime stations.json is gitignored. Also clears pre-existing flake8 debt in water_scraper_v3.py (unused imports, long lines, duplicate logging import) and dedupes the User-Agent to Config.USER_AGENT.	2026-07-22 12:51:29 +07:00
grabowski	6e78225d00	Fix optional-discharge crashes and dedupe measurement mapping ... - MeasurementResponse.discharge is now Optional[float]; measurements with a null discharge no longer raise a Pydantic ValidationError (HTTP 500) on the /measurements/latest and /measurements/station endpoints. - InfluxDB save_measurements guards float(discharge) against None instead of crashing with TypeError. - Extract the duplicated measurement->response mapping into a single _to_measurement_response helper used by both measurement endpoints.	2026-07-22 12:34:46 +07:00
grabowski	f4c63cabef	Fix Matrix message formatting and harden security ... Matrix alerts: - Send HTML formatted_body (org.matrix.custom.html) so **bold** and URLs render instead of showing literal Markdown; add plain-text body fallback. Add dependency-free markdown_to_matrix_html/strip_markdown helpers with HTML escaping of station/message data. Security: - InfluxDB: bind untrusted station_codes as query params and cast limit to int (was f-string interpolation / injection risk). - VictoriaMetrics: escape Prometheus label values and coerce metric values to float, preventing exposition-format injection and None crashes. - web_api: run blocking scrape cycle via run_in_executor so it no longer freezes the event loop; make CORS origins configurable and only allow credentials with explicit origins ("*" + credentials is invalid/unsafe). - config: remove hardcoded root/postgres password fallbacks (raise instead) and stop defaulting VM_HOST to a real infrastructure hostname. Also remove unused imports and wrap long lines to satisfy flake8.	2026-07-22 12:07:05 +07:00
grabowski	d3ec5a77e6	disable stale data alert	2026-01-12 15:21:20 +07:00
grabowskiandClaude	887b7ee938	Update alert messages to use public Grafana dashboard link ... - Changed dashboard link to public URL for easier access - Public dashboard: https://metrics.b4l.co.th/public-dashboards/655730aa044f44f49b355d01386018ca - No authentication required to view Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-31 11:33:31 +07:00
grabowskiandClaude	a424c50c5e	Change stale data alert threshold from 4 to 12 hours ... - Stale data alerts now only trigger after 12 hours without new data - Reduces false alerts during expected data gaps Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-31 11:29:52 +07:00
grabowskiandClaude	c57e46ae21	Filter alerts to upstream stations only and add Grafana dashboard link ... - Alerts now only sent for stations upstream of Chiang Mai: P.20, P.75, P.92, P.4A, P.67, P.21, P.103, and P.1 - Added Grafana dashboard link to all alert messages - Dashboard URL: https://metrics.b4l.co.th/d/ac9b26b7-d898-49bd-ad8e-32f0496f6741/psql-water - Fixed flake8 linting issues (line length, undefined variable, unused import) Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-16 17:25:27 +07:00
grabowskiandClaude	6a76a88f32	Update pyproject.toml to use dependency-groups instead of tool.uv.dev-dependencies ... - Replaced deprecated tool.uv.dev-dependencies with dependency-groups.dev - Follows new uv standard for dependency group declaration Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-09 11:56:54 +07:00
grabowskiandClaude	e62a20022e	Add pre-commit configuration ... - Added Black for code formatting (line-length 120) - Added isort for import sorting - Added flake8 for linting - Added standard pre-commit hooks for file checks Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-09 11:52:17 +07:00
grabowskiandClaude	58cc60ba19	Integrate automatic alerting into continuous monitoring ... - Alerts now run automatically after every successful new data fetch - Works for both hourly fetches and retry mode exits - Alert check runs when fresh data is saved to database - Logs alert results (total generated and sent count) Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:44:57 +07:00
grabowskiandClaude	cc007f0e0c	Add comprehensive alerting system tests ... - Created test suite for zone-based water level alerts (9 test cases) - Created test suite for rate-of-change alerts (5 test cases) - Created combined alert scenario test - Fixed rate-of-change detection to use station_code instead of station_id - All 3 test suites passing (14 total test cases) Test coverage: - Zone alerts: P.1 zones 1-8 with INFO/WARNING/CRITICAL/EMERGENCY levels - Rate-of-change: 0.15/0.25/0.40 m/h thresholds for WARNING/CRITICAL/EMERGENCY - Combined: Simultaneous zone and rate-of-change alert triggering Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:35:34 +07:00
grabowskiandClaude	de632cef90	Add rate-of-change alerting for sudden water level increases ... - Implement check_rate_of_change() to detect rapid water level rises - Monitor water level changes over configurable lookback period (default 3 hours) - Define rate-of-change thresholds for P.1 and other stations - Alert on moderate (15cm/h), rapid (25cm/h), and very rapid (40cm/h) rises - Only alert on rising water levels (positive rate of change) - Integrate rate-of-change checks into run_alert_check() cycle - Support both SQLite and PostgreSQL database adapters with fallback Rate thresholds for P.1 (Nawarat Bridge): - Warning: 0.15 m/h (15 cm/hour) - moderate rise - Critical: 0.25 m/h (25 cm/hour) - rapid rise - Emergency: 0.40 m/h (40 cm/hour) - very rapid rise Default thresholds for other stations: - Warning: 0.20 m/h, Critical: 0.35 m/h, Emergency: 0.50 m/h Alert messages include: - Rate of change in m/h and cm/h - Total level change over period - Time period analyzed This early warning system detects dangerous trends before absolute thresholds are reached, allowing for earlier response to flooding events. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:17:08 +07:00
grabowskiandClaude	e94b5b13f8	Fix Matrix API notification to use PUT method with transaction ID ... - Change HTTP method from POST to PUT for Matrix API v3 - Matrix API requires PUT when transaction ID is included in URL path - Move transaction ID construction before URL building for clarity - Fixes "405 Method Not Allowed" error when sending notifications The Matrix API v3 endpoint structure: PUT /_matrix/client/v3/rooms/{roomId}/send/{eventType}/{txnId} Previous error: POST request was being rejected with 405 Method Not Allowed Now working: PUT request successfully sends messages to Matrix rooms 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:09:23 +07:00
grabowskiandClaude	c93d340f8e	Implement zone-based alerting for P.1 (Nawarat Bridge) station ... - Add 8 water level zones plus NewEdge threshold for P.1 station - Zone 1: 3.7m (Info), Zone 2: 3.9m (Info) - Zone 3-5: 4.0-4.2m (Warning levels) - Zone 6-7: 4.3-4.6m (Critical levels) - Zone 8/NewEdge: 4.8m (Emergency level) - Implement special zone-based checking logic for P.1 - Maintain backward compatibility with standard warning/critical/emergency thresholds - Keep standard threshold checking for other stations Zone progression for P.1: - 3.7m: Zone 1 alert (Info) - 3.9m: Zone 2 alert (Info) - 4.0m: Zone 3 alert (Warning) - 4.1m: Zone 4 alert (Warning) - 4.2m: Zone 5 alert (Warning) - 4.3m: Zone 6 alert (Critical) - 4.6m: Zone 7 alert (Critical) - 4.8m: Zone 8/NewEdge alert (Emergency) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-10-03 16:04:12 +07:00
grabowskiandClaude	dff4dd067d	Implement strict freshness detection without grace periods ... - Remove tolerance windows and grace periods from data freshness checks - Require data from current hour only - no exceptions or fallbacks - If hourly check runs at 21:xx but only has data up to 20:xx, immediately switch to retry mode - Simplify logic: latest_hour >= current_hour for fresh data - Remove complex age calculations and tolerance conditions This ensures the scheduler immediately detects when new hourly data is not yet available and switches to minute-based retries without delay. Behavior: - 21:02 with data up to 21:xx → Fresh (continue hourly) - 21:02 with data up to 20:xx → Stale (immediate retry mode) - No grace periods, no tolerance windows, strict hour-based detection 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 21:03:03 +07:00
grabowskiandClaude	5c6a41b2b9	Enhance freshness detection to check for current hour data availability ... - Modify _check_data_freshness() to verify current hour data exists - If running at 20:00 but only have data up to 19:xx, consider it stale - Add tolerance: accept previous hour data if within first 10 minutes - Combine current hour check with age limit (≤2 hours) for robustness - Add detailed logging for current vs latest hour comparison This solves the core issue where scheduler stayed in hourly mode despite missing the expected current hour data from the API. Example scenarios: - 20:57 with data up to 20:xx: Fresh (has current hour) - 20:57 with data up to 19:xx: Stale (missing current hour) → Retry mode - 20:05 with data up to 19:xx: Fresh (tolerance for early hour) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 20:58:19 +07:00
grabowskiandClaude	1c023369b3	Implement intelligent data freshness detection for adaptive scheduler ... - Add _check_data_freshness() method to detect stale vs fresh data - Consider data fresh only if latest timestamp is within 2 hours - Modify run_scraping_cycle() to check data freshness, not just existence - Return False for stale data to trigger adaptive scheduler retry mode - Add detailed logging for data age and freshness decisions This solves the issue where scheduler stayed in hourly mode despite getting stale data from the API. Now it correctly detects when API returns old data and switches to retry mode until fresh data becomes available. Example behavior: - Fresh data (0.6 hours old): Returns True, stays in hourly mode - Stale data (68.6 hours old): Returns False, switches to retry mode 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 20:35:59 +07:00
grabowskiandClaude	60e70c2192	Fix validator to handle null discharge values properly ... - Make discharge field optional in data validator - Remove discharge from required fields list - Add explicit null check for discharge before float conversion - Prevent "float() argument must be a string or a real number, not 'NoneType'" errors - Allow records with valid water levels but malformed/null discharge data This completes the malformed data handling fix by updating the validator to match the parser's new behavior of allowing null discharge values. Before: Validator rejected records with null discharge After: Validator accepts records with null discharge, validates only if present 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 18:55:20 +07:00
grabowskiandClaude	cc5c4522b8	Fix malformed discharge data handling to preserve water level data ... - Change data parsing logic to make discharge data optional - Water level data is now saved even when discharge values are malformed (e.g., "***") - Handle malformed discharge values gracefully with null instead of skipping entire record - Add specific handling for "***" discharge values from API - Improve data completeness by not discarding valid water level measurements Before: Entire station record was skipped if discharge was malformed After: Water level data is preserved, discharge set to null for malformed values Example fix: - wlvalues8: 1.6 (valid) + qvalues8: "***" (malformed) - Before: No record saved - After: Record saved with water_level=1.6, discharge=null 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 18:41:39 +07:00
grabowskiandClaude	6846091522	Implement smart date selection for data fetching ... - Add intelligent date selection based on current time - Before 01:00: fetch yesterday's data only (API not updated yet) - After 01:00: try today's data first, fallback to yesterday if needed - Improve data availability by adapting to API update patterns - Add comprehensive logging for date selection decisions This ensures optimal data fetching regardless of the time of day: - Early morning (00:00-00:59): fetches yesterday (reliable) - Rest of day (01:00-23:59): tries today first, falls back to yesterday 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 18:30:53 +07:00
grabowskiandClaude	4cc792157f	Implement adaptive scheduler with intelligent retry logic ... - Replace fixed hourly schedule with adaptive scheduling system - Switch to 1-minute retries when no data is available from API - Return to hourly schedule once data is successfully fetched - Fix data fetching to use yesterday's date (API has 1-day delay) - Add comprehensive logging for scheduler mode changes - Improve resilience against API data availability issues The scheduler now intelligently adapts to data availability: - Normal mode: hourly runs at top of each hour - Retry mode: minute-based retries until data is available - Automatic mode switching based on fetch success/failure 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 18:24:33 +07:00
grabowskiandClaude	0ff58ecb13	Add historical data import functionality ... - Add import_historical_data() method to EnhancedWaterMonitorScraper - Support date range imports with Buddhist calendar API format - Add CLI arguments --import-historical and --force-overwrite - Include API rate limiting and skip existing data option - Enable importing years of historical water level data 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-28 14:46:54 +07:00
grabowskiandClaude	bd812ca5ca	Improve scheduler to run immediately then wait for next full hour ... - Run initial data collection immediately on startup - Calculate wait time to next full hour (e.g., 22:12 start waits until 23:00) - Schedule subsequent runs at top of each hour (:00 minutes) - Display next scheduled run time to user for better visibility 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-26 22:51:00 +07:00
grabowskiandClaude	ca730e484b	Add comprehensive Matrix alerting system with Grafana integration ... - Implement custom Python alerting system (src/alerting.py) with water level monitoring, data freshness checks, and Matrix notifications - Add complete Grafana Matrix alerting setup guide (docs/GRAFANA_MATRIX_SETUP.md) with webhook configuration, alert rules, and notification policies - Create Matrix quick start guide (docs/MATRIX_QUICK_START.md) for rapid deployment - Integrate alerting commands into main application (--alert-check, --alert-test) - Add Matrix configuration to environment variables (.env.example) - Update Makefile with alerting targets (alert-check, alert-test) - Enhance status command to show Matrix notification status - Support station-specific water level thresholds and escalation rules - Provide dual alerting approach: native Grafana alerts and custom Python system 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-09-26 16:18:02 +07:00