193 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
grabowski	701bb87d34	feat(p1): city flood stages and zones from CMU Water Center's 2024 flood map ... CMU Water Center's Chiang Mai flood hazard map (CMFlood) replaces the older municipal map. It is built on the October 2024 flood and has the Ping leave its banks above P.1 4.20 m, with five stages: 4.30, 4.50, 4.70, 5.00 and 5.30 m (the 2024 record). The old map's seven stages (3.70-4.60 m) said the city floods from 3.70 m; CMU calls that map out of date. - P1_FLOOD_STAGES: the five CMU stages, discharge on RID's 2024 rating. /forecast's P.1 `stages` now has five entries (same format). Thresholds are unchanged: 4.20 m danger is the bank top, 3.70 m warning half a metre below (as on CMU's gauge page). flood_config.json and the Rust parity fixtures regenerated (only the stages changed). - flood-zones-p1.geojson: CMU's KML (L1..L5) as five disjoint zones, zone k floods first at stage k; scripts/build_flood_zones.py rebuilds it. - Dashboard: CMU's colours and credit (map attribution); the stage grid follows the number of stages; zone lookup for markers honours holes. The city verdict uses statusOf's classes: flooding (over the banks) from 4.20 m, high water from 3.70 m. Zones show by themselves when the river is over its banks or stage 1 is >= 50 % likely. Alert, replay and verdict texts no longer call 3.70 m city flooding (EN + TH). - Docs: DATA_SOURCES 5.1 (CMFlood: the map, flood marks, poles, city gauges), FLOOD_FORECASTING, DASHBOARD_REDESIGN, NOTIFICATIONS, README credits.	2026-10-26 14:19:52 +07:00
grabowski	180d95c6bd	fix(forecast): no nightly gaps in the issued model forecasts ... The P.1 chart's "24 h peak, as issued" line had a gap every night and a dip at 21:00. Two causes: - A run forecasts from the newest reading only. RID publishes 22:00..06:00 in one batch in the morning (and now and then one hour late), so those hours were never forecast from. After each run the collector now forecasts the hours of the last 48 h (two weeks on the first run after a start) that have a RID reading but no model forecast, each as a run at that hour would have: gauges, HII fill and dam state up to that hour. Rain is fetched once, so a late hour's rain_fc24 partly holds rain that has since fallen (as in training). Only missing (hour, station) pairs are written. `ping-collector catch-up-forecasts --hours N` goes further back. - Overnight each hourly run re-issues the 21:00 reading, after 6 h as the heuristic, which overwrote the 21:00 model row. A heuristic row no longer replaces a model row in forecast_history. The dashboard draws only model rows on the issued line. Checked on a local copy of 18 days of RID data: the catch-up of an hour matches `ping-collector forecast --now <hour>`; 288 hours took 20 s (debug build). Tests for both database rules.	2026-10-20 15:21:48 +07:00
grabowski	cf10f2e4c8	feat(dashboard): console phase 4 - 2024 replay, simulation, replaces the old page ... The console is now the dashboard at / (src/static/dashboard.html); the old page and console.html are gone. - Replay of the October 2024 flood (flood-2024-all.json): the page's clock, verdict, list, markers, flood zones and panel follow the replay; the panel charts the season on fixed axes with the 2024 model's 24 h peak. Hourly while the model alerts or P.1 is above 3.50 m, faster through quiet days; pause and drag. Rain, basin stations and the 5-minute refresh are off during the replay (today's data). - ?demo_level= and ?demo_rise= simulate P.1. A violet REPLAY / amber SIMULATION bar, the top bar's KPI and the P.1 panel say so; the mode is drawn from state.mode alone, so a refresh or language switch cannot relabel either as live. - History: rows without discharge_percent are ThaiWater's copy of the gauge (night hours RID publishes late), with a different rating (P.1: 26 against RID's 54 m3/s); their discharge is not drawn for gauges with RID discharge. The discharge strip has headroom so its label stays clear of the line. - Swap: the old page's title, description, Open Graph and JSON-LD tags; no noindex; tests/test_dashboard.py = console tests + the old checks that still apply (one P.1 level, SEO files) + a replay/simulation labelling test. - Docs: README screenshot (docs/img/dashboard.jpg, replacing two unused images), CLAUDE.md, CONTRIBUTING.md, FLOOD_FORECASTING.md (where the model shows), DASHBOARD_REDESIGN.md (phase 4 done, mode rules). Deploy needs a restart: the server reads dashboard.html at start.	2026-10-20 13:47:15 +07:00
grabowski	ffd3030352	feat(dashboard): console phase 3 - history, forecast accuracy, data statistics ... - History in the station panel (inline SVG): 3 d / 7 d / 30 d / 90 d / 1 y / all or a date range; level and discharge, the 24 h peaks as the model issued them, warning/danger bands, hover/touch values. Up to 7 d the chart ends in the current forecast. Above 2,500 readings it shows the daily mean with the daily low-high band, so a flood peak is not averaged away; gaps (RID's nightly pause) break the line instead of being bridged. Days are Bangkok days computed arithmetically: Intl per reading took a minute for "All" (67,000 rows), now 2 s. - "How accurate is the forecast?" for P.1 (/api/forecast/skill), loaded when opened because a cold skill query takes up to 20 s. - Data statistics (/api/stats) behind "2.7M readings since 2018" in the status bar. fix(dashboard): "All time" history ended in October 2024. The history route returns the oldest rows first and stops at 50,000 by default; P.1 has about 67,000 hours. Both dashboards now ask for limit=100000 (the route's maximum).	2026-10-19 17:12:39 +07:00
grabowski	71cc1b0ca7	feat(dashboard): console phase 2 - self-hosted basemap, basin stations, markers ... - Basemap: src/static/basemap/ping-basin.pmtiles, a Protomaps extract of the Ping basin (zoom 0-12, 15 MB, OpenStreetMap data), drawn by protomaps-leaflet 5.1.0 (unpkg, SRI) in its light/dark style with labels in the page language. It replaces the dimmed OSM raster tiles, whose baked-in labels were the noisiest part of the screen; OSM stays as the fallback. The browser reads the file with range requests (ServeDir answers 206; dev_proxy.py now does too). The map stays within the basin. - Basin stations: a second list next to the RID gauges and a map layer of diamonds, from /api/hii/waterlevel/latest (falling back to /sensors/thaiwater), coloured by ThaiWater's bank-height classes (over the bank = danger, 70-100 % = watch). Gauges RID already lists appear once. The panel shows level (m MSL), % of bank height, distance to and height of the bank top, gauge reading and agency; selection in the URL (#hii-<id>). - custom-markers.json as pins from zoom 12, with the flood zone they are in. Refresh instructions for the basemap in src/static/basemap/README.md.	2026-10-19 16:58:21 +07:00
grabowski	301a629b42	feat(dashboard): the console, phase 1 of the redesign (preview at /static/console.html) ... The user chose direction B of the redesign: a full-screen operations console. It is built next to the current dashboard, served by ServeDir at /static/console.html (noindex, links back to /), and replaces dashboard.html once it covers every feature (docs/DASHBOARD_REDESIGN.md, phase 4). Phase 1: - top bar: the Chiang Mai verdict (P.1 level + 24 h forecast, the current dashboard's rule), gauges reporting, highest 24 h flood chance, data age ("overnight pause" instead of "stale" during RID's nightly gap, as in the notifier), EN/TH, light/dark (dark default), Get alerts - gauge list: one status rule for list, map and panel (normal, watch within 0.5 m or >= 25 % chance, warning, danger, no data), margin to the warning level, 72 h sparkline, 6 h change; filter; sort by river order, closest to the warning level, rising; arrow keys - map: Ping mainstem and tributaries, status markers with code + level, layers Rain 24 h (on by default only with heavy rain) and Flood zones (auto-shown while P.1 is at a stage), one legend - station panel: level, change, discharge (last RID value when the newest hour is level-only), warning/danger levels with margins, capacity, level scale with now and 24 h peak, 72 h chart with the forecast peaks, forecast table with the model and issue time, the P.1 inundation stages, data links - alerts drawer with the ntfy topics; selection in the URL (#P.84); refresh every 5 minutes - Inter and Noto Sans Thai self-hosted (src/static/fonts, 75 KB, OFL), no emoji, Thai first, Bangkok time tests/test_console.py: routed fetch paths, EN/TH parity, translated aria labels, no emoji, fonts and licences, noindex. dev_proxy.py serves src/static with real content types (it sent HTML as octet-stream).	2026-10-19 16:31:09 +07:00
grabowski	0e30ddbcb6	fix(notify): no "gauge feed stale" for RID's nightly pause on a quiet river ... ping-status sent "gauge feed stale" around midnight and "feed recovered" around 07:00 on each of the last three nights (ntfy and, since phase 3, the Matrix room). RID publishes no hourly report overnight: the 21:00 reading is the last until the 22:00..06:00 hours arrive together (06:35, 06:55, 07:22 on 2026-10-17/18/19). The ThaiWater overlay was meant to bridge that, but ThaiWater's copies of the RID gauges come from the same feed with the same delay (P.1's mirror still has no value for 22:00-00:00 this morning), so it does not. Between 21:00 and FEED_NIGHT_UNTIL_HOUR (default 9, 0 = off) the feed of a quiet river is no longer reported stale; one still without new readings at 09:00 is. While a station is at warning/danger or the P.1 outlook is on, the 3 h rule holds all night. A feed already stale stays stale through the night (no false "recovered"), and a real recovery is announced at any hour. The replay of the Python notifier runs with the window off. Docs no longer claim that ThaiWater keeps the feed current at night.	2026-10-19 09:35:10 +07:00
grabowski	7f29b1831e	fix(install): find uv for the .venv owner (root shells and sudo lack ~/.local/bin) ... Run as root, SUDO_USER is unset and uv is not on root's PATH, so the training environment was not synced. Now the sync runs as the owner of .venv, with uv from PATH, that user's ~/.local/bin or ~/.cargo/bin, /usr/local/bin, or UV=/path/to/uv.	2026-10-18 17:55:33 +07:00
grabowski	24998b033c	fix(deps): python-dotenv 1.2.2 (PYSEC-2026-2270, set_key symlink overwrite) ... Now a direct dependency, pip-audit sees it; it used to come in through uvicorn. The .env parser is unchanged: rust/collector/testdata/gen_envfile_cases.py regenerates the 614-case fixture byte-identical under 1.2.2, so envfile.rs still matches.	2026-10-18 17:31:41 +07:00
grabowski	6fe2e287e5	docs: rewrite the README for the Rust service	2026-10-18 17:15:07 +07:00
grabowski	7e642555b4	chore: remove the Python service; Python is the training stack only ... Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service.	2026-10-18 17:15:07 +07:00
grabowski	330fdf7804	docs: website/API in ping-collector since 2026-10-18; quiet the install script's port probes	2026-10-18 16:42:40 +07:00
grabowski	779728e38b	feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions ... Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what the FastAPI app (src/web_api.py) served, so one process collects, forecasts, notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside `ping-collector run`; `ping-collector serve` runs it alone. Same routes, parameters, JSON fields and field order, status codes, {"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip (>500 B), CORS, Umami API tracking, the dashboard and static files, and the stale-while-revalidate response caches (now with byte budgets: a history response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI document is built from the route table, whose per-route parameter declarations also validate the requests, so docs and behaviour cannot drift. Python output conventions are kept: isoformat(), round(), numpy's pairwise sums (catchment means), Python's summation order (skill metrics). scripts/api_parity.py compares the two servers route by route on one database (53 cases incl. errors, admin and static files; query values vary per run so FastAPI's caches answer fresh). Locally: 53 identical; the station CRUD writes byte-identical stations.json files; the skill/trend logic matches on 9 station/horizon pairs with synthetic model versions. Intended differences: /health counts the ThaiWater telemetry of the same gauges while RID lags (as the notifier); /scraping/status adds the collector version and last forecast run; an old stored forecast is served marked X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute; .geojson is served as application/geo+json. Load (local, 16 clients): the dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s; 25 MB resident when serving only. Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets (per station, outlook, feed; not ntfy's per-topic summaries), each once, with its own matrix: keys in notification_state, seeded from ntfy's on first use so nothing already announced repeats. Checked against a fake homeserver. It replaces the `run.py --alert-check` cron job (repeated alerts every 15 min), which install_collector.sh now comments out (crontab backup kept). install_collector.sh: --check-api (ping-collector serve on :8001 next to FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets WEB_LISTEN, stops and disables water-monitor.service), --rollback-api; --rollback-forecast and --rollback bring the Python app back as server. ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route tests against a throwaway schema.	2026-10-18 16:08:06 +07:00
grabowski	1d1c000922	docs: forecast + ntfy in ping-collector since 2026-10-18	2026-10-18 15:10:03 +07:00
grabowski	6ae67dec63	feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night ... Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock).	2026-10-18 14:43:39 +07:00
grabowski	5c4f464383	feat(forecast): Rust forecast crate, bit-for-bit with the Python serving path ... Phase 2a of docs/RUST_REWRITE.md. rust/forecast (ping-forecast) ports the serving half of src/ml: the HII mirror gap fill, the hourly grid, every feature (pandas' Kahan rolling sums included), the Mae Ngat dam frame, HistGradientBoosting tree inference, the heuristic fallback, stale-threshold handling and the P.1 flood-stage probabilities. Hand-over from Python (src/ml/export.py): - models/flood_<station>.json.gz next to each joblib bundle; train_all() writes it, retrain.sh promotes it, `python -m src.ml.export` exports existing ones. - rust/forecast/flood_config.json: thresholds, stages, leads and notify constants, compiled in. tests/test_model_export.py fails while it differs from the Python source. Parity (rust/forecast/tests/parity.rs, fixtures from testdata/gen_forecast_fixtures.py on the production code path): four scenarios incl. the 2024 flood with RID lagging behind HII. Merged readings, all features and all levels identical; with all 15 local models 377/384 probabilities bit-identical, the rest within 2 ulp (exp). Python refactor without behaviour change: predict.forecasts_from_long, and get_latest_forecasts/load_latest/serving_frame take now/rain for reproducible comparisons; the HII query orders its rows.	2026-10-18 14:21:56 +07:00
grabowski	6d4beae65a	fix(ml): serve the forecast rain the model was trained with ... build_features cut the rain series to the gauge grid before the forward sum, and the grid ends at the newest reading, so the row every live forecast is made from had rain_fc24 = NaN. Training and all evaluations had the real Open-Meteo forecast there. The forward sum now runs over the grid extended by 24 h; earlier rows are unchanged. Harness (new opt-in variant rise_rain_fc24_missing = rise_rain with rain_fc24 blanked on test rows, five folds, --from-cache): P.1 and P.103 nearly unaffected, P.21 false alarms 10 -> 5 and MAE 0.232 -> 0.215 with the fix. Write-up in docs/FLOOD_FORECASTING.md section 5; results in models/eval_2026-10-18_fc24.json. Also: HII mirror rows in one hour now resolve to the latest reading via a stable sort on the raw time (was an unstable sort on the floored hour, i.e. row-order dependent). Needed for an exact Rust port.	2026-10-18 14:07:56 +07:00
grabowski	64c5b02e06	fix(api): /health never serves a cached result older than 4 TTLs ... Stale-while-revalidate answered the first /health after a quiet night with the previous evening's state (seen after the collector cutover: 'Latest data is old: 21:00' at 13:47 while data was current). _cached_swr gains max_stale; /health uses 4 x HEALTH_CACHE_TTL_SECONDS.	2026-10-18 13:49:39 +07:00
grabowski	9b910c3cbf	chore(rust): gap-fill progress lines; slow-statement warning at 30 s	2026-10-17 23:15:43 +07:00
grabowski	785836dcd8	docs: ping-collector in production since 2026-10-17	2026-10-17 23:12:29 +07:00
grabowski	c8e89840b5	fix(rust): read .env like python-dotenv; dry run in the service environment ... dotenvy aborted on the production .env (a quote in an unquoted value makes it read to the end of the file) and echoed the offending line, password included. envfile.rs ports python-dotenv 1.1.1 (parser + ${VAR} interpolation, no override): unparsable statements are skipped with a warning that names only the line number. Verified against python-dotenv on 614 generated inputs (testdata/). The .env is loaded before the tokio runtime starts. install_collector.sh now runs the dry run through systemd-run with the unit's User, WorkingDirectory and EnvironmentFile, i.e. exactly what the service sees.	2026-10-17 22:58:12 +07:00
grabowski	63353f1e80	build(rust): declare rust-version 1.88; rustup setup for the server	2026-10-17 22:39:45 +07:00
grabowski	4e619aa2ae	feat(rust): ping-collector, phase 1 of the Rust rewrite ... rust/collector replaces the collection half of the Python web app: RID hourly gauges (with the late-report semantics: store everything, retry every 10 min while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain + water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and upsert semantics, one UNNEST statement per table, Bangkok wall-clock time regardless of the server time zone. Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and instead re-forecast + notify when the newest RID reading in the database advances (polled every FORECAST_POLL_SECONDS) and once per clock hour; /scraping/status then serves the collector's status file; /scrape/trigger answers 409. Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against PostgreSQL); live parity run of both collectors into two empty databases: all 9 tables row-for-row identical (scripts/collector_parity.py); the Python API served health, measurements, stats, status and model forecasts from a database filled only by ping-collector. 177 Python tests pass. reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which rustls refuses. Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service; CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service, release artifact).	2026-10-17 22:20:02 +07:00
grabowski	e3694d7591	docs: Rust rewrite assessment + forecast parity spike ... spikes/rust-forecast ports build_features (P.1 path) and HistGradientBoosting inference to Rust and checks them against the pandas/sklearn output on two fixtures (2024 record flood, summer 2026): 244,800 feature cells and every regression prediction bit-identical, classifier heads to the last ulp; 0.25 ms vs 158 ms per station. docs/RUST_REWRITE.md: what a rewrite would mean (Rust service, Python training), costs, phased plan, Python alternative.	2026-10-17 21:53:05 +07:00
grabowski	04ebdaa931	fix: RID collector keeps late readings instead of losing every evening ... RID's hourly report often lags 1-2 h in the evening. The cycle discarded the whole report unless it already held the current hour, the web-API loop then slept a full hour, and after 01:00 only today's report was fetched. Result: the 19:00-24:00 readings were dropped most nights (P.1 lost 00:00 on 23 of the last 60 days, 13 evening hours this week), the feed went stale from ~21:00 until morning and ntfy ping-status sent a stale/recovered pair daily. - run_scraping_cycle() upserts every reading the report has; "current hour present" now only decides the retry cadence (returned as before). - Yesterday's report is re-fetched while the store has holes in it (hours 22-24 are only reachable through that day's report). - Leader loop polls at :05 (RID_POLL_OFFSET_MINUTES) and retries every RID_RETRY_MINUTES (10) while RID is behind; HII/dams/rain once per clock hour; forecast precompute + ntfy when newer readings arrive or hourly. - Startup re-fetches holes from the last GAP_FILL_DAYS_ON_START (14) days. - /scraping/status: the leader shares its stats via a JSON file, so any worker reports them (was always 0 runs); adds feed_current, latest_reading, readings_saved.	2026-10-17 21:45:09 +07:00
grabowski	32399f1899	feat: recalibrate P.77/P.75 thresholds; capacity guard on alerts ... The first ntfy cycle announced "Warning level at P.77" at 3.02 m. That gauge's 2.85 m threshold sat below its own dry-season baseline (2.6-2.7 m at 8-14 % channel capacity): P.77 had been "above warning" for 761 of the last 2 146 hours, at 22 % capacity. Across 2018-2024, 75-85 % capacity reads 3.35-4.57 m and 95-105 % reads 4.27-5.08 m; set 4.30 / 4.90. P.75 moved 2.75/3.50 -> 3.20/3.65 on the same evidence (2024: 3.45 / 3.72). The predictor already handles changed thresholds (regression-derived probabilities until the Oct 1 retrain). Second line of defence in notify.py: a clear->alert transition is only announced when RID's discharge_percent for the reading is >= 60 %, so a re-rated or datum-shifted gauge cannot page subscribers again. P.1 is exempt (its stages come from the inundation map, not capacity); readings without a capacity figure fall back to level only; the all-clear edge is never blocked. 3 tests.	2026-09-12 00:34:59 +02:00
grabowski	f4d42c90f4	fix: ntfy listens on the Tailscale address; monitor publishes to it directly ... The reverse proxy is a separate VPS on the tailnet, so a loopback-only ntfy was unreachable from it. install_ntfy.sh now binds the host's Tailscale IP (NTFY_LISTEN overrides). New NTFY_PUBLISH_URL: where the monitor POSTs, separate from the public NTFY_SERVER subscribers see, so an alert never waits on DNS or the proxy (first cycle logged 502s from Cloudflare while the domain was not yet proxied).	2026-09-12 00:28:29 +02:00
grabowski	039d24a5c3	fix: init ntfy only in the collection leader ... Every uvicorn worker ran the notification_state DDL at startup; on Postgres the losers of that race get UniqueViolation on pg_type and the whole init was skipped (notifications off). Only the leader publishes, so only the leader initialises, after election. The DDL also tolerates a concurrent creator now: on failure it verifies the table exists instead of giving up.	2026-09-12 00:22:03 +02:00
grabowski	777b230baf	feat: public flood notifications over self-hosted ntfy ... Anyone can now get push alerts on their phone without an account: the monitor publishes to an ntfy server (one Go binary, ~30 MB RSS) and subscribers pick topics in the free iOS/Android/web app. Semantics are transitions, never state. One message when a gauge crosses its warning or danger threshold, one all-clear when it drops back (0.10 m hysteresis), nothing while it sits above. A three-day flood is two messages; a quiet season is zero. Topics: ping-warning / ping-danger (basin digest), ping-<station>-warning / -danger, ping-p1-outlook (opt-in: model P(warning within 24 h) at P.1 rises through 50 %, clears below 25 %, message says it is experimental), ping-status (feed stale >= 3 h / recovered). Priority 5 on danger so it rings through Do Not Disturb. src/notify.py runs once per collection cycle in the API process (leader only, after the forecast precompute, same data the dashboard shows). Last-sent state lives in a notification_state table so a restart never re-sends; a failed publish leaves state untouched so the crossing is retried next cycle instead of lost. Off unless NTFY_SERVER is set. Dashboard: a "Get alerts" button (only when configured) opens a panel with the server, per-topic cards, ntfy:// deep links and web links, app store links and a disclaimer. EN + TH. GET /api/notifications feeds it. scripts/install_ntfy.sh: .deb install, server.yml (loopback listen, anonymous read, token-only write scoped to ping-*, 72 h cache, signup/ login/metrics off, tight visitor limits), systemd, user + token, .env. Verified against ntfy 2.28.0: anon publish 403, token publish 200, token on foreign topic 403, anon read 200, and a seeded crossing through the real _notify_transitions path arrived in the topic with priority, tags, click and action button. docs/NOTIFICATIONS.md has the deployment and reverse-proxy notes. Tests: 10 for the state machine (159 total).	2026-09-12 00:18:38 +02:00
grabowski	0ec675e9c5	ci: license report from a clean venv, not the runner's site-packages	2026-09-11 23:54:44 +02:00
grabowski	7b31d4d0dd	feat: "Is the model getting better?" - live verification per model version ... src/ml/skill.py joins forecast_history (what each deployed version predicted for the 24 h peak, hourly) to water_measurements (what the river did) and reports per version: verified hours, peak MAE, bias, the persistence baseline (peak = current level), skill = 1 - MAE/persistence, and the same MAE restricted to observed peaks >= 2 m. Only forecasts whose window has elapsed with >= 75 % of hours observed count; a version needs 24 verified hours before it is compared. GET /api/forecast/skill?station_code=P.1&horizon=24 returns it (SWR cached, 15 min). The dashboard's forecast card gains a panel with a one-line verdict (current vs previous version), the per-version table, and a caveat that quiet weeks measure quiet-river accuracy only: the model is judged on flood-onset lead, which the backtests cover. EN + TH. On today's production data: hgb-v3+28b62e5 (369 h, Aug 13 - Sep 1) MAE 15.2 cm, skill -0.05; hgb-v2+f6570ac (224 h, Sep 1 - 11) MAE 12.3 cm, skill 0.36 - the "worse" v2 model scores better on a quieter fortnight, which is exactly why the panel shows the >= 2 m column and the caveat. Tests: 3, sqlite, synthetic. scripts/dev_proxy.py: DEV_PROXY_LOCAL lets a not-yet-deployed endpoint be answered from a local JSON file while everything else goes to prod.	2026-09-11 23:44:44 +02:00
grabowski	2e19974fad	docs: README describes the project that exists; CLAUDE.md for agents ... The README was the original template: P.1 "in Nakhon Sawan", P.103 "in Bangkok", VictoriaMetrics as the recommended database, Docker/Grafana sections, github.com/your-username support links. Rewritten around what runs: sources, gap fill, the forecast and its 13 h result, the live API table, systemd deployment with the retrain timer, repository layout, real docs links, data-source credits. Other DB adapters are mentioned as supported-but-not-production. CLAUDE.md replaces the untracked Ruflo boilerplate with project rules: Python 3.11, format/test gates, Bangkok timestamps, harness-first model changes judged on lead, the RainUnavailableError guard, no git add -A.	2026-09-11 23:44:43 +02:00
grabowski	b03318210c	security: pip-audit + bandit gates that can fail; patch 29 known CVEs ... security.yml previously ran safety/bandit/semgrep with `|| true` and could not go red. Now: pip-audit on requirements.txt is a hard gate (dev deps reported only), bandit HIGH fails (B104 bind-all skipped: intended behind Cloudflare/Caddy), pip-licenses uploaded as a report. Weekly + on dependency/source changes. Running it locally found 29 advisories, all in pinned-and-forgotten runtime deps: starlette 0.27 (7, incl. Host-header path confusion and form DoS), fastapi 0.104, requests 2.31 (3), pymysql 1.1. Bumped to current: fastapi 0.141.1 / starlette 1.6.0, pydantic 2.13.5, uvicorn 0.52.4, requests 2.34.2, pymysql 1.2.0; dev: pytest 9.1.1, black 26.5.1. pip-audit is now clean. requires-python narrowed to 3.11 (the truth: psycopg2-binary 2.9.9 fails on 3.13; pandas 2.0.3 has no 3.12 wheels). Full suite passes; API smoke-tested (health, stations, forecast, history, stats, docs, openapi) on the new stack. black 26 reformatted 8 files.	2026-09-11 23:44:43 +02:00
grabowski	97a6694ab2	feat(dashboard): light/dark theme ... Sun/moon button next to the language switch. Follows the OS preference until the user picks one (persisted in localStorage, tracks OS changes only while unpinned). Every colour that was a literal white / grey is now a token with a dark counterpart; flood-verdict, live-pill and lang-toggle states got their own bg/ink/border tokens. OSM tiles are inverted with a hue rotate so roads and labels stay legible while the river network, markers and rain dots (SVG, unfiltered) keep their data colours. Chart.js reads grid/tick/legend colours from the tokens and the open chart is rebuilt on toggle. Leaflet popups and controls follow the theme.	2026-09-11 23:16:30 +02:00
grabowski	5ad8e4eac3	ci: green pipelines that check what exists; one formatting contract ... The Test Suite job failed on every push since the black check was added because the tree had never been formatted, and pre-commit said 120 columns while CI ran black's default 88. pyproject.toml now carries [tool.black] / [tool.isort] (88, black profile) as the single source; pre-commit reads it; `make format` applied it (13 files, whitespace only, 146 insertions / 128 deletions, tests unchanged at 146 passed). ci.yml: lint (black, isort, flake8 hard errors) + pytest. The Docker registry push, VictoriaMetrics integration test, staging/production deploy and Apache-Bench jobs were template scaffolding for hosts and registries that do not exist; production is a systemd unit updated by git pull. Removed rather than left permanently skipped. docs.yml: the "Check markdown links" step curl'd every URL in every .md and failed on localhost examples and the Tailscale IP, and the Sphinx jobs built artifacts nobody read. Replaced by two checks that mean something: relative links/images in README, CONTRIBUTING and docs/ resolve inside the repo, and the FastAPI OpenAPI schema exports with the documented endpoints present (uploaded as an artifact).	2026-09-11 23:05:37 +02:00
grabowski	6f4a86edbb	fix(dashboard): timestamps are Asia/Bangkok everywhere; stale feed says so ... The API emits naive ICT timestamps ("2026-09-12T02:00:00"). The page fed them to new Date(), which applies the BROWSER's zone: a viewer in Europe parsed a 02:00 ICT reading as 02:00 CEST, five hours in the future, so "Last updated" clamped to "0 min ago" forever; a viewer in the Americas saw thousands of minutes. parseTs() now pins +07:00 on naive strings and every display formats with timeZone: Asia/Bangkok, so the site shows river time regardless of where it is opened. Daily/hourly chart buckets key on the Bangkok calendar day instead of UTC getters. The tile also gets a real stale state: past 3 h (RID is hourly) it turns red, reads "Feed stale · last reading <day>, N h ago", and the header pill switches from LIVE DATA to STALE FEED. Age shows hours past 2 h. scripts/dev_proxy.py serves the working-copy dashboard with API calls proxied to water.buildfor.life so browser-side changes can be checked against live data (and any browser timezone) before deploying.	2026-09-11 23:00:39 +02:00
grabowski	ce08312c0f	docs: public dashboard URL (water.buildfor.life) replaces the Tailscale IP everywhere	2026-09-11 22:38:13 +02:00
grabowski	d621aa9ce7	eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate ... Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_* caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.	2026-09-11 21:55:37 +02:00
grabowski	764764e07e	feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote ... train_all() now raises RainUnavailableError when use_rain=True and the Open-Meteo history cannot be loaded, instead of logging a warning and writing gauge-only (v2) bundles over the deployed v3 set -- which is what the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit way to get v2. CLI exits 2 with a one-line error. Three tests cover the guard, the opt-out, and the v3 happy path. scripts/retrain.sh trains into models/.staging, refuses to promote unless metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles into place (previous generation kept in models/.previous). No API restart: predict.py reloads by mtime on the hourly precompute. water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent, OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now does `uv sync` into .venv (one env rule; removes a stale venv/) and enables the timer. water-monitor.service in the repo matched neither the deployed unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).	2026-09-11 21:37:11 +02:00
grabowski	0a4bf843ff	chore: remove empty shell-accident files committed at repo root	2026-09-11 20:54:48 +02:00
grabowski	f6570ac10f	chore: declutter the repo ... Removes 26 tracked files that no longer describe or serve the running system, verified one by one against the whole repo (source, tests, docs, README, Makefile, Dockerfile, .gitea workflows, pyproject, packaging spec) plus dynamic-reference paths, before deletion. Root (11): one-off launch/setup write-ups from the project's first weeks that document events which never happened the way they describe — a github.com publication (the remote is self-hosted Gitea) and a 15-minute scheduler (the service runs hourly). Also .gitlab-ci.yml (unused, CI is .gitea/), .env.postgres and setup.py.backup (a placeholder env file and a backup in version control), and the PyInstaller packaging trio build_executable.py / build_simple.py / ping-river-monitor.spec, which bundled docs that no longer exist and is not how this deploys. docs (9): stale guides superseded by DATABASE_DEPLOYMENT_GUIDE, GITEA_WORKFLOWS, FLOOD_FORECASTING and DATA_SOURCES, plus two snapshots (PROJECT_STATUS, PROJECT_STRUCTURE) describing a 4-file src/ that is now 39. scripts (5): one-shot bootstrap tools already run — init_git.sh/.bat, generate_badges.py, migrate_geolocation.py, encode_password.py. Every inbound reference was fixed rather than left dangling: README doc index and migration section, three Makefile targets, the docs.yml summary step, and the GITEA_WORKFLOWS resource list. src/ is deliberately untouched. The audit proposed removing several live modules; verification showed those proposals were mis-scoped and would have broken production. .gitignore now covers the agent tooling dirs, model eval output and editor/shell leftovers — the working tree had collected 56 zero-byte files named after fragments of shell commands. 136 tests pass; production modules import clean.	2026-08-14 13:45:03 +07:00
grabowski	7e64e0cf18	fix: one current river level, not two ... The verdict banner and the P.1 outlook each wrote state.p1Now from a different feed — the banner from the latest measurement, the outlook from current_level on the forecast rows, which carries whatever the model saw at its as_of. Forecasts are precomputed hourly, so the two drifted apart: production showed 1.66 m in the banner and 1.52 m in the outlook directly below it. Harmless at low water; at flood stage two contradictory river levels on one screen undermine the warning. setP1Level() now arbitrates: freshest timestamp wins, and the replay and demo hooks pass force since they deliberately pin a level that is not the live one. The outlook renders the arbitrated value and clamps the shown peak to at least the current level, so a stale forecast can no longer predict a peak below where the river already is. endReplay drops the replayed level so live data re-arbitrates cleanly. Verified against a stub reproducing the exact production conditions (gauge 1.66 at 08:35 vs forecast 1.52 at 08:00): both now read 1.66; a 2.50 m rise against a stale 1.81 m peak renders 2.50/2.50; the 2024 replay still tracks its frames and returns to live on stop.	2026-08-14 09:25:39 +07:00
grabowski	382daa7d86	perf: backfill one dam per request via the api/dam range endpoint ... GET app.rid.go.th/reservoir/api/dam?dam_id&date_start&date_end returns a single dam's whole date range in one response — Mae Ngat's 2009-today archive is ~4 chunked requests instead of the ~2,900 one-day POSTs the all-dams path needs. Field names differ from api/dams and are mapped in parse_dam_range_records, verified equal on spot-checked dates; the range endpoint also carries DMD_ULevel, the reservoir level in m MSL that api/dams stopped publishing after ~2013. scripts/backfill_rid_reservoir.py defaults to the fast per-dam path (--all-dams keeps the full-fleet crawl, --refresh rewrites stored days). Already-stored dates are still skipped, junk values are still bounded, and the consecutive-failure abort still applies.	2026-08-13 23:10:36 +07:00
grabowski	b02e815d72	feat: Thai localisation and portrait-phone layout for the dashboard ... Thai is the default unless the browser prefers English, chosen by first match in navigator.languages order and remembered in localStorage. A STRINGS table carries both languages (interpolated strings as functions), applyTranslations() drives static markup via data-i18n attributes, and setLang() rebuilds everything the JS renders — including map layers, so popups render in the current language and sensor markers are replaced rather than stacked. Thai dates use the Buddhist era, matching the replay label; station names lead with the reader's language. Portrait phones: the header overflowed a 412 px Android viewport by 64 px, so the page scrolled sideways and the Refresh button sat off screen. The header now wraps into two deliberate rows (DOM order matches visual order, so focus order is unaffected), the map description box is hidden on phones, map height is capped by viewport — including a height-gated rule for landscape phones, whose 850-960 px widths never matched the width breakpoints — and the forecast grid goes single column. Verified in a real browser at 412x915, 360x800, 915x412 and 1440x900: zero horizontal overflow, no desktop change. Review-swarm fixes: a language switch no longer relabels a pinned SIMULATION or the 2024 replay as LIVE DATA (it kept the mode from state); Thai wording corrected where it asserted a rising trend the code never checks, labelled every gauge 'critical', or used a malformed compound; aria-labels, the Leaflet load failure and the flood-stage chips are translated; the language toggle states its action instead of an aria-pressed value that contradicted its label; and Thai font families sit after the Latin stack so they cannot restyle English text.	2026-08-13 23:10:34 +07:00
grabowski	5dc5850df6	docs: source sweep — P.75 already is the Mae Ngat release signal ... Verified from a Thai ISP that lsim.rid.go.th is unreachable (not geo-blocked), then swept for any better-than-daily Mae Ngat source. Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has been a model feature since v1 — the model has always read the dam's actual outflow, hourly and directly. That is the likelier reason the daily reservoir table adds nothing, beyond its publication lag. Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with no archive, so they cannot retrain against past events; the HII collector accumulates them from 2026-08-11 for a post-monsoon revisit. Also documents api/dam (whole per-dam range in one request, vs the ~2,900-request per-day loop) and several cross-check mirrors.	2026-08-13 21:58:47 +07:00
grabowski	70e4da07a0	docs: lsim.rid.go.th is unreachable, not geo-blocked ... Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s over the same connection. Correct the earlier note that assumed the timeout was a foreign-network block, and stop pointing the dam-feature follow-up at a host that cannot be reached.	2026-08-13 21:00:24 +07:00
grabowski	28b62e5a36	feat: Mae Ngat dam features — built, evaluated, defaulted OFF ... src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame (daily row visible from 07:00 its own date, ffill capped at 48 h) and is plumbed through features/train/predict/evaluate exactly like rain, gated to the six mainstem stations below the Mae Ngat confluence. The experiment concludes as a documented NEGATIVE result: on the 2024 record-flood backtest every dam-feature subset costs 1-3 h of first-alert lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID report lags up to 31 h and describes yesterday's benign absorbing reservoir during fast onset. Features therefore default OFF (--dam opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness variants, excluded from the default variant set). The ablation also isolated the HII gap-fill as lead-neutral: the acceptance gate holds at 13 h with fill enabled, and docs/img charts are regenerated with the shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5. Review-swarm fixes: evaluate.py skips variants whose feature family is absent instead of crashing the run; --dam forwards --db-url and warns loudly when no dam history loads; an empty DB result can no longer wipe a good dam cache; run-level metrics version claims v4 only when a dam station is actually in the set.	2026-08-13 20:42:21 +07:00
grabowski	6af6fbe02c	fix: survive junk RID dam values — widen storage_pct, bound inserts ... Dam 100602 reports 87798% storage on some days, overflowing NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/ MySQL tables) and every measure column is bounds-checked before insert so out-of-capacity junk becomes NULL instead of a batch-killing error. Rerunning the backfill repairs the days the overflow skipped.	2026-08-13 10:50:58 +07:00
grabowski	ba781465a9	feat: in-memory HII gap-fill in the ML data loader ... load_measurements() (DB path) patches missing station-hours from the hii_waterlevel mirror telemetry: exact mirrors (P.1/P.103/P.20/P.4A/P.67/ P.75/P.82/P.84/P.92) plus bias-corrected P.81 (+9,340 h). Per-station MSL->gauge offset is derived from >=168 h of series overlap, which reproduces the published offsets for exact mirrors and absorbs P.81's bias; P.76/P.77/P.85/P.87 HII twins are different physical sensors and stay excluded. Training and serving share the loader, so both sides see identical filled series; water_measurements is never written.	2026-08-13 10:29:54 +07:00
grabowski	6eafb353b1	feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow ... POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain.	2026-08-13 10:29:34 +07:00