Commit Graph
21 Commits
Author SHA1 Message Date
grabowski 9bf904b23c feat(rid): screen one-hour spikes; refetch and redo-from to repair the store
CI / Format & lint (push) Successful in 15s
CI / Test suite (push) Successful in 20s
Docs / Validate documentation (push) Successful in 9s
Rust / fmt, clippy, test, build (push) Successful in 3m6s
RID's report sometimes carries a wrong value for one hour and corrects it
later. P.1 read 10.6 m at 2026-10-25 00:00 between 1.04 and 1.07 (RID now
says 1.06); the store kept it, because a report is re-fetched only while it
has holes, and the model issued a 10.8 m forecast for that hour.

- rid::judge / rid::screen: a reading that leaves the previous one (within
  3 h) by more than 1 m (2 m on the flashy tributaries P.76, P.77, P.81,
  P.82, P.84, P.85, P.87) while the next reading comes back near the
  previous level is a spike: not stored, a stored copy is deleted. Limbs
  and steps the next reading confirms are kept; the newest reading waits
  for the next one when it jumps more than twice the limit. On 2018-2026
  history: 16 spikes found (slipped decimals, sign flips, -5/-6/-9 m
  placeholders), no genuine change rejected.
- Every store goes through it (rid::store: run_cycle, fill-gaps, refetch).
- `ping-collector refetch FROM [TO]`: re-fetch reports regardless of holes
  (corrects revised readings, fills the discharge of level-only evening
  hours, removes stored spikes) and print the catch-up command when
  readings changed.
- `catch-up-forecasts --redo-from T`: delete the model forecasts issued
  from T on, then recompute them.
- Tests: judge/screen unit tests; a DB test of refetch, the live hold and
  the forecast deletion. Checked end to end against RID on a scratch DB.
- Docs: DATA_SOURCES (spikes, level-only hours), FLOOD_FORECASTING,
  rust/README.
2026-10-26 15:04:01 +07:00
grabowski 701bb87d34 feat(p1): city flood stages and zones from CMU Water Center's 2024 flood map
CI / Format & lint (push) Successful in 49s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 27s
Security / Dependency vulnerabilities (push) Successful in 1m5s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 45s
Rust / fmt, clippy, test, build (push) Successful in 3m42s
CMU Water Center's Chiang Mai flood hazard map (CMFlood) replaces the older
municipal map. It is built on the October 2024 flood and has the Ping leave
its banks above P.1 4.20 m, with five stages: 4.30, 4.50, 4.70, 5.00 and
5.30 m (the 2024 record). The old map's seven stages (3.70-4.60 m) said the
city floods from 3.70 m; CMU calls that map out of date.

- P1_FLOOD_STAGES: the five CMU stages, discharge on RID's 2024 rating.
  /forecast's P.1 `stages` now has five entries (same format). Thresholds
  are unchanged: 4.20 m danger is the bank top, 3.70 m warning half a metre
  below (as on CMU's gauge page). flood_config.json and the Rust parity
  fixtures regenerated (only the stages changed).
- flood-zones-p1.geojson: CMU's KML (L1..L5) as five disjoint zones, zone k
  floods first at stage k; scripts/build_flood_zones.py rebuilds it.
- Dashboard: CMU's colours and credit (map attribution); the stage grid follows
  the number of stages; zone lookup for markers honours holes. The city
  verdict uses statusOf's classes: flooding (over the banks) from 4.20 m,
  high water from 3.70 m. Zones show by themselves when the river is over its
  banks or stage 1 is >= 50 % likely. Alert, replay and verdict texts no
  longer call 3.70 m city flooding (EN + TH).
- Docs: DATA_SOURCES 5.1 (CMFlood: the map, flood marks, poles, city gauges),
  FLOOD_FORECASTING, DASHBOARD_REDESIGN, NOTIFICATIONS, README credits.
2026-10-26 14:19:52 +07:00
grabowski 180d95c6bd fix(forecast): no nightly gaps in the issued model forecasts
CI / Format & lint (push) Successful in 28s
CI / Test suite (push) Successful in 1m4s
Docs / Validate documentation (push) Successful in 17s
Rust / fmt, clippy, test, build (push) Successful in 13m1s
The P.1 chart's "24 h peak, as issued" line had a gap every night and a
dip at 21:00. Two causes:

- A run forecasts from the newest reading only. RID publishes 22:00..06:00
  in one batch in the morning (and now and then one hour late), so those
  hours were never forecast from. After each run the collector now
  forecasts the hours of the last 48 h (two weeks on the first run after a
  start) that have a RID reading but no model forecast, each as a run at
  that hour would have: gauges, HII fill and dam state up to that hour.
  Rain is fetched once, so a late hour's rain_fc24 partly holds rain that
  has since fallen (as in training). Only missing (hour, station) pairs are
  written. `ping-collector catch-up-forecasts --hours N` goes further back.
- Overnight each hourly run re-issues the 21:00 reading, after 6 h as the
  heuristic, which overwrote the 21:00 model row. A heuristic row no longer
  replaces a model row in forecast_history.

The dashboard draws only model rows on the issued line.

Checked on a local copy of 18 days of RID data: the catch-up of an hour
matches `ping-collector forecast --now <hour>`; 288 hours took 20 s
(debug build). Tests for both database rules.
2026-10-20 15:21:48 +07:00
grabowski cf10f2e4c8 feat(dashboard): console phase 4 - 2024 replay, simulation, replaces the old page
CI / Format & lint (push) Successful in 26s
CI / Test suite (push) Successful in 1m15s
Docs / Validate documentation (push) Successful in 12s
The console is now the dashboard at / (src/static/dashboard.html); the old
page and console.html are gone.

- Replay of the October 2024 flood (flood-2024-all.json): the page's clock,
  verdict, list, markers, flood zones and panel follow the replay; the panel
  charts the season on fixed axes with the 2024 model's 24 h peak. Hourly
  while the model alerts or P.1 is above 3.50 m, faster through quiet days;
  pause and drag. Rain, basin stations and the 5-minute refresh are off
  during the replay (today's data).
- ?demo_level= and ?demo_rise= simulate P.1. A violet REPLAY / amber
  SIMULATION bar, the top bar's KPI and the P.1 panel say so; the mode is
  drawn from state.mode alone, so a refresh or language switch cannot
  relabel either as live.
- History: rows without discharge_percent are ThaiWater's copy of the gauge
  (night hours RID publishes late), with a different rating (P.1: 26 against
  RID's 54 m3/s); their discharge is not drawn for gauges with RID discharge.
  The discharge strip has headroom so its label stays clear of the line.
- Swap: the old page's title, description, Open Graph and JSON-LD tags; no
  noindex; tests/test_dashboard.py = console tests + the old checks that
  still apply (one P.1 level, SEO files) + a replay/simulation labelling test.
- Docs: README screenshot (docs/img/dashboard.jpg, replacing two unused
  images), CLAUDE.md, CONTRIBUTING.md, FLOOD_FORECASTING.md (where the model
  shows), DASHBOARD_REDESIGN.md (phase 4 done, mode rules).

Deploy needs a restart: the server reads dashboard.html at start.
2026-10-20 13:47:15 +07:00
grabowski 7e642555b4 chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
2026-10-18 17:15:07 +07:00
grabowski 6ae67dec63 feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared
.env) ping-collector, after each cycle that brought newer RID or HII readings
and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the
rain forecast, runs every exported model (rust/forecast), stores
forecast_history and forecast_current, and sends the ntfy notifications.
The Python API then serves forecast_current as /forecast (computing in-process
only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor
notifies.

ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics,
texts, hysteresis, capacity guard and notification_state keys, checked word
for word on 751 messages generated by Python (testdata/gen_notify_cases.py).
The change: for stations whose ThaiWater twin is the same sensor, a reading
newer than RID's drives the level alerts (marked as ThaiWater telemetry), and
the feed counts as stale only when neither source has a reading for 3 h. RID
alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the
nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports.

Commands: `forecast [--json] [--save] [--now] [--rain-file]`,
`notify --dry-run`. scripts/forecast_parity.py compares Python and the
collector on the live database (same moment, same rain); locally 48/48 rows
identical. install_collector.sh: models export + that check on every install,
--cutover-forecast (refuses on a failed check) and --rollback-forecast.

Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140).
Forecast values are written to NUMERIC as shortest round-trip text, so they
round as psycopg2's do. DB test schemas get a counter (parallel tests collided
on Windows' coarse clock).
2026-10-18 14:43:39 +07:00
grabowski 6d4beae65a fix(ml): serve the forecast rain the model was trained with
CI / Format & lint (push) Successful in 24s
CI / Test suite (push) Successful in 48s
Docs / Validate documentation (push) Successful in 24s
Security / Dependency vulnerabilities (push) Successful in 2m20s
Security / Static analysis (push) Successful in 21s
Security / License report (push) Successful in 1m40s
build_features cut the rain series to the gauge grid before the forward sum,
and the grid ends at the newest reading, so the row every live forecast is
made from had rain_fc24 = NaN. Training and all evaluations had the real
Open-Meteo forecast there. The forward sum now runs over the grid extended by
24 h; earlier rows are unchanged.

Harness (new opt-in variant rise_rain_fc24_missing = rise_rain with
rain_fc24 blanked on test rows, five folds, --from-cache): P.1 and P.103
nearly unaffected, P.21 false alarms 10 -> 5 and MAE 0.232 -> 0.215 with the
fix. Write-up in docs/FLOOD_FORECASTING.md section 5; results in
models/eval_2026-10-18_fc24.json.

Also: HII mirror rows in one hour now resolve to the latest reading via a
stable sort on the raw time (was an unstable sort on the floored hour, i.e.
row-order dependent). Needed for an exact Rust port.
2026-10-18 14:07:56 +07:00
grabowski ce08312c0f docs: public dashboard URL (water.buildfor.life) replaces the Tailscale IP everywhere
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 32s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 16s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
2026-09-11 22:38:13 +02:00
grabowski d621aa9ce7 eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 41s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Documentation / Validate Documentation (push) Failing after 16s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw,
rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all
opt-in, plus --from-cache for reproducible offline reruns. Results in
models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md:

- quantile point prediction: better MAE, worse first-alert lead at 5 of
  11 events (P.103 2022-08-14 +6h -> +1h) -> rejected
- quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads
- rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1
  -> deferred to after the 2026 season

src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in
the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a
training feature (table exists only since 2026-08-11, no archive); exposed
at GET /api/hii/rainfall/catchment so the two sources' agreement is on
record by the time a fold can test it.

data._read_cache now skips non-station files in models/cache/ (the shared
dir also holds rain_openmeteo / dam_* caches, which crashed the reader).
scripts/summarize_eval.py prints per-variant lead/peak-error tables.
2026-09-11 21:55:37 +02:00
grabowski 764764e07e feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote
train_all() now raises RainUnavailableError when use_rain=True and the
Open-Meteo history cannot be loaded, instead of logging a warning and
writing gauge-only (v2) bundles over the deployed v3 set -- which is what
the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit
way to get v2. CLI exits 2 with a one-line error. Three tests cover the
guard, the opt-out, and the v3 happy path.

scripts/retrain.sh trains into models/.staging, refuses to promote unless
metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles
into place (previous generation kept in models/.previous). No API restart:
predict.py reloads by mtime on the hourly precompute.

water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent,
OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now
does `uv sync` into .venv (one env rule; removes a stale venv/) and enables
the timer. water-monitor.service in the repo matched neither the deployed
unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).
2026-09-11 21:37:11 +02:00
grabowski 5dc5850df6 docs: source sweep — P.75 already is the Mae Ngat release signal
Verified from a Thai ISP that lsim.rid.go.th is unreachable (not
geo-blocked), then swept for any better-than-daily Mae Ngat source.

Key finding: P.75 sits 3.8 km below the dam, reports hourly, and has
been a model feature since v1 — the model has always read the dam's
actual outflow, hourly and directly. That is the likelier reason the
daily reservoir table adds nothing, beyond its publication lag.

Intraday reservoir feeds do exist and are open (bigdata-api.rid.go.th
SWOC, ThaiWater ridhydro_TUP.16 at the dam) but are snapshot-only with
no archive, so they cannot retrain against past events; the HII
collector accumulates them from 2026-08-11 for a post-monsoon revisit.
Also documents api/dam (whole per-dam range in one request, vs the
~2,900-request per-day loop) and several cross-check mirrors.
2026-08-13 21:58:47 +07:00
grabowski 70e4da07a0 docs: lsim.rid.go.th is unreachable, not geo-blocked
Probed from a Thai consumer ISP (AIS Fibre): DNS resolves but ICMP and
ports 80/443/8080 are filtered, while app.rid.go.th answers in 0.27 s
over the same connection. Correct the earlier note that assumed the
timeout was a foreign-network block, and stop pointing the dam-feature
follow-up at a host that cannot be reached.
2026-08-13 21:00:24 +07:00
grabowski 28b62e5a36 feat: Mae Ngat dam features — built, evaluated, defaulted OFF
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 27s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
src/ml/dam.py loads rid_reservoir_daily into a leakage-safe hourly frame
(daily row visible from 07:00 its own date, ffill capped at 48 h) and is
plumbed through features/train/predict/evaluate exactly like rain, gated
to the six mainstem stations below the Mae Ngat confluence.

The experiment concludes as a documented NEGATIVE result: on the 2024
record-flood backtest every dam-feature subset costs 1-3 h of first-alert
lead (13h -> 10-12h) for <=3 cm of peak-error gain, because the daily RID
report lags up to 31 h and describes yesterday's benign absorbing
reservoir during fast onset. Features therefore default OFF (--dam
opt-in on the training and backtest CLIs; rise_rain_dam/rise_dam harness
variants, excluded from the default variant set). The ablation also
isolated the HII gap-fill as lead-neutral: the acceptance gate holds at
13 h with fill enabled, and docs/img charts are regenerated with the
shipping configuration. Full table in docs/FLOOD_FORECASTING.md §5.

Review-swarm fixes: evaluate.py skips variants whose feature family is
absent instead of crashing the run; --dam forwards --db-url and warns
loudly when no dam history loads; an empty DB result can no longer wipe
a good dam cache; run-level metrics version claims v4 only when a dam
station is actually in the set.
2026-08-13 20:42:21 +07:00
grabowski df0ae8cda3 feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
The rolling-origin harness (models/eval_rain.json) showed catchment rain
halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and
extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at
P.103). Ported: train_all loads the catchment-mean series (use_rain /
--no-rain to opt out; without it bundles train as v2), predict fetches
live rain hourly and passes an empty series on failure so rain-trained
bundles serve with NaN features instead of tripping the feature guard,
and the leader worker persists hourly per-point + catchment-mean rows to
a new openmeteo_rain table.

Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING
(alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h
acceptance gate PASSES for the first time. Journey on that crossing:
v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its
artifact +46h latch for a calibrated +2h with zero false alarms. P.1
MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout.
2026-08-12 17:05:01 +07:00
grabowski 21e9d2e114 feat: hgb-v2 — regression heads predict rise, recovering flood warning lead
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 28s
Documentation / Documentation Summary (push) Successful in 2s
Rolling-origin evaluation (5 monsoon folds x 4 variants, P.1 + P.103;
results in models/eval_variants.json) showed the absolute-level target
alerting AT the crossing on essentially every event, while the rise
target (future max - current level, level added back at serving) gives
+6h on the hard 2024 crossings, +45h in 2025, fewer false alarms than
weighted/quantile variants, and ~11% better MAE. Weighted and quantile
variants rejected: more false alarms, no Brier-score calibration gain.

Ported to production: train.py fits rise in both eval and refit passes
(sigma/metrics computed in absolute space), bundles stamped hgb-v2 with
regression_target='rise', predict.py adds the level back for v2 and
stays compatible with v1 bundles, backtest_render.py mirrors the same
math. Regenerated backtest charts: 2024 first alert 11:00 24 Sep (6h
BEFORE the 17:00 crossing, was 18h after), 2025 alert 45h ahead, and
the record-peak underprediction is gone (rise models can exceed the
training max). The >=12h acceptance gate still fails honestly at +6h —
closing that needs rainfall inputs. New P.1 MAE 5.0/7.2/9.4 cm at
6/12/24h; docs updated throughout.
2026-08-12 15:43:29 +07:00
grabowski 0005f7dce1 feat: codified backtests, honest docs, belt-and-braces serving, perf fixes
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the
flood backtests, now reproducible via scripts/backtest_render.py (renders
the three docs/img charts and gates on a >=12h 2024 first-alert lead —
currently failing by design and documented as such).

Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing
was 24 Sep 17:00 (8h earlier than recorded; confirmed against the
independent HII sensor), the historical 24h-warning claim was partly a
missing-data artifact, and retrained warn classifiers collapse on the
filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves
(11.3 -> 10.5 cm). Serving therefore becomes max(classifier,
sigmoid(regression)) so alerting is never worse than the regression path;
metrics table, head-gating tiers, honest-limits and runbook expectations
all updated to the current model (hgb-v1+d2d0e65).

Perf, from Locust load testing (scripts/locustfile.py + load_test.py):
single-flight lock around /forecast inference (concurrent cache misses
previously each ran ~18s inference and starved the shared thread pool;
200-user run after: 105 rps, 0.01% errors), and /measurements/latest +
/health moved off the event loop (synchronous DB/network calls in async
handlers were stalling every request under load).
2026-08-12 10:46:00 +07:00
grabowski 27fa292e09 docs: September 2025 flood render - deployed config, 24 h warning, peak within 7 cm
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 16s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
Hour-by-hour chart of the 25-28 Sep 2025 event as forecast by the exact
deployed configuration (trained through 2024, event unseen): first alert
26 Sep 18:00, flooding began 27 Sep 18:00 (24 h lead), predicted peak
4.00 m vs actual 3.93 m. The near-miss 3.51 m crest on 26 Sep correctly
never alerted.
2026-08-10 18:35:55 +07:00
grabowski f0183faa62 feat: discharge-driven river animation speed, replay stat tiles, hourly doc render
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Validate Documentation (push) Failing after 10s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 20s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
- river dash animation now flows at a speed continuously derived from
  each segment's discharge (period 260/(Q+45) s, clamped 0.5-5.5 s) via
  inline per-path animation-duration, so it updates live and per-frame
  during the replay (CSS speed classes removed - setStyle cannot change
  classes)
- the replay drives the Combined discharge and Strongest flow stat
  tiles each frame (marked '2024 replay'), restored on finish
- docs: hour-by-hour detection detail render (22-28 Sep 2024) showing
  the model alert at 24 Sep 01:00, flooding at 25 Sep 01:00, and the
  24 h warning between them; embedded with commentary
- Matrix alerts now link to https://water.buildfor.life/ (override via
  ALERT_DASHBOARD_URL), replacing the Grafana public dashboard link
2026-08-10 18:29:01 +07:00
grabowski 6112c681a7 docs: render of actual vs predicted through the 2024 flood; adaptive replay speed
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
docs/img/backtest-2024-p1.png: observed P.1 level vs the 24h-ahead
predicted peak issued at each hour by a model trained only on pre-flood
data, with the warning-probability panel below (first alert 24 Sep
01:00, 24 h before flooding began). Embedded in FLOOD_FORECASTING.md's
headline-validation section with an honest reading, including the
~0.4 m peak under-prediction.

Replay pacing is now adaptive: 4 h/frame through quiet days, 2 h when
risk is elevated, 1 h (hour-by-hour) while the model is alerting or the
river is near/above flood stage - so viewers can watch the detection
sequence unfold.
2026-08-10 18:10:48 +07:00
grabowski e4d5d274f0 feat: per-station flood thresholds and Chiang Mai inundation stages for P.1
Replace the network-wide (3.0, 4.5) m thresholds with per-station values
calibrated from the DB's discharge_percent (RID % of channel capacity):
warning = median level at 75-85% capacity, danger = median at 95-105%.
Fixes P.103 over-alerting (bank-full ~6.75 m, not 4.5) and P.67
under-alerting (overflow ~2.9 m). Requires a retrain to take effect in
the classifier heads.

P.1 uses the official Chiang Mai municipal inundation map instead:
warning 3.70 m (stage 1, city flooding begins), danger 4.20 m (stage 5),
with the full 7-stage table (3.70-4.60 m + discharge) in
features.P1_FLOOD_STAGES. Forecast rows for P.1 now include per-stage
exceedance probabilities computed from the regression head + calibration
sigma - available immediately without retraining.

Dashboard: "Chiang Mai city flood outlook" block above the forecast grid
(predicted peak + 7 stage-probability chips) and a toggleable
georeferenced overlay of the official flood-zone map
(static/flood-zones-p1.jpg, bounds tunable in FLOOD_ZONE_BOUNDS).
2026-08-10 15:35:00 +07:00
grabowski 4358d52d55 feat: ML flood-event forecasting from 8 years of gauge history
Security & Dependency Updates / Dependency Security Scan (push) Successful in 1m8s
Security & Dependency Updates / License Compliance (push) Successful in 25s
Security & Dependency Updates / Check for Dependency Updates (push) Successful in 20s
Security & Dependency Updates / Code Quality Metrics (push) Successful in 17s
Security & Dependency Updates / Security Summary (push) Successful in 9s
Add src/ml/ package predicting, per station and per 6/12/24 h horizon,
the probability of exceeding warning (3.0 m) and danger (4.5 m) levels
plus expected peak level, trained on the 592k-row PostgreSQL history:

- features.py: hourly grid with coverage gating and no future leakage;
  upstream stations enter at empirically measured travel-time lags
  (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded
  (it encodes the scrape schedule, not hydrology)
- train.py: HistGradientBoosting regression + warn/danger classifier
  heads per station x horizon, >=30-positives gate with calibrated
  sigmoid-on-regression fallback, strict temporal splits, per-event
  lead-time evaluation; guards against sklearn 1.9.0 crash on
  degenerate feature columns
- predict.py: bundle loading with feature-name checks, heuristic
  fallback tier, get_latest_forecasts() for the API; raises when no
  models are trained so the endpoint 503s instead of serving
  persistence output as forecasts
- data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API
  fallback (flagged: that path backfills synthetic discharge), csv.gz
  cache
- /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel
  (hidden until models exist)
- docs/FLOOD_FORECASTING.md: full system doc with measured deployment
  numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and
  retraining policy

Validation: out-of-sample backtest of the record 2024 flood season
(train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26
test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate.

Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km
out of basin; now 18.6936 N 99.0819 E per RID station page), pin
scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB,
train on the server via scripts/train_flood_model.py).
2026-08-10 12:49:47 +07:00