CMU Water Center's Chiang Mai flood hazard map (CMFlood) replaces the older
municipal map. It is built on the October 2024 flood and has the Ping leave
its banks above P.1 4.20 m, with five stages: 4.30, 4.50, 4.70, 5.00 and
5.30 m (the 2024 record). The old map's seven stages (3.70-4.60 m) said the
city floods from 3.70 m; CMU calls that map out of date.
- P1_FLOOD_STAGES: the five CMU stages, discharge on RID's 2024 rating.
/forecast's P.1 `stages` now has five entries (same format). Thresholds
are unchanged: 4.20 m danger is the bank top, 3.70 m warning half a metre
below (as on CMU's gauge page). flood_config.json and the Rust parity
fixtures regenerated (only the stages changed).
- flood-zones-p1.geojson: CMU's KML (L1..L5) as five disjoint zones, zone k
floods first at stage k; scripts/build_flood_zones.py rebuilds it.
- Dashboard: CMU's colours and credit (map attribution); the stage grid follows
the number of stages; zone lookup for markers honours holes. The city
verdict uses statusOf's classes: flooding (over the banks) from 4.20 m,
high water from 3.70 m. Zones show by themselves when the river is over its
banks or stage 1 is >= 50 % likely. Alert, replay and verdict texts no
longer call 3.70 m city flooding (EN + TH).
- Docs: DATA_SOURCES 5.1 (CMFlood: the map, flood marks, poles, city gauges),
FLOOD_FORECASTING, DASHBOARD_REDESIGN, NOTIFICATIONS, README credits.
The P.1 chart's "24 h peak, as issued" line had a gap every night and a
dip at 21:00. Two causes:
- A run forecasts from the newest reading only. RID publishes 22:00..06:00
in one batch in the morning (and now and then one hour late), so those
hours were never forecast from. After each run the collector now
forecasts the hours of the last 48 h (two weeks on the first run after a
start) that have a RID reading but no model forecast, each as a run at
that hour would have: gauges, HII fill and dam state up to that hour.
Rain is fetched once, so a late hour's rain_fc24 partly holds rain that
has since fallen (as in training). Only missing (hour, station) pairs are
written. `ping-collector catch-up-forecasts --hours N` goes further back.
- Overnight each hourly run re-issues the 21:00 reading, after 6 h as the
heuristic, which overwrote the 21:00 model row. A heuristic row no longer
replaces a model row in forecast_history.
The dashboard draws only model rows on the issued line.
Checked on a local copy of 18 days of RID data: the catch-up of an hour
matches `ping-collector forecast --now <hour>`; 288 hours took 20 s
(debug build). Tests for both database rules.
ping-status sent "gauge feed stale" around midnight and "feed recovered"
around 07:00 on each of the last three nights (ntfy and, since phase 3, the
Matrix room). RID publishes no hourly report overnight: the 21:00 reading is
the last until the 22:00..06:00 hours arrive together (06:35, 06:55, 07:22 on
2026-10-17/18/19). The ThaiWater overlay was meant to bridge that, but
ThaiWater's copies of the RID gauges come from the same feed with the same
delay (P.1's mirror still has no value for 22:00-00:00 this morning), so it
does not.
Between 21:00 and FEED_NIGHT_UNTIL_HOUR (default 9, 0 = off) the feed of a
quiet river is no longer reported stale; one still without new readings at
09:00 is. While a station is at warning/danger or the P.1 outlook is on, the
3 h rule holds all night. A feed already stale stays stale through the night
(no false "recovered"), and a real recovery is announced at any hour. The
replay of the Python notifier runs with the window off. Docs no longer claim
that ThaiWater keeps the feed current at night.
Now a direct dependency, pip-audit sees it; it used to come in through uvicorn. The .env parser is unchanged: rust/collector/testdata/gen_envfile_cases.py regenerates the 614-case fixture byte-identical under 1.2.2, so envfile.rs still matches.
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.
Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.
Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.
- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
the stores lost their sqlite/mysql branches (and the 2026-08 column
migration). The database tests run against a throwaway schema from
TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
a checkout without touching its ownership): binary, units, uv sync, .env
defaults, removes the retired water-monitor.service and any old
--alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
and Rust doc comments updated to the Rust service.
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared
.env) ping-collector, after each cycle that brought newer RID or HII readings
and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the
rain forecast, runs every exported model (rust/forecast), stores
forecast_history and forecast_current, and sends the ntfy notifications.
The Python API then serves forecast_current as /forecast (computing in-process
only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor
notifies.
ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics,
texts, hysteresis, capacity guard and notification_state keys, checked word
for word on 751 messages generated by Python (testdata/gen_notify_cases.py).
The change: for stations whose ThaiWater twin is the same sensor, a reading
newer than RID's drives the level alerts (marked as ThaiWater telemetry), and
the feed counts as stale only when neither source has a reading for 3 h. RID
alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the
nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports.
Commands: `forecast [--json] [--save] [--now] [--rain-file]`,
`notify --dry-run`. scripts/forecast_parity.py compares Python and the
collector on the live database (same moment, same rain); locally 48/48 rows
identical. install_collector.sh: models export + that check on every install,
--cutover-forecast (refuses on a failed check) and --rollback-forecast.
Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140).
Forecast values are written to NUMERIC as shortest round-trip text, so they
round as psycopg2's do. DB test schemas get a counter (parallel tests collided
on Windows' coarse clock).
Phase 2a of docs/RUST_REWRITE.md. rust/forecast (ping-forecast) ports the
serving half of src/ml: the HII mirror gap fill, the hourly grid, every
feature (pandas' Kahan rolling sums included), the Mae Ngat dam frame,
HistGradientBoosting tree inference, the heuristic fallback, stale-threshold
handling and the P.1 flood-stage probabilities.
Hand-over from Python (src/ml/export.py):
- models/flood_<station>.json.gz next to each joblib bundle; train_all() writes
it, retrain.sh promotes it, `python -m src.ml.export` exports existing ones.
- rust/forecast/flood_config.json: thresholds, stages, leads and notify
constants, compiled in. tests/test_model_export.py fails while it differs
from the Python source.
Parity (rust/forecast/tests/parity.rs, fixtures from
testdata/gen_forecast_fixtures.py on the production code path): four
scenarios incl. the 2024 flood with RID lagging behind HII. Merged readings,
all features and all levels identical; with all 15 local models 377/384
probabilities bit-identical, the rest within 2 ulp (exp).
Python refactor without behaviour change: predict.forecasts_from_long, and
get_latest_forecasts/load_latest/serving_frame take now/rain for
reproducible comparisons; the HII query orders its rows.
dotenvy aborted on the production .env (a quote in an unquoted value makes it
read to the end of the file) and echoed the offending line, password
included. envfile.rs ports python-dotenv 1.1.1 (parser + ${VAR} interpolation,
no override): unparsable statements are skipped with a warning that names only
the line number. Verified against python-dotenv on 614 generated inputs
(testdata/). The .env is loaded before the tokio runtime starts.
install_collector.sh now runs the dry run through systemd-run with the unit's
User, WorkingDirectory and EnvironmentFile, i.e. exactly what the service sees.
rust/collector replaces the collection half of the Python web app: RID hourly
gauges (with the late-report semantics: store everything, retry every 10 min
while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain +
water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and
upsert semantics, one UNNEST statement per table, Bangkok wall-clock time
regardless of the server time zone.
Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and
instead re-forecast + notify when the newest RID reading in the database
advances (polled every FORECAST_POLL_SECONDS) and once per clock hour;
/scraping/status then serves the collector's status file; /scrape/trigger
answers 409.
Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against
PostgreSQL); live parity run of both collectors into two empty databases:
all 9 tables row-for-row identical (scripts/collector_parity.py); the Python
API served health, measurements, stats, status and model forecasts from a
database filled only by ping-collector. 177 Python tests pass.
reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which
rustls refuses.
Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service;
CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service,
release artifact).
grabowski