180d95c6bd11b53cb5b281baef5423a593723438
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 grabowski
|
cf10f2e4c8 |
feat(dashboard): console phase 4 - 2024 replay, simulation, replaces the old page
The console is now the dashboard at / (src/static/dashboard.html); the old page and console.html are gone. - Replay of the October 2024 flood (flood-2024-all.json): the page's clock, verdict, list, markers, flood zones and panel follow the replay; the panel charts the season on fixed axes with the 2024 model's 24 h peak. Hourly while the model alerts or P.1 is above 3.50 m, faster through quiet days; pause and drag. Rain, basin stations and the 5-minute refresh are off during the replay (today's data). - ?demo_level= and ?demo_rise= simulate P.1. A violet REPLAY / amber SIMULATION bar, the top bar's KPI and the P.1 panel say so; the mode is drawn from state.mode alone, so a refresh or language switch cannot relabel either as live. - History: rows without discharge_percent are ThaiWater's copy of the gauge (night hours RID publishes late), with a different rating (P.1: 26 against RID's 54 m3/s); their discharge is not drawn for gauges with RID discharge. The discharge strip has headroom so its label stays clear of the line. - Swap: the old page's title, description, Open Graph and JSON-LD tags; no noindex; tests/test_dashboard.py = console tests + the old checks that still apply (one P.1 level, SEO files) + a replay/simulation labelling test. - Docs: README screenshot (docs/img/dashboard.jpg, replacing two unused images), CLAUDE.md, CONTRIBUTING.md, FLOOD_FORECASTING.md (where the model shows), DASHBOARD_REDESIGN.md (phase 4 done, mode rules). Deploy needs a restart: the server reads dashboard.html at start. |
||
|
grabowski
|
301a629b42 |
feat(dashboard): the console, phase 1 of the redesign (preview at /static/console.html)
The user chose direction B of the redesign: a full-screen operations console.
It is built next to the current dashboard, served by ServeDir at
/static/console.html (noindex, links back to /), and replaces dashboard.html
once it covers every feature (docs/DASHBOARD_REDESIGN.md, phase 4).
Phase 1:
- top bar: the Chiang Mai verdict (P.1 level + 24 h forecast, the current
dashboard's rule), gauges reporting, highest 24 h flood chance, data age
("overnight pause" instead of "stale" during RID's nightly gap, as in the
notifier), EN/TH, light/dark (dark default), Get alerts
- gauge list: one status rule for list, map and panel (normal, watch within
0.5 m or >= 25 % chance, warning, danger, no data), margin to the warning
level, 72 h sparkline, 6 h change; filter; sort by river order, closest to
the warning level, rising; arrow keys
- map: Ping mainstem and tributaries, status markers with code + level,
layers Rain 24 h (on by default only with heavy rain) and Flood zones
(auto-shown while P.1 is at a stage), one legend
- station panel: level, change, discharge (last RID value when the newest
hour is level-only), warning/danger levels with margins, capacity, level
scale with now and 24 h peak, 72 h chart with the forecast peaks, forecast
table with the model and issue time, the P.1 inundation stages, data links
- alerts drawer with the ntfy topics; selection in the URL (#P.84); refresh
every 5 minutes
- Inter and Noto Sans Thai self-hosted (src/static/fonts, 75 KB, OFL), no
emoji, Thai first, Bangkok time
tests/test_console.py: routed fetch paths, EN/TH parity, translated aria
labels, no emoji, fonts and licences, noindex. dev_proxy.py serves
src/static with real content types (it sent HTML as octet-stream).
|
||
|
grabowski
|
7e642555b4 |
chore: remove the Python service; Python is the training stack only
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/) collects, forecasts, notifies (ntfy + Matrix) and serves the website/API; nothing in production ran the Python service code any more. Removed (~17k lines): the FastAPI app (web_api, schemas, health_check, metrics, postgres_history, forecast_history, thaiwater), the Python RID collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/ logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling (api_parity, collector_parity, e2e_notify, install_collector.sh, gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files, sql/, the release workflow and the docs for all of that. Their tests went with them. Kept in Python: src/ml (training, evaluation harness, export; predict.py as the reference the Rust forecaster is tested against), the HII and RID-dam clients/stores for the archive backfills (their live collectors removed), the backfill scripts, retrain.sh + timer, forecast_parity.py. - PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env; the stores lost their sqlite/mysql branches (and the 2026-08 column migration). The database tests run against a throwaway schema from TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB). - The notifier settings moved from src/notify.py into src/ml/export.py; flood_config.json regenerates byte-identical. - tests/test_dashboard.py checks every path the dashboard fetches against ping-collector's router. - Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil, schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv 1.1.1 explicit (it came in through uvicorn[standard]). - Formatting contract extended to scripts/ and tests/ (reformatted once). - scripts/install.sh is the one installer (fresh host or update, in place in a checkout without touching its ownership): binary, units, uv sync, .env defaults, removes the retired water-monitor.service and any old --alert-check cron job, dry run + forecast check, restart, readiness wait. - Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments and Rust doc comments updated to the Rust service. |
||
|
grabowski
|
779728e38b |
feat(web): the website and API in ping-collector; Matrix gets the ntfy transitions
Phase 3 of docs/RUST_REWRITE.md: rust/collector/src/web (axum) serves what
the FastAPI app (src/web_api.py) served, so one process collects, forecasts,
notifies and serves. With WEB_LISTEN=0.0.0.0:8000 it runs inside
`ping-collector run`; `ping-collector serve` runs it alone.
Same routes, parameters, JSON fields and field order, status codes,
{"detail": ...} errors and pydantic-style 422 lists, Cache-Control, gzip
(>500 B), CORS, Umami API tracking, the dashboard and static files, and the
stale-while-revalidate response caches (now with byte budgets: a history
response can be 10 MB). /docs is the same Swagger UI (+ /redoc); its OpenAPI
document is built from the route table, whose per-route parameter
declarations also validate the requests, so docs and behaviour cannot drift.
Python output conventions are kept: isoformat(), round(), numpy's pairwise
sums (catchment means), Python's summation order (skill metrics).
scripts/api_parity.py compares the two servers route by route on one
database (53 cases incl. errors, admin and static files; query values vary
per run so FastAPI's caches answer fresh). Locally: 53 identical; the
station CRUD writes byte-identical stations.json files; the skill/trend
logic matches on 9 station/horizon pairs with synthetic model versions.
Intended differences: /health counts the ThaiWater telemetry of the same
gauges while RID lags (as the notifier); /scraping/status adds the collector
version and last forecast run; an old stored forecast is served marked
X-Data-Stale instead of recomputed; /sensors/thaiwater is cached a minute;
.geojson is served as application/geo+json. Load (local, 16 clients): the
dashboard's routes 1375 vs 552 req/s, uncached history 714 vs 53 req/s;
25 MB resident when serving only.
Matrix (MATRIX_*): the notifier sends the room the transitions ntfy gets
(per station, outlook, feed; not ntfy's per-topic summaries), each once,
with its own matrix: keys in notification_state, seeded from ntfy's on first
use so nothing already announced repeats. Checked against a fake homeserver.
It replaces the `run.py --alert-check` cron job (repeated alerts every 15
min), which install_collector.sh now comments out (crontab backup kept).
install_collector.sh: --check-api (ping-collector serve on :8001 next to
FastAPI, api_parity.py), --cutover-api (refuses on a difference; sets
WEB_LISTEN, stops and disables water-monitor.service), --rollback-api;
--rollback-forecast and --rollback bring the Python app back as server.
ping-collector.service: MemoryMax 512M. Config::from_vars for tests; route
tests against a throwaway schema.
|
||
|
grabowski
|
6ae67dec63 |
feat(collector): forecast and ntfy in ping-collector; ThaiWater keeps alerts current at night
CI / Format & lint (push) Successful in 22s
CI / Test suite (push) Successful in 59s
Docs / Validate documentation (push) Successful in 25s
Rust / fmt, clippy, test, build (push) Successful in 10m2s
Security / Dependency vulnerabilities (push) Successful in 2m11s
Security / Static analysis (push) Successful in 22s
Security / License report (push) Successful in 1m37s
Phase 2b/2c of docs/RUST_REWRITE.md. With FORECAST_IN_COLLECTOR=true (shared .env) ping-collector, after each cycle that brought newer RID or HII readings and at least hourly: loads the last 336 h (RID + HII mirror fill), fetches the rain forecast, runs every exported model (rust/forecast), stores forecast_history and forecast_current, and sends the ntfy notifications. The Python API then serves forecast_current as /forecast (computing in-process only when it is older than FORECAST_MAX_AGE_SECONDS) and neither forecasts nor notifies. ntfy (collector/src/notify.rs): a port of src/notify.py with the same topics, texts, hysteresis, capacity guard and notification_state keys, checked word for word on 751 messages generated by Python (testdata/gen_notify_cases.py). The change: for stations whose ThaiWater twin is the same sensor, a reading newer than RID's drives the level alerts (marked as ThaiWater telemetry), and the feed counts as stale only when neither source has a reading for 3 h. RID alone lagging is reported once after RID_STALE_ALERT_HOURS (18). This ends the nightly "gauge feed stale" / "recovered" pairs caused by RID's late reports. Commands: `forecast [--json] [--save] [--now] [--rain-file]`, `notify --dry-run`. scripts/forecast_parity.py compares Python and the collector on the live database (same moment, same rain); locally 48/48 rows identical. install_collector.sh: models export + that check on every install, --cutover-forecast (refuses on a failed check) and --rollback-forecast. Trees are packed into 24-byte nodes: 70 MB peak with all 15 models (was 140). Forecast values are written to NUMERIC as shortest round-trip text, so they round as psycopg2's do. DB test schemas get a counter (parallel tests collided on Windows' coarse clock). |
||
|
grabowski
|
4e619aa2ae |
feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
rust/collector replaces the collection half of the Python web app: RID hourly gauges (with the late-report semantics: store everything, retry every 10 min while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain + water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and upsert semantics, one UNNEST statement per table, Bangkok wall-clock time regardless of the server time zone. Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and instead re-forecast + notify when the newest RID reading in the database advances (polled every FORECAST_POLL_SECONDS) and once per clock hour; /scraping/status then serves the collector's status file; /scrape/trigger answers 409. Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against PostgreSQL); live parity run of both collectors into two empty databases: all 9 tables row-for-row identical (scripts/collector_parity.py); the Python API served health, measurements, stats, status and model forecasts from a database filled only by ping-collector. 177 Python tests pass. reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which rustls refuses. Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service; CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service, release artifact). |
||
|
grabowski
|
2e19974fad |
docs: README describes the project that exists; CLAUDE.md for agents
The README was the original template: P.1 "in Nakhon Sawan", P.103 "in Bangkok", VictoriaMetrics as the recommended database, Docker/Grafana sections, github.com/your-username support links. Rewritten around what runs: sources, gap fill, the forecast and its 13 h result, the live API table, systemd deployment with the retrain timer, repository layout, real docs links, data-source credits. Other DB adapters are mentioned as supported-but-not-production. CLAUDE.md replaces the untracked Ruflo boilerplate with project rules: Python 3.11, format/test gates, Bangkok timestamps, harness-first model changes judged on lead, the RainUnavailableError guard, no git add -A. |