6eafb353b107be32c26a51ec5de56a4148a22e49
37
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6eafb353b1 |
feat: RID large-dam daily collector — Mae Ngat storage/inflow/outflow
POST app.rid.go.th/reservoir/api/dams (open, archive >=2009) collected hourly into rid_dams + rid_reservoir_daily; backfill script fetches only missing days so reruns repair holes and are safe alongside the live collector. /api/stats counts the new table via an engine fallback that works when HII collection is disabled. Mae Ngat (DAM_ID 200103) hit 113% usable capacity with ~19 MCM/day inflow in the Oct 2024 flood — candidate features for the next retrain. |
||
|
|
811af1625b |
feat: dashboard shows the live model version; DB stats count openmeteo_rain
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Documentation Summary (push) Successful in 2s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
The flood-outlook subtitle now reads '... · model hgb-v3+<sha> (trained <date>)' from the forecast rows, so the deployed model is always visible on the page. /api/stats adds openmeteo_rain_measurements (guarded — the table may not exist on older DBs) to the whole-DB total, and the Total-datapoints tile note gains a 'forecast rain' entry. |
||
|
|
df0ae8cda3 |
feat: hgb-v3 — Open-Meteo rain features clear the 12h warning gate
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
The rolling-origin harness (models/eval_rain.json) showed catchment rain halving flood-year Brier scores, cutting flood-regime MAE 20-40%, and extending the hard 2024 leads (+6h -> +11h at P.1, +10h -> +19h at P.103). Ported: train_all loads the catchment-mean series (use_rain / --no-rain to opt out; without it bundles train as v2), predict fetches live rain hourly and passes an empty series on failure so rain-trained bundles serve with NaN features instead of tripping the feature guard, and the leader worker persists hourly per-point + catchment-mean rows to a new openmeteo_rain table. Regenerated backtest: the 2024 record flood now gets a 13-HOUR WARNING (alert 04:00 vs 17:00 crossing, river at 2.9m at alert time) — the >=12h acceptance gate PASSES for the first time. Journey on that crossing: v1 -18h, v2 +6h, v3 +13h. The marginal 2025 double-crest trades its artifact +46h latch for a calibrated +2h with zero false alarms. P.1 MAE 4.9/7.2/8.7 cm at 6/12/24h. Docs updated throughout. |
||
|
|
98023243af |
feat: forecast precompute + issued-forecast archive + dashboard overlay
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Build Sphinx Documentation (push) Successful in 14s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
The collection-leader worker now precomputes forecasts after every
scrape cycle: primes the /forecast cache (users never trigger the
multi-second inference — its TTL rises to 4500s so the hourly refresh
always wins) and persists every issued forecast to a new
forecast_history table keyed by (as_of, station, horizon) with
predicted max level, warn/danger probabilities, current level, and
model_version. This is the operational record the backtests lacked —
predicted-vs-actual becomes a simple join instead of retraining
historical models.
GET /api/forecast/history/{station} serves the archive (hours or
start/end + horizon filters, 5-min edge cache), and the station history
chart overlays 'Model 24 h peak (as issued)' as a dashed violet line
once data accumulates.
|
||
|
|
731f10910e |
perf: stale-while-revalidate caching, bigger executor, cheap DB health probe
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 13s
Documentation / Generate API Documentation (push) Successful in 11s
Third on-box run: p50s healthy everywhere but tails at 60s — when a TTL expired under load, every concurrent miss parked an executor thread on the single-flight lock, exhausting the ~12-thread pool and timing out unrelated endpoints. All cached endpoints (latest, HII, stats, health) now use _cached_swr: fresh -> inline; expired-but-present -> the stale value is returned immediately and ONE background task refreshes; only a cold key (first request since startup) waits. Plus: dedicated ThreadPoolExecutor (EXECUTOR_THREADS, default 48) replaces the cpu+4 default, and DatabaseHealthCheck no longer re-runs the CREATE TABLE DDL suite on every probe (connect only when no live engine). |
||
|
|
96fedb3991 |
perf: single-flight /api/stats; fast-fail health probe
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Generate API Documentation (push) Successful in 10s
Documentation / Build Sphinx Documentation (push) Successful in 16s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 4s
Documentation / Validate Documentation (push) Failing after 8s
The on-box rerun after the inline fast path showed the cached endpoints healthy (latest p50 72-160ms, HII ~100ms) but /api/stats at 81% timeouts and /health at 33%: stats had a cache but NO single-flight, so every concurrent miss ran the heavy whole-DB counts (~1.7M rows) in parallel, re-jamming Postgres and the executor — which also dragged uncached history windows into 60s timeouts. /api/stats now computes through _ttl_cached_stale (one computation per 5min TTL, stale served on failure). The health API probe fails fast (5s instead of 30s) and its cache TTL rises to 30s, so a slow upstream can no longer pin executor threads longer than the cache lifetime. |
||
|
|
0b14e394ad |
perf: Cache-Control headers so browsers and edge caches absorb traffic
Bandwidth diagnosis: iperf shows the proxy<->API VPN link at 65-78 Mbit/s and a fast client pulls 15.7 MB/s from a CDN but only ~66 KB/s from the site — the Caddy host's public uplink is the scarce resource. Max-ages mirror the server cache TTLs (latest 30s, HII 60s, forecast 120s, history/stats/stations 300s, static 1h, dashboard HTML 120s), so repeat views come from browser cache and an edge proxy (e.g. Cloudflare free tier in front of Caddy) can serve most traffic without touching the uplink. |
||
|
|
d9c65bcf0c |
perf: inline cache fast path; cache /health checks
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Build Sphinx Documentation (push) Successful in 14s
Documentation / Generate API Documentation (push) Successful in 8s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 3s
The on-box load test exposed the real stall: cache HITS were dispatched through asyncio.to_thread, so under load a microsecond lookup queued ~11s behind slow work in the ~8-thread default executor (endpoints answered inline — /forecast 4ms, /api/stats 2ms — while every to_thread endpoint sat at p50 8-17s). Handlers now check TTL caches inline in the async path via _cache_fresh() and only pay for a thread on a miss. /health results are cached for HEALTH_CACHE_TTL_SECONDS (10): its external RID-API probe plus DB query were occupying executor threads on every hit, which is what jammed the pool in the first place. |
||
|
|
1ec5cfb4df |
perf: gzip responses; multi-worker serving with single collection leader
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
GZipMiddleware (min 500 bytes) compresses the dashboard HTML ~4x and station JSON up to ~100x, end-to-end through the Caddy TLS terminator — production load testing showed the deployment is bandwidth-bound once the response caches hit, so compression is the capacity lever. WEB_WORKERS (default 2) runs uvicorn multi-process via the app import string. Every worker executes the lifespan, so a localhost lock port (COLLECTION_LEADER_PORT, default 8901) elects exactly one background-collection leader per machine — RID/HII polling stays once-per-cycle instead of once-per-worker; the lock releases with the process. Locust clients now send Accept-Encoding so future runs measure compressed transfer, as browsers do. |
||
|
|
039af8caac |
perf: cache /measurements/latest; stale-on-error fallback for cached endpoints
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Documentation Summary (push) Successful in 2s
The response caches (HII feeds, and now /measurements/latest at 45s TTL — the endpoint every dashboard poll hits) share one helper, _ttl_cached_stale: single-flight per key, empty results never cached, and expired entries kept as a fallback. If a recompute fails (DB unreachable), the last good response is served with an X-Data-Stale: true header instead of a 5xx — during an outage the dashboard keeps showing the last real readings with their honest timestamps. TTLs: LATEST_CACHE_TTL_SECONDS (45), HII_CACHE_TTL_SECONDS (120). |
||
|
|
d27ca8bf40 |
perf: 120s TTL cache with single-flight on /api/hii/*/latest
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 59s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
Documentation / Generate API Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 24s
Documentation / Validate Documentation (push) Failing after 15s
Documentation / Build Sphinx Documentation (push) Successful in 26s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 2s
Documentation / Documentation Summary (push) Successful in 5s
These endpoints ran an uncached latest-per-station aggregation on every request (p50 0.6-0.9s under load, ~30% of the traffic mix) for data the collector refreshes hourly. Responses are now cached per (feed, hours) for HII_CACHE_TTL_SECONDS (default 120) with a per-feed compute lock so a cache miss runs one query regardless of concurrency; empty results are never cached so recovery is immediate. Cached hits measure ~8ms. Cache is process-local behind a single helper — the seam where a shared backend (Redis) would slot in if the deployment ever moves to multiple workers; not warranted at one. |
||
|
|
0005f7dce1 |
feat: codified backtests, honest docs, belt-and-braces serving, perf fixes
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Validate Documentation (push) Failing after 7s
Documentation / Generate API Documentation (push) Successful in 8s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Retrained on the gap-filled DB (592k -> 976k rows) and re-examined the flood backtests, now reproducible via scripts/backtest_render.py (renders the three docs/img charts and gates on a >=12h 2024 first-alert lead — currently failing by design and documented as such). Findings, all documented in FLOOD_FORECASTING.md: the true 2024 crossing was 24 Sep 17:00 (8h earlier than recorded; confirmed against the independent HII sensor), the historical 24h-warning claim was partly a missing-data artifact, and retrained warn classifiers collapse on the filled grid (P.1 24h PR-AUC 0.900 -> 0.288) while regression MAE improves (11.3 -> 10.5 cm). Serving therefore becomes max(classifier, sigmoid(regression)) so alerting is never worse than the regression path; metrics table, head-gating tiers, honest-limits and runbook expectations all updated to the current model (hgb-v1+d2d0e65). Perf, from Locust load testing (scripts/locustfile.py + load_test.py): single-flight lock around /forecast inference (concurrent cache misses previously each ran ~18s inference and starved the shared thread pool; 200-user run after: 105 rps, 0.01% errors), and /measurements/latest + /health moved off the event loop (synchronous DB/network calls in async handlers were stalling every request under load). |
||
|
|
b89c7e1915 |
feat: flood-verdict banner, violet rain palette, per-station bands, chip contrast, Umami
Documentation / Build Sphinx Documentation (push) Successful in 27s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Validate Documentation (push) Failing after 11s
Documentation / Generate API Documentation (push) Successful in 11s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 1m0s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 30s
UI/UX review backlog round 2 plus analytics: - at-a-glance verdict banner (role=status, aria-live) above the tiles: ok/watch/danger from P.1 level vs the official 3.70 m stage and the 24 h forecast, with a not-an-official-warning disclaimer + ThaiWater link (also added to the forecast card) - rainfall dots switch to a violet sequential ramp so they can never be confused with flow-status colors; legend gains 'Other markers' rows (grey no-data gauges, + bank-height sensors) - history-chart flood bands now use each station's own /forecast thresholds (P.1 official fallback) instead of hardcoded 3.0/4.5 - chipStyle(): contrast-safe text on all code/risk/stage chips (dark ink on amber/ochre, darkened fills for white ink); risk/stage ramps drop the blue step for a monotonic green-ochre-amber-red escalation - plain-language tooltips on the flow and capacity tiles - Umami: client script tag plus opt-out server-side middleware posting api-request events (fire-and-forget, 3 s timeout, never blocks) |
||
|
|
09c1c84153 |
feat: search-engine indexing — robots.txt, sitemap.xml, llms.txt, SEO meta
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 16s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 13s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Documentation Summary (push) Successful in 2s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
robots.txt (sitemap pointer, /metrics and /scrape/ disallowed), sitemap.xml (/ and /docs), and llms.txt (site + API summary for LLM crawlers) served at the domain root. Dashboard head gains title with keywords, meta description, canonical, Open Graph/Twitter tags, schema.org WebApplication JSON-LD, theme-color, and an inline SVG favicon (also silences the /favicon.ico 404). |
||
|
|
e27d418a1a |
feat: station search in side panel; DB stats cover HII tables
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 38s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 10s
Documentation / Build Sphinx Documentation (push) Successful in 16s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
A search box above the station lists filters both the RID flow list and the basin-station list by code, name (Thai included), or river, and persists across refreshes. /api/stats now reports whole-DB totals — RID + hii_rainfall + hii_waterlevel counts, combined station count, and a date range spanning all sources — with per-source breakdown fields the stats strip shows as tile notes. Sensor section renamed to 'Additional basin stations · ThaiWater/HII'. |
||
|
|
9516857d44 |
feat: date-range picker on the station history panel; drop PostgreSQL naming
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 14s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 10s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 24s
Documentation / Build Sphinx Documentation (push) Successful in 19s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
/measurements/history/{code} accepts optional start/end date params that
override the hours window (end date inclusive). The dashboard history
card gains from/to date inputs beside the quick-range dropdown —
explicit dates win, changing the dropdown clears them. All user-facing
'PostgreSQL history' labels renamed to 'Station history'.
|
||
|
|
4f3f19f6db |
feat: rainfall + HII water-level layers on the dashboard
Documentation / Build Sphinx Documentation (push) Successful in 1m3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 1m4s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Generate API Documentation (push) Successful in 23s
Documentation / Validate Documentation (push) Failing after 18s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Failing after 11m16s
New endpoints GET /api/hii/rainfall/latest and /api/hii/waterlevel/latest serve the latest per-station rows from the hii_* tables. The map gains a TWA-style rain layer: circle markers binned by TMD 24-h classes (blues for light/moderate/heavy, site amber/red for very-heavy/extreme), with a legend block and show/hide toggle. The ThaiWater sensor panel now prefers the DB-backed HII feed (no API key required, 97+ stations, ThaiWater storage-percent situation colors, gauge-datum conversion via offset_msl) and falls back to the live /sensors/thaiwater passthrough. |
||
|
|
1845ef7203 |
feat: hourly HII/ThaiWater rainfall + backup water-level collection
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 37s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 10s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
Poll the open api-v3.thaiwater.net public endpoints (rain_24h, waterlevel_load), filter to the Ping basin, and persist to new hii_rain_stations/hii_rainfall and hii_wl_stations/hii_waterlevel tables (auto-created; sqlite/postgresql/mysql). Water levels stay in their own tables since HII reports m MSL from a different station set; rid_code maps mirrors like ridhydro_P.1 to P.1 and offset_msl converts MSL to gauge datum. Runs every scraping cycle in web-api and continuous modes (hourly cadence even during 1-minute RID retry), one-shot via --collect-hii; docs/DATA_SOURCES.md catalogs all probed endpoints. |
||
|
|
7befc82ff5 |
feat: database stats on the dashboard via GET /api/stats
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 23s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 14s
Documentation / Validate Documentation (push) Failing after 8s
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
New SQLAdapter.get_database_stats() aggregates totals, station count, date range, and hourly-slot coverage in one query per dialect. The endpoint follows the existing 503-guard/to_thread/TTL-cache pattern; the dashboard gains a five-tile stats strip on the existing refresh cadence. Coverage denominator is hour-truncated so off-hour endpoints cannot push it past 100%; MySQL slot expression avoids % characters that would break under pyformat bind interpolation. |
||
|
|
410faeddd5 |
fix: protect admin API endpoints; stop rejecting flood-peak measurements
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 12s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 10s
Documentation / Build Sphinx Documentation (push) Successful in 16s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 0s
Documentation / Documentation Summary (push) Successful in 2s
Security: POST/PUT/DELETE /stations, POST /scrape/trigger and GET /config now require an X-API-Key header matching ADMIN_API_KEY. Secure by default - with no key configured those endpoints return 503 instead of being open. Comparison via secrets.compare_digest. Dashboard and read endpoints stay public. Data: the validator rejected any measurement whose discharge_percent exceeded 200 - which silently deleted the Oct 2024 record-flood peaks (the river genuinely ran at 201-226% of channel capacity). The cap is now 500%, and an out-of-range percent nulls that auxiliary field instead of discarding the whole row (water level and discharge are the data that matter). Surfaced by the user's historical backfill log. |
||
|
|
9cac9c4d2a |
style: apply black/isort across the repo; make CI mypy advisory
The push-CI gates (black/isort/mypy) had never actually run before the branch-trigger fix, and the codebase predates them. Formatting is now black/isort clean repo-wide. mypy keeps running but non-blocking: 86 pre-existing errors are a separate cleanup, not a gate to hold hostage. |
||
|
|
4358d52d55 |
feat: ML flood-event forecasting from 8 years of gauge history
Security & Dependency Updates / Dependency Security Scan (push) Successful in 1m8s
Security & Dependency Updates / License Compliance (push) Successful in 25s
Security & Dependency Updates / Check for Dependency Updates (push) Successful in 20s
Security & Dependency Updates / Code Quality Metrics (push) Successful in 17s
Security & Dependency Updates / Security Summary (push) Successful in 9s
Add src/ml/ package predicting, per station and per 6/12/24 h horizon, the probability of exceeding warning (3.0 m) and danger (4.5 m) levels plus expected peak level, trained on the 592k-row PostgreSQL history: - features.py: hourly grid with coverage gating and no future leakage; upstream stations enter at empirically measured travel-time lags (P.20 +17h ... P.103 +1h vs P.1); hour-of-day deliberately excluded (it encodes the scrape schedule, not hydrology) - train.py: HistGradientBoosting regression + warn/danger classifier heads per station x horizon, >=30-positives gate with calibrated sigmoid-on-regression fallback, strict temporal splits, per-event lead-time evaluation; guards against sklearn 1.9.0 crash on degenerate feature columns - predict.py: bundle loading with feature-name checks, heuristic fallback tier, get_latest_forecasts() for the API; raises when no models are trained so the endpoint 503s instead of serving persistence output as forecasts - data.py: Postgres-first loader (FLOOD_ML_DB_URL override), HTTP API fallback (flagged: that path backfills synthetic discharge), csv.gz cache - /forecast endpoint (15-min TTL cache) + dashboard flood-risk panel (hidden until models exist) - docs/FLOOD_FORECASTING.md: full system doc with measured deployment numbers (~335 MB RSS, CPU negligible, ~6 min full retrain) and retraining policy Validation: out-of-sample backtest of the record 2024 flood season (train <= Aug 2024) alerted 24-48 h ahead of the Oct 5 peak; 2025-26 test split: P.1 6h PR-AUC 0.974, recall 98.3% at 1% false-alarm rate. Also: fix P.81 station coordinates (was Ban Pong/Ratchaburi, 493 km out of basin; now 18.6936 N 99.0819 E per RID station page), pin scikit-learn==1.9.0 and numpy<2, gitignore model artifacts (~100 MB, train on the server via scripts/train_flood_model.py). |
||
|
|
49a3de0087 |
fix: restore station telemetry and make river flow visualization data-driven
Station selection showed no history since
|
||
|
|
9f26b32c86 | fix: bump history limits to 100k so all-time shows full dataset | ||
|
|
bbbf548d66 | fix: align web API limit cap with 10k and clean up postgres_history imports | ||
|
|
2fe1dcf4da | feat: add 5-minute TTL cache for PostgreSQL history queries | ||
|
|
6f2a8a0d8f | feat: add all-time history option and increase limit to 10k | ||
|
|
ae5d0a13d7 |
[verified] feat: add live river dashboard
Add mapped river and ThaiWater sensor layers, PostgreSQL history charts, API endpoints, and dashboard tests. |
||
|
|
e5936d5717 |
Move dashboard HTML out of web_api into a static file
Extract the inline root() dashboard markup into src/static/dashboard.html, loaded once at import. Keeps HTML out of the Python module (web_api 616 -> 576 lines) with a defensive fallback if the file is missing. Full <500 compliance for web_api still needs the endpoints split into APIRouter modules; tracked as remaining #4 work. |
||
|
|
ad7f7b8c76 |
Extract API Pydantic models into schemas.py
Move the seven request/response models out of web_api.py into a dedicated src/schemas.py (separation of concerns; first step of the file-size cleanup). web_api.py imports them back, so behaviour is unchanged. Note: web_api.py is still over the 500-line guideline; the remaining bulk is the inline HTML dashboard in root(), to be extracted in a follow-up. |
||
|
|
4bc3d82773 |
Persist station CRUD across restarts via JSON config
Station CRUD via the API previously mutated the scraper's in-memory station_mapping only, so changes were lost on restart (and the systemd service auto-restarts). - Extract the 130-line hardcoded station_mapping into bundled defaults at src/data/stations.json; the scraper loads from a runtime-writable config file (STATION_CONFIG_PATH, default stations.json) and falls back to the bundled defaults to seed it. - Add scraper.save_stations() with an atomic temp-file + os.replace write. - create/update/delete station endpoints now persist and roll back the in-memory change if the write fails; re-raise HTTPException so persistence errors surface as real 500s instead of being swallowed. - Backend-agnostic (works for the VictoriaMetrics deployment, which has no relational stations table). Runtime stations.json is gitignored. Also clears pre-existing flake8 debt in water_scraper_v3.py (unused imports, long lines, duplicate logging import) and dedupes the User-Agent to Config.USER_AGENT. |
||
|
|
6e78225d00 |
Fix optional-discharge crashes and dedupe measurement mapping
- MeasurementResponse.discharge is now Optional[float]; measurements with a null discharge no longer raise a Pydantic ValidationError (HTTP 500) on the /measurements/latest and /measurements/station endpoints. - InfluxDB save_measurements guards float(discharge) against None instead of crashing with TypeError. - Extract the duplicated measurement->response mapping into a single _to_measurement_response helper used by both measurement endpoints. |
||
|
|
f4c63cabef |
Fix Matrix message formatting and harden security
Matrix alerts:
- Send HTML formatted_body (org.matrix.custom.html) so **bold** and URLs
render instead of showing literal Markdown; add plain-text body fallback.
Add dependency-free markdown_to_matrix_html/strip_markdown helpers with
HTML escaping of station/message data.
Security:
- InfluxDB: bind untrusted station_codes as query params and cast limit to
int (was f-string interpolation / injection risk).
- VictoriaMetrics: escape Prometheus label values and coerce metric values
to float, preventing exposition-format injection and None crashes.
- web_api: run blocking scrape cycle via run_in_executor so it no longer
freezes the event loop; make CORS origins configurable and only allow
credentials with explicit origins ("*" + credentials is invalid/unsafe).
- config: remove hardcoded root/postgres password fallbacks (raise instead)
and stop defaulting VM_HOST to a real infrastructure hostname.
Also remove unused imports and wrap long lines to satisfy flake8.
|
||
|
|
17a716fcd0 |
Version bump: 3.1.2 3.1.3 (Force new build)
Release - Northern Thailand Ping River Monitor / Create Release (push) Successful in 7s
Security & Dependency Updates / Dependency Security Scan (push) Successful in 35s
Security & Dependency Updates / Check for Dependency Updates (push) Has been cancelled
Security & Dependency Updates / Code Quality Metrics (push) Has been cancelled
Security & Dependency Updates / Security Summary (push) Has been cancelled
Security & Dependency Updates / License Compliance (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.11) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.12) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.9) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Build Release Images (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Security Scan (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Deploy Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Validate Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Notify Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.10) (push) Has been cancelled
Version Updates: - Core application: src/__init__.py, src/main.py, src/web_api.py - Package configuration: setup.py - Documentation: README.md, docs/GITEA_WORKFLOWS.md - Workflows: .gitea/workflows/docs.yml, .gitea/workflows/release.yml - Scripts: generate_badges.py, init_git scripts - Tests: test_integration.py - Deployment docs: GITEA_SETUP_SUMMARY.md, DEPLOYMENT_CHECKLIST.md Purpose: - Force new build process after workflow fixes - Test updated security.yml without YAML errors - Verify setup.py robustness improvements - Trigger clean CI/CD pipeline execution All version references synchronized at v3.1.3 Ready for new build and deployment testing |
||
|
|
40aef686af |
Fix: Replace GitHub checkout with Gitea checkout + Version bump
Release - Northern Thailand Ping River Monitor / Create Release (push) Failing after 1s
Release - Northern Thailand Ping River Monitor / Build Release Images (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.10) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.11) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.12) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.9) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Security Scan (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Deploy Release (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Validate Release (push) Has been skipped
Security & Dependency Updates / Dependency Security Scan (push) Failing after 1s
Security & Dependency Updates / Docker Security Scan (push) Failing after 1s
Security & Dependency Updates / License Compliance (push) Failing after 1s
Security & Dependency Updates / Check for Dependency Updates (push) Failing after 1s
Security & Dependency Updates / Code Quality Metrics (push) Failing after 1s
Release - Northern Thailand Ping River Monitor / Notify Release (push) Successful in 1s
Security & Dependency Updates / Security Summary (push) Failing after 3s
Checkout Action Migration: - Replace all 'actions/checkout@v4' with 'https://gitea.com/actions/checkout' - Fixes 'Bad credentials' errors when workflows try to access GitHub API - Native Gitea checkout action eliminates authentication issues - Applied across all 4 workflow files (CI, Security, Release, Docs) Version Increment: 3.1.1 3.1.2 - Core application version updates - Web API version synchronization - Documentation version alignment - Badge and release example updates Problem Solved: - Workflows no longer attempt GitHub API calls - Gitea-native checkout action handles repository access properly - Eliminates 'Retrieving the default branch name' failures - Cleaner workflow execution without authentication errors Files Updated: - 4 workflow files: checkout action replacement - 13 files: version number updates - Consistent v3.1.2 across all components Benefits: - Workflows will now run successfully in Gitea - No more GitHub API authentication failures - Native Gitea action compatibility - Ready for successful CI/CD pipeline execution |
||
|
|
19e182c53b |
Version bump: 3.1.0 3.1.1
Release - Northern Thailand Ping River Monitor / Create Release (push) Successful in 6s
Security & Dependency Updates / Dependency Security Scan (push) Successful in 19s
Security & Dependency Updates / Docker Security Scan (push) Successful in 1m12s
Security & Dependency Updates / License Compliance (push) Successful in 11s
Security & Dependency Updates / Check for Dependency Updates (push) Successful in 14s
Security & Dependency Updates / Code Quality Metrics (push) Successful in 9s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.10) (push) Failing after 1m17s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.11) (push) Failing after 23s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.9) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Build Release Images (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Security Scan (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Deploy Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Validate Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Notify Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.12) (push) Has been cancelled
Security & Dependency Updates / Security Summary (push) Has been cancelled
Version Updates: - Core application version (src/__init__.py) - Web API version (src/web_api.py) - Main application logging (src/main.py) - Package setup version (setup.py) - Documentation generation (docs workflow) - Release workflow example version - Badge generation script - Integration test version display - README.md badge version - Setup and deployment documentation - Git initialization scripts Patch Release (3.1.1): - Workflow token migration fixes (GITHUB_TOKEN GH_TOKEN) - Pip installation warning elimination - Improved workflow reliability and logging - Better Gitea compatibility - Enhanced error handling and validation Files Updated: - 13 files with version references updated - Consistent versioning across all components - Ready for release tagging and deployment |
||
|
|
af62cfef0b |
Initial commit: Northern Thailand Ping River Monitor v3.1.0
Security & Dependency Updates / Dependency Security Scan (push) Successful in 29s
Security & Dependency Updates / Docker Security Scan (push) Failing after 53s
Security & Dependency Updates / License Compliance (push) Successful in 13s
Security & Dependency Updates / Check for Dependency Updates (push) Successful in 19s
Security & Dependency Updates / Code Quality Metrics (push) Successful in 11s
Security & Dependency Updates / Security Summary (push) Successful in 7s
Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure! |