feat: "Is the model getting better?" - live verification per model version

src/ml/skill.py joins forecast_history (what each deployed version predicted for the 24 h peak, hourly) to water_measurements (what the river did) and reports per version: verified hours, peak MAE, bias, the persistence baseline (peak = current level), skill = 1 - MAE/persistence, and the same MAE restricted to observed peaks >= 2 m. Only forecasts whose window has elapsed with >= 75 % of hours observed count; a version needs 24 verified hours before it is compared. GET /api/forecast/skill?station_code=P.1&horizon=24 returns it (SWR cached, 15 min). The dashboard's forecast card gains a panel with a one-line verdict (current vs previous version), the per-version table, and a caveat that quiet weeks measure quiet-river accuracy only: the model is judged on flood-onset lead, which the backtests cover. EN + TH. On today's production data: hgb-v3+28b62e5 (369 h, Aug 13 - Sep 1) MAE 15.2 cm, skill -0.05; hgb-v2+f6570ac (224 h, Sep 1 - 11) MAE 12.3 cm, skill 0.36 - the "worse" v2 model scores better on a quieter fortnight, which is exactly why the panel shows the >= 2 m column and the caveat. Tests: 3, sqlite, synthetic. scripts/dev_proxy.py: DEV_PROXY_LOCAL lets a not-yet-deployed endpoint be answered from a local JSON file while everything else goes to prod.
docs: README describes the project that exists; CLAUDE.md for agents
2026-09-11 23:44:44 +02:00 · 2026-09-11 23:44:43 +02:00 · 2026-09-11 23:44:43 +02:00
21 changed files with 872 additions and 1513 deletions
@@ -38,7 +38,7 @@ jobs:
      - name: Install tools
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore black==23.11.0 isort==5.12.0 flake8==6.1.0
+          pip install --root-user-action=ignore black==26.5.1 isort==5.12.0 flake8==6.1.0
      - name: black
        run: black --check --diff src/ *.py
@@ -67,7 +67,7 @@ jobs:
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore -r requirements.txt
-          pip install --root-user-action=ignore pytest==7.4.3 pytest-asyncio==0.21.1
+          pip install --root-user-action=ignore pytest==9.1.1 pytest-asyncio==0.21.1
      - name: pytest
        env:
@@ -1,293 +1,104 @@
-name: Security & Dependency Updates
+name: Security
 # Two gates that can actually fail, plus one report:
 #   - pip-audit against requirements.txt: any known vulnerability in a runtime
 #     dependency fails the job (dev-only tools are reported, not gated)
 #   - bandit on src/: HIGH severity findings fail; medium/low are listed.
 #     B104 (bind 0.0.0.0) is skipped: the service is meant to listen on all
 #     interfaces behind Cloudflare/Caddy.
 #   - pip-licenses report as an artifact (informational; the project is MIT
 #     and its runtime deps are MIT/BSD/Apache/PSF)
 # The old file ran safety/bandit/semgrep with `|| true` and could not go red.
 on:
  schedule:
-    # Run security scans daily at 3 AM UTC
+    - cron: "0 3 * * 1" # weekly, Monday 03:00 UTC
    - cron: "0 3 * * *"
  workflow_dispatch:
  push:
    paths:
      - "requirements*.txt"
-      - "Dockerfile"
+      - "pyproject.toml"
      - "uv.lock"
      - "src/**/*.py"
      - ".gitea/workflows/security.yml"
  pull_request:
    paths:
      - "requirements*.txt"
      - "pyproject.toml"
      - "src/**/*.py"
 env:
  PYTHON_VERSION: "3.11"
  # GitHub token for better rate limits and authentication
  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 jobs:
-  # Dependency vulnerability scan
+  dependencies:
-  dependency-scan:
+    name: Dependency vulnerabilities
    name: Dependency Security Scan
    runs-on: ubuntu-latest
    steps:
-      - name: Checkout code
+      - uses: actions/checkout@v4
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
-      - name: Set up Python
+      - uses: actions/setup-python@v5
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install dependencies
+      - name: Install pip-audit
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore safety bandit semgrep
+          pip install --root-user-action=ignore pip-audit
-      - name: Run Safety check
+      - name: Runtime dependencies (gate)
-        run: |
+        run: pip-audit -r requirements.txt --strict --desc on
          safety check -r requirements.txt --json --output safety-report.json || true
          safety check -r requirements-dev.txt --json --output safety-dev-report.json || true
-      - name: Run Bandit security scan
+      - name: Dev dependencies (report only)
-        run: |
+        run: pip-audit -r requirements-dev.txt --desc on || echo "::warning::dev-only dependency advisories above"
          bandit -r src/ -f json -o bandit-report.json || true
-      - name: Run Semgrep security scan
+  code:
-        run: |
+    name: Static analysis
          semgrep --config=auto src/ --json --output=semgrep-report.json || true
      - name: Upload security reports
        uses: actions/upload-artifact@v3
        with:
          name: security-reports-${{ github.run_number }}
          path: |
            safety-report.json
            safety-dev-report.json
            bandit-report.json
            semgrep-report.json
      - name: Check for critical vulnerabilities
        run: |
          echo "Checking for critical vulnerabilities..."
          # Check Safety results
          if [ -f safety-report.json ]; then
            critical_count=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null || echo "0")
            if [ "$critical_count" -gt 0 ]; then
              echo "Found $critical_count dependency vulnerabilities"
              jq '.vulnerabilities[] | "- \(.package_name) \(.installed_version): \(.vulnerability_id)"' safety-report.json
            else
              echo "No dependency vulnerabilities found"
            fi
          fi
          # Check Bandit results
          if [ -f bandit-report.json ]; then
            high_severity=$(jq '.results[] | select(.issue_severity == "HIGH") | length' bandit-report.json 2>/dev/null | wc -l)
            if [ "$high_severity" -gt 0 ]; then
              echo "Found $high_severity high-severity security issues"
            else
              echo "No high-severity security issues found"
            fi
          fi
  # License compliance check
  license-check:
    name: License Compliance
    runs-on: ubuntu-latest
    steps:
-      - name: Checkout code
+      - uses: actions/checkout@v4
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
-      - name: Set up Python
+      - uses: actions/setup-python@v5
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install pip-licenses
+      - name: Install bandit
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-licenses
+          pip install --root-user-action=ignore bandit
          pip install --root-user-action=ignore -r requirements.txt
-      - name: Check licenses
+      - name: bandit (HIGH fails; medium/low listed)
        run: |
-          echo "Checking dependency licenses..."
+          bandit -r src/ -q --skip B104 -ll -ii || true
          bandit -r src/ -q --skip B104 --severity-level high --confidence-level medium
  licenses:
    name: License report
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
          cache-dependency-path: requirements.txt
      - name: Install
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore -r requirements.txt pip-licenses
      - name: Report
        run: |
          pip-licenses --format=markdown --with-urls --output-file=licenses.md
          pip-licenses --format=json --output-file=licenses.json
-          pip-licenses --format=markdown --output-file=licenses.md
+          echo "Copyleft licenses among runtime deps (informational):"
          pip-licenses --format=plain | grep -iE 'GPL|AGPL|LGPL' || echo "  none"
-          # Check for problematic licenses
+      - uses: actions/upload-artifact@v3
          problematic_licenses=("GPL" "AGPL" "LGPL")
          for license in "${problematic_licenses[@]}"; do
            if grep -i "$license" licenses.json; then
              echo "Found potentially problematic license: $license"
            fi
          done
          echo "License check completed"
      - name: Upload license report
        uses: actions/upload-artifact@v3
        with:
-          name: license-report-${{ github.run_number }}
+          name: licenses-${{ github.run_number }}
          path: |
            licenses.json
            licenses.md
-
+            licenses.json
  # Dependency update check
  dependency-update:
    name: Check for Dependency Updates
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Install pip-check-updates equivalent
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore pip-review
      - name: Check for outdated packages
        run: |
          echo "Checking for outdated packages..."
          pip install --root-user-action=ignore -r requirements.txt
          pip list --outdated --format=json > outdated-packages.json || true
          if [ -s outdated-packages.json ]; then
            echo "Outdated packages found:"
            cat outdated-packages.json | jq -r '.[] | "- \(.name): \(.version) -> \(.latest_version)"'
          else
            echo "All packages are up to date"
          fi
      - name: Upload dependency reports
        uses: actions/upload-artifact@v3
        with:
          name: dependency-reports-${{ github.run_number }}
          path: |
            outdated-packages.json
  # Code quality metrics
  code-quality:
    name: Code Quality Metrics
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Install quality tools
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore radon xenon vulture
          pip install --root-user-action=ignore -r requirements.txt
      - name: Calculate code complexity
        run: |
          echo "Calculating code complexity..."
          radon cc src/ --json > complexity-report.json
          radon mi src/ --json > maintainability-report.json
          echo "Complexity Summary:"
          radon cc src/ --average
          echo "Maintainability Summary:"
          radon mi src/
      - name: Find dead code
        run: |
          echo "Checking for dead code..."
          vulture src/ --json > dead-code-report.json || true
      - name: Check for code smells
        run: |
          echo "Checking for code smells..."
          xenon --max-absolute B --max-modules A --max-average A src/ || true
      - name: Upload quality reports
        uses: actions/upload-artifact@v3
        with:
          name: code-quality-reports-${{ github.run_number }}
          path: |
            complexity-report.json
            maintainability-report.json
            dead-code-report.json
  # Security summary
  security-summary:
    name: Security Summary
    runs-on: ubuntu-latest
    needs: [dependency-scan, license-check, code-quality]
    if: always()
    steps:
      - name: Download all artifacts
        uses: actions/download-artifact@v3
      - name: Generate security summary
        run: |
          echo "# Security Scan Summary" > security-summary.md
          echo "" >> security-summary.md
          echo "**Scan Date:** $(date -u)" >> security-summary.md
          echo "**Repository:** ${{ github.repository }}" >> security-summary.md
          echo "**Commit:** ${{ github.sha }}" >> security-summary.md
          echo "" >> security-summary.md
          echo "## Results" >> security-summary.md
          echo "" >> security-summary.md
          # Dependency scan results
          if [ -f security-reports-*/safety-report.json ]; then
            vuln_count=$(jq '.vulnerabilities | length' security-reports-*/safety-report.json 2>/dev/null || echo "0")
            if [ "$vuln_count" -eq 0 ]; then
              echo "- Dependency Scan: No vulnerabilities found" >> security-summary.md
            else
              echo "- Dependency Scan: $vuln_count vulnerabilities found" >> security-summary.md
            fi
          else
            echo "- Dependency Scan: Results not available" >> security-summary.md
          fi
          # Docker scan results (removed Trivy)
          echo "- Docker Scan: Skipped (Trivy removed)" >> security-summary.md
          # License check results
          if [ -f license-report-*/licenses.json ]; then
            echo "- License Check: Completed" >> security-summary.md
          else
            echo "- License Check: Results not available" >> security-summary.md
          fi
          # Code quality results
          if [ -f code-quality-reports-*/complexity-report.json ]; then
            echo "- Code Quality: Analyzed" >> security-summary.md
          else
            echo "- Code Quality: Results not available" >> security-summary.md
          fi
          echo "" >> security-summary.md
          echo "## Detailed Reports" >> security-summary.md
          echo "" >> security-summary.md
          echo "Detailed reports are available in the workflow artifacts." >> security-summary.md
          cat security-summary.md
      - name: Upload security summary
        uses: actions/upload-artifact@v3
        with:
          name: security-summary-${{ github.run_number }}
          path: security-summary.md
@@ -19,7 +19,7 @@ repos:
  # Python code formatting with Black
  - repo: https://github.com/psf/black
-    rev: 23.11.0
+    rev: 26.5.1
    hooks:
      - id: black
        language_version: python3
@@ -0,0 +1,40 @@
 # CLAUDE.md
 Guidance for AI coding agents working in this repository.
 ## What this is
 Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
 API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
 Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
 `water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
 ## Rules
 - Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
 - `make format` (black 88 / isort black profile, config in pyproject.toml) before
  committing; CI fails on formatting. `make test` must stay green — tests are
  synthetic-data only, never add one that needs the DB or network.
 - Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
  parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
 - Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
  and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive
  or negative, in `docs/FLOOD_FORECASTING.md` section 5. Do not change what is deployed
  (`rise_rain` / hgb-v3) without a harness result that beats it on lead.
 - `train_all()` must never silently produce a gauge-only (v2) model; the guard that
  raises `RainUnavailableError` stays.
 - No `git add -A`: zero-byte shell-accident files (`#`, `$(wc`, ...) have been committed
  before. Stage files by name.
 - Do not add Co-Authored-By trailers.
 - The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
  table: every user-visible string needs both languages.
 ## Where things are
 - `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
  `src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
  `rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
 - `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
  promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
 - `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
  the source catalog.
@@ -1,494 +1,150 @@
-# Northern Thailand Ping River Monitor 🏔️
+# Northern Thailand Ping River Monitor
-A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.
+Live water levels, discharge, rainfall and machine-learning flood forecasts for the
 Ping River basin around Chiang Mai. Collects hourly gauge data from public sources,
 keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST
 API, and 6/12/24-hour flood-risk forecasts per gauge.
-**Live dashboard: [water.buildfor.life](https://water.buildfor.life/)** — water levels, discharge, rainfall and 6/12/24 h flood forecasts for Chiang Mai, in English and Thai. Background: [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
+**Live: [water.buildfor.life](https://water.buildfor.life/)** · API reference at
 [/docs](https://water.buildfor.life/docs) · built by [buildfor.life](https://buildfor.life)
 after the [October 2024 flood](https://buildfor.life/blog/chiang-mai-flood-2024/) —
 background in [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
-[![CI/CD](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Documentation](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-0.104+-green.svg)](https://fastapi.tiangolo.com) [![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Version](https://img.shields.io/badge/Version-v3.1.3-blue.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/releases)
+[![CI](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Docs](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Python 3.11](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
-## 🌟 Features
+## What it does
-### 📊 **Real-time Data Collection**
+- **Collects** hourly water level and discharge from 16 Royal Irrigation Department
- **16 Monitoring Stations** across Thailand
+  (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly
- **15-minute Collection Frequency** with intelligent scheduling
+  rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment
- **Automatic Gap Filling** for missing historical data
+  rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and
- **Data Validation** and error recovery mechanisms
+  its quirks: [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md).
- **Rate Limiting** to prevent API abuse
+- **Fills gaps.** The raw RID grid had readings for ~56 % of hours; a full-history
  re-fetch plus HII cross-fill brought it to ~93 %. `GET /api/stats` reports the
  current figure.
 - **Forecasts.** Per gauge and horizon, a gradient-boosted model predicts the rise
  within 6/12/24 h and the probability of crossing the station's warning and danger
  levels. Trained on the monitor's own history plus catchment rain; evaluated
  rolling-origin, event by event. On the October 2024 record flood, trained only on
  data through August 2024, the first alert came **13 hours before** P.1 crossed
  3.70 m. Everything about the model, including what did not work:
  [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md).
 - **Shows it.** A Leaflet map with the river drawn as OSM geometry and styled by live
  discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the
  forecast card, a replay of the 2024 flood, English/Thai, light/dark.
 - **Alerts** (optional) to a Matrix room when a gauge crosses its thresholds.
-### 🌐 **Web API Interface (NEW!)**
+## Quick start
 - **FastAPI-powered REST API** with interactive documentation
 - **Station Management** - Add, update, and remove monitoring stations
 - **Real-time health monitoring** and system status
 - **Manual data collection triggers** via web interface
 - **Comprehensive metrics** and performance monitoring
 - **CORS support** for web applications
-### 🗄️ **Multi-Database Support**
+Python **3.11** (3.13 breaks the pinned `psycopg2-binary`), PostgreSQL for anything
- **VictoriaMetrics** (Recommended) - High-performance time-series
+beyond a quick look, [uv](https://docs.astral.sh/uv/).
 - **InfluxDB** - Purpose-built time-series database
 - **PostgreSQL + TimescaleDB** - Relational with time-series optimization
 - **MySQL** - Traditional relational database
 - **SQLite** - Local development and testing
 ### 🗺️ **Geolocation Support**
 - **Grafana Geomap** integration ready
 - **GPS coordinates** and geohash support
 - **Interactive mapping** of water stations
 ### 📈 **Visualization & Monitoring**
 - **Pre-built Grafana dashboards**
 - **Real-time alerts** and notifications
 - **Historical trend analysis**
 - **Built-in metrics collection** (counters, gauges, histograms)
 - **Health checks** for database, API, and system resources
 ### 🚀 **Production Ready**
 - **Docker containerization** with multi-service support
 - **Systemd service** configuration
 - **HTTPS support** with SSL certificates
 - **Comprehensive logging** with rotation and colored output
 - **Type safety** with Pydantic models and type hints
 - **Custom exception handling** for better error management
 ## 🚀 Quick Start
 ### Prerequisites
 - Python 3.9 or higher
 - Internet connection for data fetching
 - Database server (optional - SQLite works out of the box)
 ### Installation
 ```bash
 # Clone the repository
 git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 cd Northern-Thailand-Ping-River-Monitor
-
+uv sync --python 3.11
-# Quick setup with Make
+cp .env.example .env            # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
-make dev-setup
+uv run python run.py --web-api  # dashboard + API on http://localhost:8000
 # Or manual setup:
 python -m venv venv
 source venv/bin/activate  # Windows: venv\Scripts\activate
 pip install -r requirements.txt
 cp .env.example .env
 ```
-### Basic Usage
+`DB_TYPE=sqlite` works for the dashboard and API; the forecasting path expects the
 PostgreSQL history.
 ```bash
-# Test run with SQLite (default)
+uv run python run.py --status          # collector status
-make run-test
+uv run python run.py --test            # one collection cycle
-# or: python run.py --test
+uv run python run.py --fill-gaps 7     # re-fetch the last 7 days from RID
-
+uv run python run.py --collect-hii     # one ThaiWater/HII collection cycle
-# Run continuous monitoring
+uv run python run.py --alert-check     # evaluate thresholds, notify Matrix
-make run
+uv run python scripts/train_flood_model.py --stations all   # retrain (~12 min)
-# or: python run.py
+make test                              # pytest, synthetic data, no network
-
+make format                            # black + isort (the CI contract)
 # Start web API server (NEW!)
 make run-api
 # or: python run.py --web-api
 # Run all tests
 make test
 # Demo different databases
 python src/demo_databases.py
 ```
-### 🌐 Web API Interface (NEW!)
+## API
-The system now includes a comprehensive FastAPI web interface:
+Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
 Asia/Bangkok wall-clock without an offset suffix.
 | Endpoint | Returns |
 | --- | --- |
 | `GET /stations` | The 16 RID gauges: code, Thai/English names, coordinates |
 | `GET /measurements/latest?limit=N` | Newest reading per station |
 | `GET /measurements/history/{code}?hours=N` | Hourly history; or `?start=YYYY-MM-DD&end=YYYY-MM-DD`; `limit` ≤ 100000 |
 | `GET /forecast` | Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
 | `GET /api/forecast/history/{code}?hours=N&horizon=24` | Forecasts as issued, for auditing lead time after the fact |
 | `GET /api/hii/rainfall/latest`, `/api/hii/waterlevel/latest` | Latest ThaiWater/HII gauge readings |
 | `GET /api/hii/rainfall/catchment?days=N` | HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
 | `GET /api/stats` | Row counts per source, date range, coverage |
 | `GET /health` | DB / upstream / memory checks |
 Interactive reference with schemas: [water.buildfor.life/docs](https://water.buildfor.life/docs).
 Responses are cached briefly server-side; poll no faster than once a minute — the data
 changes hourly.
 ## Deployment
 Production is a systemd unit on a small VPS behind Cloudflare, updated by `git pull`.
 `scripts/install.sh` (run as root from a checkout) creates the `water-monitor` user,
 deploys to `/opt/thailand-water-monitor`, runs `uv sync` into `.venv`, installs
 `water-monitor.service` and the monthly `water-monitor-retrain.timer`.
 ```bash
 # Start the web API
 python run.py --web-api
 # Access the API at:
 # - Dashboard: http://localhost:8000
 # - Interactive docs: http://localhost:8000/docs
 # - Health check: http://localhost:8000/health
 # - Latest data: http://localhost:8000/measurements/latest
 ```
 **Key API Endpoints:**
 - `GET /` - Web dashboard
 - `GET /health` - System health status
 - `GET /metrics` - Application metrics
 - `GET /stations` - List all monitoring stations
 - `POST /stations` - Add new monitoring station
 - `PUT /stations/{id}` - Update station information
 - `DELETE /stations/{id}` - Remove monitoring station
 - `GET /measurements/latest` - Latest measurements
 - `GET /measurements/station/{code}` - Station-specific data
 - `POST /scrape/trigger` - Trigger manual data collection
 ## 📊 Station Information
 The system monitors **16 water stations** along the Ping River Basin in Northern Thailand:
 | Station | Thai Name | English Name | Location |
 |---------|-----------|--------------|----------|
 | P.1 | สะพานนวรัฐ | Nawarat Bridge | Nakhon Sawan |
 | P.5 | สะพานท่านาง | Tha Nang Bridge | - |
 | P.20 | บ้านเชียงดาว | Ban Chiang Dao | Chiang Mai |
 | P.21 | บ้านริมใต้ | Ban Rim Tai | - |
 | P.4A | บ้านแม่แตง | Ban Mae Taeng | Chiang Mai |
 | P.67 | บ้านแม่แต | Ban Tae | - |
 | P.75 | บ้านช่อแล | Ban Chai Lat | - |
 | P.76 | บ้านแม่อีไฮ | Banb Mae I Hai | - |
 | P.77 | บ้านสบแม่สะป๊วด | Baan Sop Mae Sapuord | - |
 | P.81 | บ้านโป่ง | Ban Pong | - |
 | P.82 | บ้านสบวิน | Ban Sob win | - |
 | P.84 | บ้านพันตน | Ban Panton | - |
 | P.85 | บ้านหล่ายแก้ว | Baan Lai Kaew | - |
 | P.87 | บ้านป่าซาง | Ban Pa Sang | - |
 | P.92 | บ้านเมืองกึ๊ด | Ban Muang Aut | - |
 | P.103 | สะพานวงแหวนรอบ 3 | Ring Bridge 3 | Bangkok |
 ### Data Metrics
 - **Water Level**: Measured in meters (m)
 - **Discharge**: Flow rate in cubic meters per second (cms)
 - **Discharge Percentage**: Relative to station capacity
 - **Timestamp**: Thai time (UTC+7) with Buddhist calendar support
 ## 🗄️ Database Configuration
 ### VictoriaMetrics (Recommended)
 **High-performance time-series database with excellent compression and query speed.**
 ```bash
 # Environment variables
 export DB_TYPE=victoriametrics
 export VM_HOST=localhost
 export VM_PORT=8428
 # Quick start with Docker
 docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428
 ```
 ### Complete Stack with Grafana
 ```bash
 # Start the complete monitoring stack
 docker-compose -f docker-compose.victoriametrics.yml up -d
 # Access Grafana at http://localhost:3000
 # Username: admin, Password: admin_password
 ```
 ### Other Database Options
 <details>
 <summary>InfluxDB Configuration</summary>
 ```bash
 export DB_TYPE=influxdb
 export INFLUX_HOST=localhost
 export INFLUX_PORT=8086
 export INFLUX_DATABASE=water_monitoring
 export INFLUX_USERNAME=water_user
 export INFLUX_PASSWORD=your_password
 ```
 </details>
 <details>
 <summary>PostgreSQL Configuration</summary>
 ```bash
 export DB_TYPE=postgresql
 export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring
 ```
 </details>
 <details>
 <summary>MySQL Configuration</summary>
 ```bash
 export DB_TYPE=mysql
 export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring
 ```
 </details>
 ## 📈 Grafana Dashboards
 ### Pre-built Dashboard Features
 - **Real-time water levels** across all stations
 - **Historical trends** and patterns
 - **Discharge monitoring** with percentage indicators
 - **Station status** and health monitoring
 - **Geomap visualization** of station locations
 - **Alert thresholds** for critical water levels
 ### Sample Queries
 **VictoriaMetrics/Prometheus:**
 ```promql
 # Current water levels
 water_level
 # High discharge alerts
 water_discharge_percent > 80
 # Station-specific data
 water_level{station_code="P.1"}
 ```
 **SQL Databases:**
 ```sql
 -- Latest readings from all stations
 SELECT s.station_code, s.english_name, m.water_level, m.discharge
 FROM stations s
 JOIN water_measurements m ON s.id = m.station_id
 WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);
 ```
 ## 🚀 Production Deployment
 ### Docker Deployment
 ```bash
 # Build the image
 docker build -t thailand-water-monitor .
 # Run with environment variables
 docker run -d \
  --name water-monitor \
  -e DB_TYPE=victoriametrics \
  -e VM_HOST=victoriametrics \
  thailand-water-monitor
 ```
 ### Systemd Service (Linux)
 The install script sets everything up: a dedicated `water-monitor` system user,
 a deploy to `/opt/thailand-water-monitor`, a uv-managed virtualenv, and the
 enabled systemd unit.
 ```bash
 # From a checkout of the repo, as root:
 sudo bash scripts/install.sh
 # Then start and check:
 sudo systemctl start water-monitor.service
-systemctl status water-monitor.service
+systemctl list-timers water-monitor-retrain.timer
 ```
-Fill in `/opt/thailand-water-monitor/.env` (Matrix token/room, DB settings)
+The retrain timer runs `scripts/retrain.sh`, which trains into `models/.staging`,
-before starting if the script reports it is missing.
+refuses to promote anything that is not a rain-enabled (`hgb-v3+`) set covering the
 expected stations, and renames the bundles into place. Details and the operations
 runbook: [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) sections 6–8.
-<details>
+## Repository layout
 <summary>Manual setup (if you prefer not to use the script)</summary>
 ```bash
 sudo useradd --system --no-create-home --shell /usr/sbin/nologin water-monitor
 uv sync --python 3.11            # creates .venv, the interpreter both units run
 sudo cp scripts/water-monitor.service scripts/water-monitor-retrain.service scripts/water-monitor-retrain.timer /etc/systemd/system/
 sudo systemctl enable --now water-monitor.service water-monitor-retrain.timer
 ```
 </details>
 ## 🔧 Command Line Tools
 ### Main Application
 ```bash
 python src/water_scraper_v3.py                    # Run continuous monitoring
 python src/water_scraper_v3.py --test             # Single test cycle
 python src/water_scraper_v3.py --help             # Show help
 ```
 ### Data Management
 ```bash
 python src/water_scraper_v3.py --check-gaps 7     # Check for missing data (7 days)
 python src/water_scraper_v3.py --fill-gaps 7      # Fill missing data gaps
 python src/water_scraper_v3.py --update-data 2    # Update existing data (2 days)
 ```
 ### Database Testing
 ```bash
 python src/demo_databases.py                      # SQLite demo
 python src/demo_databases.py victoriametrics      # VictoriaMetrics demo
 python src/demo_databases.py all                  # Test all databases
 ```
 ## 📚 Documentation
 ### Core Documentation
 - **[Data Sources & API Catalog](docs/DATA_SOURCES.md)** - Every ingested and available data source (RID, ThaiWater/HII, dams, rainfall, forecasts)
 - **[Installation Guide](docs/DATABASE_DEPLOYMENT_GUIDE.md)** - Complete setup instructions
 - **[Gap Filling Guide](docs/GAP_FILLING_GUIDE.md)** - Data integrity management
 ### Deployment Guides
 - **[VictoriaMetrics Setup](docs/VICTORIAMETRICS_SETUP.md)** - High-performance deployment
 - **[Debian Troubleshooting](docs/DEBIAN_TROUBLESHOOTING.md)** - Linux deployment issues
 ### References
 - **[Notable Documents](docs/references/NOTABLE_DOCUMENTS.md)** - Official Thai government resources
 ## 🔍 Troubleshooting
 ### Common Issues
 **Database Connection Errors:**
 ```bash
 # Check database status
 python src/demo_databases.py
 # Test specific database
 python src/demo_databases.py victoriametrics
 ```
 **Missing Data:**
 ```bash
 # Check for gaps
 python src/water_scraper_v3.py --check-gaps 7
 # Fill missing data
 python src/water_scraper_v3.py --fill-gaps 7
 ```
 **Service Issues:**
 ```bash
 # Check service status
 sudo systemctl status water-monitor
 # View logs
 sudo journalctl -u water-monitor -f
 ```
 ### Health Checks
 ```bash
 # VictoriaMetrics health
 curl http://localhost:8428/health
 # Check latest data
 curl "http://localhost:8428/api/v1/query?query=water_level"
 # Application logs
 tail -f water_monitor.log
 ```
 ## 🌐 API Integration
 ### VictoriaMetrics API Examples
 ```bash
 # Query current water levels
 curl "http://localhost:8428/api/v1/query?query=water_level"
 # Query discharge rates for last hour
 curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"
 # Query specific station
 curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
 # High discharge alerts
 curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
 ```
 ## 📊 Performance
 ### System Requirements
 - **CPU**: 1-2 cores (minimal load)
 - **RAM**: 512MB - 2GB (depending on database)
 - **Storage**: 1GB+ (for historical data)
 - **Network**: Stable internet connection
 ### Performance Metrics
 - **Data Collection**: ~300 data points every 15 minutes
 - **Database Write Speed**: 1000+ points/second (VictoriaMetrics)
 - **Query Response**: <100ms for recent data
 - **Storage Efficiency**: 70x compression vs. raw data
 ## 🤝 Contributing
 Contributions are welcome! Please:
 1. Fork the repository
 2. Create a feature branch
 3. Make your changes
 4. Add tests if applicable
 5. Submit a pull request
 ### Development Setup
 ```bash
 # Clone your fork
 git clone https://github.com/your-username/thailand-water-monitor.git
 cd thailand-water-monitor
 # Install development dependencies
 pip install -r requirements.txt
 pip install pytest black flake8
 # Run tests
 pytest
 # Format code
 black src/
 ```
 ## 📄 License
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
 ## 🙏 Acknowledgments
 - **Royal Irrigation Department (RID)** of Thailand for providing the data API
 - **VictoriaMetrics** team for the excellent time-series database
 - **Grafana** team for the visualization platform
 - **Python community** for the amazing libraries and tools
 ## 📞 Support
 - **Issues**: [GitHub Issues](https://github.com/your-username/thailand-water-monitor/issues)
 - **Discussions**: [GitHub Discussions](https://github.com/your-username/thailand-water-monitor/discussions)
 - **Documentation**: [Project Wiki](https://github.com/your-username/thailand-water-monitor/wiki)
 ---
 ## 📁 Project Structure
 ```
-Northern-Thailand-Ping-River-Monitor/
+src/                 collector, API (web_api.py), dashboard (static/dashboard.html)
-├── src/                    # Main application code
+src/ml/              features, training, evaluation harness, prediction, rain/dam/HII loaders
-├── tests/                  # Test suite
+scripts/             train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
-├── docs/                   # Documentation
+tests/               pytest suite (synthetic data; no DB or network)
-├── grafana/               # Grafana dashboards
+docs/                FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
-├── scripts/               # Utility scripts
+models/              trained bundles + metrics.json (gitignored) and evaluation results (tracked)
-├── docker-compose.yml     # Docker deployment
+.gitea/workflows/    ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
 ├── Makefile              # Development tasks
 └── requirements.txt      # Dependencies
 ```
-See [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) for the forecasting architecture and [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) for the data pipeline.
+## Documentation
-## 🔄 CI/CD & Automation
+- [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
 - [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) — every ingested and candidate source, endpoints, quirks
 - [docs/STATION_MANAGEMENT_GUIDE.md](docs/STATION_MANAGEMENT_GUIDE.md) — adding/editing gauges
 - [docs/DATABASE_DEPLOYMENT_GUIDE.md](docs/DATABASE_DEPLOYMENT_GUIDE.md), [POSTGRESQL_SETUP.md](POSTGRESQL_SETUP.md) — database setup
 - [docs/MATRIX_QUICK_START.md](docs/MATRIX_QUICK_START.md) — alert delivery
 - [docs/GAP_FILLING_GUIDE.md](docs/GAP_FILLING_GUIDE.md) — data integrity tooling
 - [docs/references/NOTABLE_DOCUMENTS.md](docs/references/NOTABLE_DOCUMENTS.md) — official Thai government resources
 - Public overview: [buildfor.life/docs/tooling/ping-river-monitor](https://buildfor.life/docs/tooling/ping-river-monitor/)
-The project includes comprehensive Gitea Actions workflows:
+Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
 dashboards under `grafana/` are supported by the adapters but not what production
 runs; see [docs/VICTORIAMETRICS_SETUP.md](docs/VICTORIAMETRICS_SETUP.md) if you want them.
- **🧪 CI/CD Pipeline** - Automated testing, building, and deployment
+## Contributing
 - **🔒 Security Scanning** - Daily vulnerability and dependency checks
 - **📚 Documentation** - Automated API docs and validation
 - **🚀 Release Management** - Automated releases with multi-arch Docker builds
-See [docs/GITEA_WORKFLOWS.md](docs/GITEA_WORKFLOWS.md) for detailed workflow documentation.
+`make format` before committing (black 88 columns, isort black profile — the CI gate),
 `make test` must stay green, tests use synthetic data only. See
 [CONTRIBUTING.md](CONTRIBUTING.md). Issues and merge requests on
 [git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
-## 🔗 Repository
+## Data sources and thanks
- **Main Repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
+Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) /
- **Issues**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
+ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry;
- **Actions**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
+Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All
- **Documentation**: [docs/](docs/)
+instruments are theirs; we aggregate, store, fill gaps and forecast.
-**Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin**
+## License
 MIT — see [LICENSE](LICENSE).
@@ -34,23 +34,23 @@ classifiers = [
    "Environment :: Web Environment",
    "Framework :: FastAPI"
 ]
-requires-python = ">=3.11"
+requires-python = ">=3.11,<3.12"
 dependencies = [
    # Core dependencies
-    "requests==2.31.0",
+    "requests==2.34.2",
    "schedule==1.2.0",
    "pandas==2.0.3",
    "numpy>=1.24,<2",
    # Flood forecasting (ML)
    "scikit-learn==1.9.0",
    # Web API framework
-    "fastapi==0.104.1",
+    "fastapi==0.141.1",
-    "uvicorn[standard]==0.24.0",
+    "uvicorn[standard]==0.52.4",
-    "pydantic==2.5.0",
+    "pydantic==2.13.5",
    # Database adapters
    "sqlalchemy==2.0.23",
    "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
    "psycopg2-binary==2.9.9",
    # Monitoring and metrics
    "psutil==5.9.6"
@@ -59,11 +59,11 @@ dependencies = [
 [project.optional-dependencies]
 dev = [
    # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
    "pytest-cov==4.1.0",
    "pytest-asyncio==0.21.1",
    # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
    "flake8==6.1.0",
    "isort==5.12.0",
    "mypy==1.7.1",
@@ -73,7 +73,7 @@ dev = [
    "ipython==8.17.2",
    "jupyter==1.0.0",
    # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
    "types-python-dateutil==2.8.19.14"
 ]
 docs = [
@@ -83,7 +83,7 @@ docs = [
 ]
 all = [
    "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
    "psycopg2-binary==2.9.9"
 ]
@@ -100,11 +100,11 @@ Documentation = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/
 [dependency-groups]
 dev = [
    # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
    "pytest-cov==4.1.0",
    "pytest-asyncio==0.21.1",
    # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
    "flake8==6.1.0",
    "isort==5.12.0",
    "mypy==1.7.1",
@@ -114,7 +114,7 @@ dev = [
    "ipython==8.17.2",
    "jupyter==1.0.0",
    # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
    "types-python-dateutil==2.8.19.14",
    # Documentation
    "sphinx==7.2.6",
@@ -2,12 +2,12 @@
 -r requirements.txt
 # Testing
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
 pytest-asyncio==0.21.1
 # Code formatting and linting
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 isort==5.12.0
 mypy==1.7.1
@@ -25,5 +25,5 @@ ipython==8.17.2
 jupyter==1.0.0
 # Type stubs
-types-requests==2.31.0.10
+types-requests==2.33.0.20260906
 types-python-dateutil==2.8.19.14
@@ -1,5 +1,5 @@
 # Core dependencies
-requests==2.31.0
+requests==2.34.2
 schedule==1.2.0
 pandas==2.0.3
 numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
@@ -8,23 +8,23 @@ numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
 scikit-learn==1.9.0
 # Web API framework
-fastapi==0.104.1
+fastapi==0.141.1
-uvicorn[standard]==0.24.0
+uvicorn[standard]==0.52.4
-pydantic==2.5.0
+pydantic==2.13.5
 # Database adapters
 sqlalchemy==2.0.23
 influxdb==5.3.1
-pymysql==1.1.0
+pymysql==1.2.0
 psycopg2-binary==2.9.9
 # Monitoring and metrics
 psutil==5.9.6
 # Development dependencies (optional)
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 mypy==1.7.1
 pre-commit==3.5.0
@@ -3,6 +3,7 @@ live server, so browser-side changes can be checked against real data
 before deploy. Usage: python scripts/dev_proxy.py [port]"""
 import http.server
 import os
 import sys
 import urllib.request
 from pathlib import Path
@@ -17,6 +18,12 @@ class Handler(http.server.BaseHTTPRequestHandler):
            body = (STATIC / "dashboard.html").read_bytes()
            self._send(200, "text/html; charset=utf-8", body)
            return
        # Local overrides for endpoints not yet deployed: DEV_PROXY_LOCAL=/api/x=file.json,...
        for pair in filter(None, os.environ.get("DEV_PROXY_LOCAL", "").split(",")):
            prefix, file = pair.split("=", 1)
            if self.path.split("?")[0] == prefix:
                self._send(200, "application/json", Path(file).read_bytes())
                return
        if self.path.startswith("/static/"):
            f = STATIC / self.path[len("/static/"):].split("?")[0]
            if f.is_file():
@@ -139,12 +139,16 @@ class InfluxDBAdapter(DatabaseAdapter):
                    "time": measurement["timestamp"].isoformat(),
                    "fields": {
                        "water_level": float(measurement["water_level"]),
-                        "discharge": float(measurement["discharge"])
+                        "discharge": (
                            float(measurement["discharge"])
                            if measurement.get("discharge") is not None
-                        else None,
+                            else None
-                        "discharge_percent": float(measurement["discharge_percent"])
+                        ),
                        "discharge_percent": (
                            float(measurement["discharge_percent"])
                            if measurement.get("discharge_percent")
-                        else None,
+                            else None
                        ),
                    },
                }
                points.append(point)
@@ -551,13 +555,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_code": row[1],
                            "station_name_en": row[2],
                            "station_name_th": row[3],
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -611,13 +615,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_code": row[1],
                            "station_name_en": row[2],
                            "station_name_th": row[3],
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -666,13 +670,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_id": row[1],
                            "station_code": row[2] or f"Station_{row[1]}",
                            "station_name_th": row[3] or f"Station {row[1]}",
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -125,9 +125,9 @@ class DatabaseHealthCheck(HealthCheck):
                "message": "Database connection OK",
                "details": {
                    "latest_data_count": len(latest_data),
-                    "latest_timestamp": str(latest_data[0].get("timestamp"))
+                    "latest_timestamp": (
-                    if latest_data
+                        str(latest_data[0].get("timestamp")) if latest_data else None
-                    else None,
+                    ),
                },
            }
@@ -207,9 +207,9 @@ def fill_from_hii(
                "timestamp": missing["timestamp"],
                "station_code": code,
                "water_level": missing["wl_msl"] - offset,
-                "discharge": missing["discharge"]
+                "discharge": (
-                if code in _HII_EXACT_MIRRORS
+                    missing["discharge"] if code in _HII_EXACT_MIRRORS else float("nan")
-                else float("nan"),
+                ),
            }
        )
        fills.append(fill)
@@ -0,0 +1,170 @@
 """Live forecast skill: what the deployed model said versus what the river did.
 Every hour the precompute stores the issued 24 h forecast (forecast_history);
 water_measurements holds what actually happened. Joining the two gives a
 verification that needs no retraining and answers the question the dashboard
 is asked most: "is the model getting better?" — per model version, on the
 hours that version was actually serving.
 Metrics per version and horizon:
  n           verified forecasts (issued, and the horizon has since elapsed)
  mae         |predicted_max - observed_max| over the horizon window, metres
  bias        mean(predicted - observed): >0 over-predicts the peak
  persistence MAE of the trivial "peak = current level" forecast on the
              same rows; a model is only useful if it beats this
  skill       1 - mae/persistence  (0 = no better than persistence, 1 = perfect)
  above_2m    same MAE restricted to rows where the observed peak >= 2 m,
              i.e. the flood-relevant regime
 Only the P.1 gauge is verified by default: it is the one the city threshold
 is keyed to, and one station keeps the query cheap enough to run on request.
 """
 import datetime
 import logging
 from typing import Dict, List, Optional
 logger = logging.getLogger(__name__)
 DEFAULT_STATION = "P.1"
 DEFAULT_HORIZON = 24
 MIN_VERIFIED = 24  # fewer than a day of verified hours is not a number
 def _sql_for(db_type: str) -> str:
    """Join each issued forecast to the observed max over (as_of, as_of + h]."""
    if db_type == "postgresql":
        window_end = "f.as_of + (f.horizon_hours || ' hours')::interval"
    elif db_type == "mysql":
        window_end = "DATE_ADD(f.as_of, INTERVAL f.horizon_hours HOUR)"
    else:  # sqlite
        window_end = "datetime(f.as_of, '+' || f.horizon_hours || ' hours')"
    return f"""
        SELECT f.as_of, f.model_version, f.predicted_max_level, f.current_level,
               (SELECT MAX(m.water_level) FROM water_measurements m
                 JOIN stations s ON s.id = m.station_id
                WHERE s.station_code = f.station_code
                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_max,
               (SELECT COUNT(m.water_level) FROM water_measurements m
                 JOIN stations s ON s.id = m.station_id
                WHERE s.station_code = f.station_code
                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_n
        FROM forecast_history f
        WHERE f.station_code = :code AND f.horizon_hours = :horizon
          AND f.source = 'model' AND f.predicted_max_level IS NOT NULL
          AND f.as_of <= :verifiable_before
        ORDER BY f.as_of
    """
 def compute_skill(
    engine,
    db_type: str,
    station_code: str = DEFAULT_STATION,
    horizon_hours: int = DEFAULT_HORIZON,
    now: Optional[datetime.datetime] = None,
 ) -> Dict:
    """Per-model-version verification of issued forecasts against observations."""
    from sqlalchemy import text
    now = now or datetime.datetime.now()
    verifiable_before = now - datetime.timedelta(hours=horizon_hours)
    with engine.connect() as conn:
        rows = [
            dict(r._mapping)
            for r in conn.execute(
                text(_sql_for(db_type)),
                {
                    "code": station_code,
                    "horizon": horizon_hours,
                    "verifiable_before": verifiable_before,
                },
            )
        ]
    def _ts(value):
        # sqlite hands back strings; postgres/mysql give datetimes
        if isinstance(value, datetime.datetime):
            return value
        return datetime.datetime.fromisoformat(str(value).replace(" ", "T"))
    by_version: Dict[str, List[dict]] = {}
    for r in rows:
        r["as_of"] = _ts(r["as_of"])
        # need most of the window observed, or the "max" is not the peak
        if r["observed_max"] is None or (r["observed_n"] or 0) < horizon_hours * 0.75:
            continue
        by_version.setdefault(r["model_version"] or "unknown", []).append(r)
    versions = []
    for version, vrows in by_version.items():
        pred = [float(r["predicted_max_level"]) for r in vrows]
        obs = [float(r["observed_max"]) for r in vrows]
        cur = [
            float(r["current_level"]) if r["current_level"] is not None else None
            for r in vrows
        ]
        err = [p - o for p, o in zip(pred, obs)]
        mae = sum(abs(e) for e in err) / len(err)
        bias = sum(err) / len(err)
        pers_rows = [(c, o) for c, o in zip(cur, obs) if c is not None]
        persistence = (
            sum(abs(c - o) for c, o in pers_rows) / len(pers_rows)
            if pers_rows
            else None
        )
        high = [(p, o) for p, o in zip(pred, obs) if o >= 2.0]
        versions.append(
            {
                "model_version": version,
                "first_issued": min(r["as_of"] for r in vrows).isoformat(),
                "last_issued": max(r["as_of"] for r in vrows).isoformat(),
                "n": len(vrows),
                "mae_m": round(mae, 3),
                "bias_m": round(bias, 3),
                "persistence_mae_m": (
                    None if persistence is None else round(persistence, 3)
                ),
                "skill": (
                    None if not persistence else round(1.0 - mae / persistence, 3)
                ),
                "above_2m_n": len(high),
                "above_2m_mae_m": (
                    round(sum(abs(p - o) for p, o in high) / len(high), 3)
                    if high
                    else None
                ),
                "enough_data": len(vrows) >= MIN_VERIFIED,
            }
        )
    versions.sort(key=lambda v: v["first_issued"])
    # Headline: current version vs the previous one that had enough data
    current = versions[-1] if versions else None
    previous = (
        next((v for v in reversed(versions[:-1]) if v["enough_data"]), None)
        if versions
        else None
    )
    trend = None
    if current and previous and current["enough_data"]:
        trend = {
            "previous_version": previous["model_version"],
            "mae_delta_m": round(current["mae_m"] - previous["mae_m"], 3),
            "skill_delta": (
                None
                if current["skill"] is None or previous["skill"] is None
                else round(current["skill"] - previous["skill"], 3)
            ),
            "better": current["mae_m"] < previous["mae_m"],
        }
    return {
        "station_code": station_code,
        "horizon_hours": horizon_hours,
        "verified_until": verifiable_before.isoformat(),
        "min_verified": MIN_VERIFIED,
        "versions": versions,
        "current": current,
        "trend": trend,
    }
@@ -320,9 +320,9 @@ def train_station(
                    skipped_heads,
                )
            else:
-                skipped_heads[
+                skipped_heads[head_key] = (
-                    head_key
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
            heads[head_key] = clf
            if not skip_eval:
@@ -417,9 +417,9 @@ def train_station(
                if clf is not None:
                    skipped_heads.pop(head_key, None)
            else:
-                skipped_heads[
+                skipped_heads[head_key] = (
-                    head_key
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
                final_heads[head_key] = None
    # v4 = + Mae Ngat dam features; v3 = rise + rain; v2 = rise target only
@@ -51,8 +51,7 @@ class PostgresHistory:
        if start >= end:
            raise ValueError("start must be before end")
-        query = text(
+        query = text("""
            """
            SELECT m.timestamp, s.station_code, m.water_level,
                   m.discharge, m.discharge_percent
            FROM water_measurements m
@@ -62,8 +61,7 @@ class PostgresHistory:
              AND m.timestamp <= :end_time
            ORDER BY m.timestamp ASC
            LIMIT :limit
-            """
+            """)
        )
        with self.engine.connect() as connection:
            rows = connection.execute(
                query,
@@ -91,9 +89,9 @@ class PostgresHistory:
                        "station_code": station_code,
                        "water_level": water_level,
                        "discharge": discharge,
-                        "discharge_percent": float(row[4])
+                        "discharge_percent": (
-                        if row[4] is not None
+                            float(row[4]) if row[4] is not None else None
-                        else None,
+                        ),
                    }
                )
            return result
@@ -173,8 +173,10 @@ class RequestTracker:
                "failed_requests": self.failed_requests,
                "success_rate": self.successful_requests / self.total_requests,
                "average_response_time": self.total_response_time / self.total_requests,
-                "last_request_time": self.last_request_time.isoformat()
+                "last_request_time": (
                    self.last_request_time.isoformat()
                    if self.last_request_time
-                else None,
+                    else None
                ),
                "error_breakdown": dict(self.error_count_by_type),
            }
@@ -323,9 +323,11 @@ class RidReservoirStore:
            if not preserve_cols:
                return f"INSERT OR REPLACE INTO {table} ({col_list}) VALUES ({params})"
            updates = ", ".join(
                (
                    f"{c} = COALESCE(excluded.{c}, {table}.{c})"
                    if c in preserve_cols
                    else f"{c} = excluded.{c}"
                )
                for c in value_cols
            )
            return (
@@ -334,9 +336,11 @@ class RidReservoirStore:
            )
        if self.db_type == "postgresql":
            updates = ", ".join(
                (
                    f"{c} = COALESCE(EXCLUDED.{c}, {table}.{c})"
                    if c in preserve_cols
                    else f"{c} = EXCLUDED.{c}"
                )
                for c in value_cols
            )
            return (
@@ -344,9 +348,11 @@ class RidReservoirStore:
                f"ON CONFLICT ({conflict}) DO UPDATE SET {updates}"
            )
        updates = ", ".join(
            (
                f"{c} = COALESCE(VALUES({c}), {c})"
                if c in preserve_cols
                else f"{c} = VALUES({c})"
            )
            for c in value_cols
        )
        return (
@@ -155,6 +155,18 @@
        .stat-value { margin-top: 9px; font-size: 1.65rem; font-weight: 800; letter-spacing: -.04em; white-space: nowrap; }
        .stat-note { color: var(--muted); margin-top: 3px; font-size: .77rem; }
        .stat.stale { border-color: var(--red); background: rgba(204,75,55,.08); }
        .skill-panel { border: 1px solid var(--border); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: var(--surface-3); }
        .skill-head { display: flex; justify-content: space-between; align-items: baseline; gap: 12px; flex-wrap: wrap; }
        .skill-headline { margin: 8px 0 10px; font-weight: 700; font-size: .9rem; }
        .skill-headline.better { color: var(--green); }
        .skill-headline.worse { color: var(--amber); }
        .skill-table-wrap { overflow-x: auto; }
        .skill-table { border-collapse: collapse; font-size: .76rem; width: 100%; min-width: 560px; }
        .skill-table th { text-align: left; color: var(--muted); font-weight: 700; font-size: .66rem; text-transform: uppercase; letter-spacing: .06em; padding: 4px 8px; border-bottom: 1px solid var(--border); }
        .skill-table td { padding: 5px 8px; border-bottom: 1px solid var(--border); white-space: nowrap; }
        .skill-table tr.current td { font-weight: 700; }
        .skill-table td.num { text-align: right; font-variant-numeric: tabular-nums; }
        .skill-table td.dim { color: var(--muted); }
        .stat.stale .stat-value, .stat.stale .stat-note { color: var(--red); }
        .workspace { display: grid; grid-template-columns: minmax(0, 1fr) 330px; gap: 14px; min-height: 640px; }
        .map-card, .side-card { background: var(--card); border: 1px solid var(--border); border-radius: 19px; box-shadow: var(--shadow); overflow: hidden; }
@@ -417,6 +429,16 @@
            <div class="p1-peak" style="margin-top:7px" data-i18n="outlook.explainer">Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.</div>
        </div>
        <button type="button" class="zones-button" id="forecast-expand" style="display:none;margin-top:12px">Show all station forecasts ▾</button>
        <div class="skill-panel" id="skill-panel" style="display:none">
            <div class="skill-head">
                <strong data-i18n="skill.title">Is the model getting better?</strong>
                <span class="subtitle" id="skill-sub"></span>
            </div>
            <div class="skill-headline" id="skill-headline"></div>
            <p class="subtitle skill-caveat" id="skill-caveat" style="margin:-4px 0 10px"></p>
            <div class="skill-table-wrap"><table class="skill-table" id="skill-table"></table></div>
            <p class="subtitle" style="margin:8px 0 0" data-i18n="skill.explain">Every hour the deployed model's 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.</p>
        </div>
        <div class="forecast-grid" id="forecast-grid" style="display:none"></div>
    </section>
@@ -556,6 +578,26 @@
            'forecast.chip.peak': (lvl) => ` · peak ~${lvl} m`,
            'forecast.chip.heuristic': ' · heuristic fallback',
            'forecast.expand': (n) => `Show all ${n} station forecasts ▾`,
            'skill.title': 'Is the model getting better?',
            'skill.sub': (n, since) => `${n} verified 24 h forecasts for P.1 since ${since}`,
            'skill.explain': 'Every hour the deployed model\'s 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.',
            'skill.better': (v, prev, d) => `Current model ${v} is more accurate than ${prev}: peak error ${d} cm lower on the hours it has served.`,
            'skill.worse': (v, prev, d) => `Current model ${v} has a higher peak error than ${prev} so far (+${d} cm).`,
            'skill.caveat.quiet': 'All verified hours so far were below 2 m: this measures quiet-river accuracy only. The model is built and judged for flood onset (lead time before 3.70 m), which no quiet week can test — see the backtests in the documentation.',
            'skill.caveat.regime': 'Versions served different weeks; the ≥ 2 m column compares them on the hours that matter.',
            'skill.single': (v) => `Only ${v} has enough verified hours yet; the next retrain adds a row to compare.`,
            'skill.young': (v, n, min) => `${v} has ${n} verified hours; a comparison needs ${min}.`,
            'skill.none': 'No verified forecasts yet — the first appear 24 h after a model starts serving.',
            'skill.col.version': 'Model',
            'skill.col.period': 'Served',
            'skill.col.n': 'Hours',
            'skill.col.mae': 'Peak error',
            'skill.col.bias': 'Bias',
            'skill.col.pers': 'Persistence',
            'skill.col.skill': 'Skill',
            'skill.col.high': '≥ 2 m error',
            'skill.cm': (v) => `${v} cm`,
            'skill.na': '—',
            'forecast.collapse': 'Hide station forecasts ▴',
            'outlook.title': 'Chiang Mai city flood outlook · P.1 Nawarat Bridge',
            'outlook.explainer': 'Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.',
@@ -732,6 +774,26 @@
            'forecast.chip.peak': (lvl) => ` · ระดับสูงสุดประมาณ ${lvl} ม.`,
            'forecast.chip.heuristic': ' · ใช้การประมาณอย่างง่าย',
            'forecast.expand': (n) => `แสดงพยากรณ์ทั้ง ${n} สถานี ▾`,
            'skill.title': 'โมเดลแม่นยำขึ้นหรือไม่?',
            'skill.sub': (n, since) => `พยากรณ์ 24 ชม. ของ P.1 ที่ตรวจสอบแล้ว ${n} ครั้ง ตั้งแต่ ${since}`,
            'skill.explain': 'ทุกชั่วโมงระบบบันทึกค่าพยากรณ์ระดับน้ำสูงสุดใน 24 ชม. ของ P.1 ไว้ เมื่อครบ 24 ชม. จึงนำมาเทียบกับระดับน้ำจริง "ทักษะ" คือโมเดลดีกว่าการสมมติว่าระดับน้ำคงที่มากเพียงใด (0 = ไม่ดีกว่า, 1 = สมบูรณ์แบบ) โมเดลที่ฝึกใหม่ด้วยข้อมูลมากขึ้นจะปรากฏเป็นแถวใหม่ จึงเห็นได้ว่าดีขึ้นจริงหรือไม่',
            'skill.better': (v, prev, d) => `โมเดลปัจจุบัน ${v} แม่นยำกว่า ${prev}: ค่าคลาดเคลื่อนต่ำกว่า ${d} ซม. ในช่วงที่ให้บริการ`,
            'skill.worse': (v, prev, d) => `โมเดลปัจจุบัน ${v} มีค่าคลาดเคลื่อนสูงกว่า ${prev} (+${d} ซม.)`,
            'skill.caveat.quiet': 'ชั่วโมงที่ตรวจสอบทั้งหมดอยู่ต่ำกว่า 2 ม.: วัดได้เพียงความแม่นยำช่วงน้ำปกติ โมเดลถูกสร้างและประเมินสำหรับช่วงน้ำเริ่มท่วม (เวลาเตือนล่วงหน้าก่อน 3.70 ม.) ซึ่งสัปดาห์ปกติทดสอบไม่ได้ — ดูผลทดสอบย้อนหลังในเอกสาร',
            'skill.caveat.regime': 'แต่ละเวอร์ชันให้บริการคนละช่วงเวลา คอลัมน์ ≥ 2 ม. เปรียบเทียบเฉพาะชั่วโมงที่สำคัญ',
            'skill.single': (v) => `มีเพียง ${v} ที่มีข้อมูลตรวจสอบเพียงพอ การฝึกครั้งถัดไปจะเพิ่มแถวให้เปรียบเทียบ`,
            'skill.young': (v, n, min) => `${v} มีข้อมูลตรวจสอบ ${n} ชั่วโมง ต้องการอย่างน้อย ${min} เพื่อเปรียบเทียบ`,
            'skill.none': 'ยังไม่มีพยากรณ์ที่ตรวจสอบได้ — จะเริ่มมี 24 ชม. หลังโมเดลเริ่มทำงาน',
            'skill.col.version': 'โมเดล',
            'skill.col.period': 'ช่วงเวลา',
            'skill.col.n': 'ชั่วโมง',
            'skill.col.mae': 'คลาดเคลื่อน',
            'skill.col.bias': 'อคติ',
            'skill.col.pers': 'ระดับคงที่',
            'skill.col.skill': 'ทักษะ',
            'skill.col.high': 'คลาดเคลื่อน ≥ 2 ม.',
            'skill.cm': (v) => `${v} ซม.`,
            'skill.na': '—',
            'forecast.collapse': 'ซ่อนพยากรณ์รายสถานี ▴',
            'outlook.title': 'แนวโน้มน้ำท่วมเมืองเชียงใหม่ · P.1 สะพานนวรัฐ',
            'outlook.explainer': 'โอกาสที่ระดับน้ำจะถึงแต่ละระดับการท่วมตามประกาศทางการภายใน 24 ชม. — น้ำเริ่มท่วมเมืองที่ระดับ 1 (3.70 ม.) และแต่ละระดับจะท่วมพื้นที่เพิ่มขึ้น',
@@ -1989,11 +2051,59 @@
                ? t('forecast.collapse')
                : t('forecast.expand', stations.length);
            card.style.display = 'block';
            loadSkill(); // non-blocking; panel stays hidden until there is verified data
        } catch (error) {
            card.style.display = 'none';
        }
    }
    async function loadSkill() {
        const panel = $('skill-panel');
        try {
            const response = await fetch('/api/forecast/skill?station_code=P.1&horizon=24');
            if (!response.ok) throw new Error(`HTTP ${response.status}`);
            const data = await response.json();
            const versions = (data.versions || []).filter((v) => v.n > 0);
            if (!versions.length) { panel.style.display = 'none'; return; }
            const cm = (m) => m == null ? t('skill.na') : t('skill.cm', (m * 100).toFixed(1));
            const fmtDay = (v) => parseTs(v).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' });
            const total = versions.reduce((a, v) => a + v.n, 0);
            $('skill-sub').textContent = t('skill.sub', total.toLocaleString(loc()), fmtDay(versions[0].first_issued));
            const head = $('skill-headline');
            head.className = 'skill-headline';
            const cur = data.current;
            if (data.trend && cur) {
                const delta = Math.abs(data.trend.mae_delta_m * 100).toFixed(1);
                head.textContent = data.trend.better
                    ? t('skill.better', cur.model_version, data.trend.previous_version, delta)
                    : t('skill.worse', cur.model_version, data.trend.previous_version, delta);
                head.classList.add(data.trend.better ? 'better' : 'worse');
            } else if (cur && cur.enough_data) {
                head.textContent = t('skill.single', cur.model_version);
            } else if (cur) {
                head.textContent = t('skill.young', cur.model_version, cur.n, data.min_verified);
            } else head.textContent = t('skill.none');
            const anyHigh = versions.some((v) => v.above_2m_n > 0);
            const compared = versions.filter((v) => v.enough_data).length > 1;
            $('skill-caveat').textContent = !anyHigh ? t('skill.caveat.quiet') : compared ? t('skill.caveat.regime') : '';
            const cols = ['version', 'period', 'n', 'mae', 'bias', 'pers', 'skill', 'high'];
            const rows = versions.map((v) => `<tr class="${v === cur ? 'current' : ''}${v.enough_data ? '' : ' young'}">`
                + `<td>${escapeHtml(v.model_version)}</td>`
                + `<td class="dim">${fmtDay(v.first_issued)} – ${fmtDay(v.last_issued)}</td>`
                + `<td class="num">${v.n.toLocaleString(loc())}</td>`
                + `<td class="num">${cm(v.mae_m)}</td>`
                + `<td class="num">${v.bias_m == null ? t('skill.na') : (v.bias_m >= 0 ? '+' : '') + (v.bias_m * 100).toFixed(1)}</td>`
                + `<td class="num dim">${cm(v.persistence_mae_m)}</td>`
                + `<td class="num">${v.skill == null ? t('skill.na') : v.skill.toFixed(2)}</td>`
                + `<td class="num">${v.above_2m_n ? `${cm(v.above_2m_mae_m)} <span class="dim">(${v.above_2m_n})</span>` : t('skill.na')}</td>`
                + '</tr>').join('');
            $('skill-table').innerHTML = `<thead><tr>${cols.map((c) => `<th>${escapeHtml(t('skill.col.' + c))}</th>`).join('')}</tr></thead><tbody>${rows}</tbody>`;
            panel.style.display = 'block';
        } catch (error) {
            panel.style.display = 'none';
        }
    }
    async function loadDbStats() {
        const strip = $('db-stats');
        try {
@@ -864,7 +864,7 @@ def _hii_rows(sql: str, params: Dict[str, Any]) -> List[Dict[str, Any]]:
 # flood, slightly stale readings with a visible timestamp beat an error page.
 HII_CACHE: Dict[str, Any] = {}
 HII_CACHE_LOCK = Lock()
-_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock()}
+_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock(), "skill": Lock()}
 LATEST_CACHE: Dict[str, Any] = {}
 LATEST_CACHE_LOCK = Lock()
 _LATEST_COMPUTE_LOCK = Lock()
@@ -1230,6 +1230,56 @@ async def get_forecast_history(
    return await asyncio.to_thread(store.fetch, station_code, start_dt, end_dt, horizon)
@app.get("/api/forecast/skill")
 async def get_forecast_skill(
    response: Response,
    station_code: str = Query("P.1"),
    horizon: int = Query(24, ge=1, le=48),
 ):
    """Is the model getting better? Issued forecasts verified against what the
    river then did, per model version, with a persistence baseline.
    Read from forecast_history (what each deployed version predicted, hourly)
    joined to water_measurements; no retraining involved. Cached like the HII
    feeds because the join is a few hundred correlated subqueries.
    """
    increment_counter("api_requests", labels={"endpoint": "forecast_skill"})
    store = app_state.get("forecast_store")
    if not store:
        return {
            "station_code": station_code,
            "horizon_hours": horizon,
            "versions": [],
            "current": None,
            "trend": None,
        }
    def compute():
        from .ml import skill
        if not store.engine and not store.connect():
            return {
                "station_code": station_code,
                "horizon_hours": horizon,
                "versions": [],
                "current": None,
                "trend": None,
            }
        return skill.compute_skill(store.engine, store.db_type, station_code, horizon)
    payload, stale = await _cached_swr(
        HII_CACHE,
        HII_CACHE_LOCK,
        _HII_COMPUTE_LOCKS["skill"],
        f"skill:{station_code}:{horizon}",
        max(Config.HII_CACHE_TTL_SECONDS, 900),
        compute,
    )
    if stale:
        response.headers["X-Data-Stale"] = "true"
    return payload
@app.get("/measurements/latest", response_model=List[MeasurementResponse])
 async def get_latest_measurements(response: Response, limit: int = 100):
    """Get latest measurements from all stations"""
@@ -1313,9 +1363,7 @@ async def get_database_stats():
        from sqlalchemy import text
        with engine.connect() as conn:
-            return conn.execute(
+            return conn.execute(text("""
                text(
                    """
                    SELECT (SELECT COUNT(*) FROM hii_rainfall) AS rain_n,
                           (SELECT COUNT(*) FROM hii_waterlevel) AS wl_n,
                           (SELECT COUNT(*) FROM hii_rain_stations) AS rain_s,
@@ -1324,9 +1372,7 @@ async def get_database_stats():
                           (SELECT MAX(timestamp) FROM hii_rainfall) AS rain_hi,
                           (SELECT MIN(timestamp) FROM hii_waterlevel) AS wl_lo,
                           (SELECT MAX(timestamp) FROM hii_waterlevel) AS wl_hi
-                    """
+                    """)).one()
                )
            ).one()
    def compute():
        # Heavy: full-table counts and coverage over ~1.7M rows. Runs at most
@@ -0,0 +1,93 @@
 """Forecast skill verification: issued forecasts vs observed peaks (sqlite)."""
 import datetime
 import pytest
 from sqlalchemy import create_engine, text
 from src.ml import skill
@pytest.fixture
 def engine(tmp_path):
    eng = create_engine(f"sqlite:///{tmp_path / 'skill.db'}")
    with eng.begin() as c:
        c.execute(text("CREATE TABLE stations (id INTEGER PRIMARY KEY, station_code TEXT)"))
        c.execute(text("INSERT INTO stations VALUES (1, 'P.1')"))
        c.execute(
            text(
                "CREATE TABLE water_measurements (timestamp DATETIME, station_id INTEGER, water_level REAL)"
            )
        )
        c.execute(
            text(
                "CREATE TABLE forecast_history (as_of TIMESTAMP, station_code TEXT, horizon_hours INTEGER, "
                "predicted_max_level REAL, p_warning REAL, p_danger REAL, current_level REAL, "
                "model_version TEXT, source TEXT)"
            )
        )
    return eng
 def _fill(engine, start, hours, level_fn, forecasts):
    """hours of hourly observations from `start`, plus (as_of_offset_h, version, pred) rows."""
    with engine.begin() as c:
        for h in range(hours):
            ts = start + datetime.timedelta(hours=h)
            c.execute(
                text("INSERT INTO water_measurements VALUES (:t, 1, :l)"),
                {"t": ts, "l": level_fn(h)},
            )
        for off, version, pred in forecasts:
            ts = start + datetime.timedelta(hours=off)
            c.execute(
                text(
                    "INSERT INTO forecast_history VALUES (:t, 'P.1', 24, :p, 0, 0, :cur, :v, 'model')"
                ),
                {"t": ts, "p": pred, "cur": level_fn(off), "v": version},
            )
 def test_skill_per_version_and_trend(engine):
    start = datetime.datetime(2026, 8, 1)
    # river: flat 1.5 m, with a bump to 2.4 m around hour 100
    level = lambda h: 2.4 if 96 <= h <= 104 else 1.5
    forecasts = []
    # old version: always predicts 1.5 (persistence-like, misses the bump)
    for off in range(0, 60):
        forecasts.append((off, "hgb-v2+aaaaaaa", 1.5))
    # new version: predicts 1.5 normally and 2.3 ahead of the bump
    for off in range(60, 200):
        pred = 2.3 if 72 <= off <= 104 else 1.5
        forecasts.append((off, "hgb-v3+bbbbbbb", pred))
    _fill(engine, start, 260, level, forecasts)
    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=300))
    assert [v["model_version"] for v in out["versions"]] == ["hgb-v2+aaaaaaa", "hgb-v3+bbbbbbb"]
    old, new = out["versions"]
    assert old["n"] == 60 and old["enough_data"]
    assert new["n"] == 140 and new["enough_data"]
    # the old version issued only on flat hours: perfect there, no bump rows
    assert old["mae_m"] == 0.0 and old["above_2m_n"] == 0
    # the new version saw the bump: nonzero MAE but positive skill vs persistence
    assert new["above_2m_n"] > 0
    assert new["skill"] is not None and new["skill"] > 0
    assert out["current"]["model_version"] == "hgb-v3+bbbbbbb"
    assert out["trend"]["previous_version"] == "hgb-v2+aaaaaaa"
    assert out["trend"]["better"] is False  # honest: old had an easier period
 def test_skill_requires_full_window(engine):
    start = datetime.datetime(2026, 8, 1)
    # forecasts issued at the very end have no observed window yet
    _fill(engine, start, 30, lambda h: 1.5, [(o, "hgb-v3+ccccccc", 1.5) for o in range(0, 30)])
    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=30))
    # only as_of <= now-24h AND with >= 18 observed hours in the window count
    assert out["versions"] and out["versions"][0]["n"] == 7  # as_of 0..6 h: <= now-24h with >= 18 observed hours
    assert out["versions"][0]["enough_data"] is False
    assert out["trend"] is None
 def test_skill_empty(engine):
    out = skill.compute_skill(engine, "sqlite", "P.1", 24)
    assert out["versions"] == [] and out["current"] is None and out["trend"] is None