feat: recalibrate P.77/P.75 thresholds; capacity guard on alerts

The first ntfy cycle announced "Warning level at P.77" at 3.02 m. That
gauge's 2.85 m threshold sat below its own dry-season baseline (2.6-2.7 m
at 8-14 % channel capacity): P.77 had been "above warning" for 761 of the
last 2 146 hours, at 22 % capacity. Across 2018-2024, 75-85 % capacity
reads 3.35-4.57 m and 95-105 % reads 4.27-5.08 m; set 4.30 / 4.90. P.75
moved 2.75/3.50 -> 3.20/3.65 on the same evidence (2024: 3.45 / 3.72).
The predictor already handles changed thresholds (regression-derived
probabilities until the Oct 1 retrain).

Second line of defence in notify.py: a clear->alert transition is only
announced when RID's discharge_percent for the reading is >= 60 %, so a
re-rated or datum-shifted gauge cannot page subscribers again. P.1 is
exempt (its stages come from the inundation map, not capacity); readings
without a capacity figure fall back to level only; the all-clear edge is
never blocked. 3 tests.

.env.example

@@ -84,6 +84,19 @@ SMTP_PORT=587
 SMTP_USERNAME=
 SMTP_PASSWORD=
 
+# Public push notifications via self-hosted ntfy (https://ntfy.sh, single binary).
+# Leave NTFY_SERVER empty to disable. Topics published: <prefix>-<station>-warning,
+# <prefix>-<station>-danger, <prefix>-warning, <prefix>-danger, <prefix>-p1-outlook,
+# <prefix>-status. See docs/NOTIFICATIONS.md.
+NTFY_SERVER=
+# Where the monitor POSTs (defaults to NTFY_SERVER). Use the local ntfy
+# address (loopback or Tailscale IP) so publishing does not depend on
+# DNS / the reverse proxy being up.
+NTFY_PUBLISH_URL=
+NTFY_TOPIC_PREFIX=ping
+NTFY_TOKEN=
+PUBLIC_URL=https://water.buildfor.life/
+
 # Matrix Alerting Configuration
 MATRIX_HOMESERVER=https://matrix.org
 MATRIX_ACCESS_TOKEN=

.gitea/workflows/ci.yml

@@ -38,7 +38,7 @@ jobs:
       - name: Install tools
         run: |
           python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore black==23.11.0 isort==5.12.0 flake8==6.1.0
+          pip install --root-user-action=ignore black==26.5.1 isort==5.12.0 flake8==6.1.0
 
       - name: black
         run: black --check --diff src/ *.py
@@ -67,7 +67,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip --root-user-action=ignore
           pip install --root-user-action=ignore -r requirements.txt
-          pip install --root-user-action=ignore pytest==7.4.3 pytest-asyncio==0.21.1
+          pip install --root-user-action=ignore pytest==9.1.1 pytest-asyncio==0.21.1
 
       - name: pytest
         env:

.gitea/workflows/security.yml

@@ -1,293 +1,109 @@
-name: Security & Dependency Updates
+name: Security
+
+# Two gates that can actually fail, plus one report:
+#   - pip-audit against requirements.txt: any known vulnerability in a runtime
+#     dependency fails the job (dev-only tools are reported, not gated)
+#   - bandit on src/: HIGH severity findings fail; medium/low are listed.
+#     B104 (bind 0.0.0.0) is skipped: the service is meant to listen on all
+#     interfaces behind Cloudflare/Caddy.
+#   - pip-licenses report as an artifact (informational; the project is MIT
+#     and its runtime deps are MIT/BSD/Apache/PSF)
+# The old file ran safety/bandit/semgrep with `|| true` and could not go red.
 
 on:
   schedule:
-    # Run security scans daily at 3 AM UTC
-    - cron: "0 3 * * *"
+    - cron: "0 3 * * 1" # weekly, Monday 03:00 UTC
   workflow_dispatch:
   push:
     paths:
       - "requirements*.txt"
-      - "Dockerfile"
+      - "pyproject.toml"
+      - "uv.lock"
+      - "src/**/*.py"
       - ".gitea/workflows/security.yml"
+  pull_request:
+    paths:
+      - "requirements*.txt"
+      - "pyproject.toml"
+      - "src/**/*.py"
 
 env:
   PYTHON_VERSION: "3.11"
-  # GitHub token for better rate limits and authentication
-  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 
 jobs:
-  # Dependency vulnerability scan
-  dependency-scan:
-    name: Dependency Security Scan
+  dependencies:
+    name: Dependency vulnerabilities
     runs-on: ubuntu-latest
-
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
+      - uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 
-      - name: Install dependencies
+      - name: Install pip-audit
         run: |
           python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore safety bandit semgrep
+          pip install --root-user-action=ignore pip-audit
 
-      - name: Run Safety check
-        run: |
-          safety check -r requirements.txt --json --output safety-report.json || true
-          safety check -r requirements-dev.txt --json --output safety-dev-report.json || true
+      - name: Runtime dependencies (gate)
+        run: pip-audit -r requirements.txt --strict --desc on
 
-      - name: Run Bandit security scan
-        run: |
-          bandit -r src/ -f json -o bandit-report.json || true
+      - name: Dev dependencies (report only)
+        run: pip-audit -r requirements-dev.txt --desc on || echo "::warning::dev-only dependency advisories above"
 
-      - name: Run Semgrep security scan
-        run: |
-          semgrep --config=auto src/ --json --output=semgrep-report.json || true
-
-      - name: Upload security reports
-        uses: actions/upload-artifact@v3
-        with:
-          name: security-reports-${{ github.run_number }}
-          path: |
-            safety-report.json
-            safety-dev-report.json
-            bandit-report.json
-            semgrep-report.json
-
-      - name: Check for critical vulnerabilities
-        run: |
-          echo "Checking for critical vulnerabilities..."
-
-          # Check Safety results
-          if [ -f safety-report.json ]; then
-            critical_count=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null || echo "0")
-            if [ "$critical_count" -gt 0 ]; then
-              echo "Found $critical_count dependency vulnerabilities"
-              jq '.vulnerabilities[] | "- \(.package_name) \(.installed_version): \(.vulnerability_id)"' safety-report.json
-            else
-              echo "No dependency vulnerabilities found"
-            fi
-          fi
-
-          # Check Bandit results
-          if [ -f bandit-report.json ]; then
-            high_severity=$(jq '.results[] | select(.issue_severity == "HIGH") | length' bandit-report.json 2>/dev/null | wc -l)
-            if [ "$high_severity" -gt 0 ]; then
-              echo "Found $high_severity high-severity security issues"
-            else
-              echo "No high-severity security issues found"
-            fi
-          fi
-
-  # License compliance check
-  license-check:
-    name: License Compliance
+  code:
+    name: Static analysis
     runs-on: ubuntu-latest
-
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
+      - uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 
-      - name: Install pip-licenses
+      - name: Install bandit
         run: |
           python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-licenses
-          pip install --root-user-action=ignore -r requirements.txt
+          pip install --root-user-action=ignore bandit
 
-      - name: Check licenses
+      - name: bandit (HIGH fails; medium/low listed)
         run: |
-          echo "Checking dependency licenses..."
+          bandit -r src/ -q --skip B104 -ll -ii || true
+          bandit -r src/ -q --skip B104 --severity-level high --confidence-level medium
+
+  licenses:
+    name: License report
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: pip
+          cache-dependency-path: requirements.txt
+
+      # A fresh venv, not the runner's site-packages: the report must list the
+      # project's runtime deps, not whatever the runner image or a previous
+      # workflow happened to leave installed (semgrep once showed up here).
+      - name: Install into a clean venv
+        run: |
+          python -m venv .lic && . .lic/bin/activate
+          pip install --upgrade pip --root-user-action=ignore
+          pip install --root-user-action=ignore -r requirements.txt pip-licenses
+
+      - name: Report
+        run: |
+          . .lic/bin/activate
+          pip-licenses --format=markdown --with-urls --output-file=licenses.md
           pip-licenses --format=json --output-file=licenses.json
-          pip-licenses --format=markdown --output-file=licenses.md
+          echo "Copyleft licenses among runtime deps (informational; LGPL is fine to link from MIT):"
+          pip-licenses --format=plain --ignore-packages pip-licenses | grep -iE 'GPL|AGPL|LGPL' || echo "  none"
 
-          # Check for problematic licenses
-          problematic_licenses=("GPL" "AGPL" "LGPL")
-
-          for license in "${problematic_licenses[@]}"; do
-            if grep -i "$license" licenses.json; then
-              echo "Found potentially problematic license: $license"
-            fi
-          done
-
-          echo "License check completed"
-
-      - name: Upload license report
-        uses: actions/upload-artifact@v3
+      - uses: actions/upload-artifact@v3
         with:
-          name: license-report-${{ github.run_number }}
+          name: licenses-${{ github.run_number }}
           path: |
-            licenses.json
             licenses.md
-
-  # Dependency update check
-  dependency-update:
-    name: Check for Dependency Updates
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
-
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-
-      - name: Install pip-check-updates equivalent
-        run: |
-          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-review
-
-      - name: Check for outdated packages
-        run: |
-          echo "Checking for outdated packages..."
-          pip install --root-user-action=ignore -r requirements.txt
-          pip list --outdated --format=json > outdated-packages.json || true
-
-          if [ -s outdated-packages.json ]; then
-            echo "Outdated packages found:"
-            cat outdated-packages.json | jq -r '.[] | "- \(.name): \(.version) -> \(.latest_version)"'
-          else
-            echo "All packages are up to date"
-          fi
-
-      - name: Upload dependency reports
-        uses: actions/upload-artifact@v3
-        with:
-          name: dependency-reports-${{ github.run_number }}
-          path: |
-            outdated-packages.json
-
-  # Code quality metrics
-  code-quality:
-    name: Code Quality Metrics
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
-
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-
-      - name: Install quality tools
-        run: |
-          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore radon xenon vulture
-          pip install --root-user-action=ignore -r requirements.txt
-
-      - name: Calculate code complexity
-        run: |
-          echo "Calculating code complexity..."
-          radon cc src/ --json > complexity-report.json
-          radon mi src/ --json > maintainability-report.json
-
-          echo "Complexity Summary:"
-          radon cc src/ --average
-
-          echo "Maintainability Summary:"
-          radon mi src/
-
-      - name: Find dead code
-        run: |
-          echo "Checking for dead code..."
-          vulture src/ --json > dead-code-report.json || true
-
-      - name: Check for code smells
-        run: |
-          echo "Checking for code smells..."
-          xenon --max-absolute B --max-modules A --max-average A src/ || true
-
-      - name: Upload quality reports
-        uses: actions/upload-artifact@v3
-        with:
-          name: code-quality-reports-${{ github.run_number }}
-          path: |
-            complexity-report.json
-            maintainability-report.json
-            dead-code-report.json
-
-  # Security summary
-  security-summary:
-    name: Security Summary
-    runs-on: ubuntu-latest
-    needs: [dependency-scan, license-check, code-quality]
-    if: always()
-
-    steps:
-      - name: Download all artifacts
-        uses: actions/download-artifact@v3
-
-      - name: Generate security summary
-        run: |
-          echo "# Security Scan Summary" > security-summary.md
-          echo "" >> security-summary.md
-          echo "**Scan Date:** $(date -u)" >> security-summary.md
-          echo "**Repository:** ${{ github.repository }}" >> security-summary.md
-          echo "**Commit:** ${{ github.sha }}" >> security-summary.md
-          echo "" >> security-summary.md
-
-          echo "## Results" >> security-summary.md
-          echo "" >> security-summary.md
-
-          # Dependency scan results
-          if [ -f security-reports-*/safety-report.json ]; then
-            vuln_count=$(jq '.vulnerabilities | length' security-reports-*/safety-report.json 2>/dev/null || echo "0")
-            if [ "$vuln_count" -eq 0 ]; then
-              echo "- Dependency Scan: No vulnerabilities found" >> security-summary.md
-            else
-              echo "- Dependency Scan: $vuln_count vulnerabilities found" >> security-summary.md
-            fi
-          else
-            echo "- Dependency Scan: Results not available" >> security-summary.md
-          fi
-
-          # Docker scan results (removed Trivy)
-          echo "- Docker Scan: Skipped (Trivy removed)" >> security-summary.md
-
-          # License check results
-          if [ -f license-report-*/licenses.json ]; then
-            echo "- License Check: Completed" >> security-summary.md
-          else
-            echo "- License Check: Results not available" >> security-summary.md
-          fi
-
-          # Code quality results
-          if [ -f code-quality-reports-*/complexity-report.json ]; then
-            echo "- Code Quality: Analyzed" >> security-summary.md
-          else
-            echo "- Code Quality: Results not available" >> security-summary.md
-          fi
-
-          echo "" >> security-summary.md
-          echo "## Detailed Reports" >> security-summary.md
-          echo "" >> security-summary.md
-          echo "Detailed reports are available in the workflow artifacts." >> security-summary.md
-
-          cat security-summary.md
-
-      - name: Upload security summary
-        uses: actions/upload-artifact@v3
-        with:
-          name: security-summary-${{ github.run_number }}
-          path: security-summary.md
+            licenses.json

.pre-commit-config.yaml

@@ -19,7 +19,7 @@ repos:
 
   # Python code formatting with Black
   - repo: https://github.com/psf/black
-    rev: 23.11.0
+    rev: 26.5.1
     hooks:
       - id: black
         language_version: python3

CLAUDE.md

@@ -0,0 +1,40 @@
+# CLAUDE.md
+
+Guidance for AI coding agents working in this repository.
+
+## What this is
+
+Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
+API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
+Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
+`water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
+
+## Rules
+
+- Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
+- `make format` (black 88 / isort black profile, config in pyproject.toml) before
+  committing; CI fails on formatting. `make test` must stay green — tests are
+  synthetic-data only, never add one that needs the DB or network.
+- Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
+  parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
+- Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
+  and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive
+  or negative, in `docs/FLOOD_FORECASTING.md` section 5. Do not change what is deployed
+  (`rise_rain` / hgb-v3) without a harness result that beats it on lead.
+- `train_all()` must never silently produce a gauge-only (v2) model; the guard that
+  raises `RainUnavailableError` stays.
+- No `git add -A`: zero-byte shell-accident files (`#`, `$(wc`, ...) have been committed
+  before. Stage files by name.
+- Do not add Co-Authored-By trailers.
+- The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
+  table: every user-visible string needs both languages.
+
+## Where things are
+
+- `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
+  `src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
+  `rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
+- `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
+  promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
+- `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
+  the source catalog.

README.md

@@ -1,494 +1,157 @@
-# Northern Thailand Ping River Monitor 🏔️
+# Northern Thailand Ping River Monitor
 
-A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.
+Live water levels, discharge, rainfall and machine-learning flood forecasts for the
+Ping River basin around Chiang Mai. Collects hourly gauge data from public sources,
+keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST
+API, and 6/12/24-hour flood-risk forecasts per gauge.
 
-**Live dashboard: [water.buildfor.life](https://water.buildfor.life/)** — water levels, discharge, rainfall and 6/12/24 h flood forecasts for Chiang Mai, in English and Thai. Background: [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
+**Live: [water.buildfor.life](https://water.buildfor.life/)** · API reference at
+[/docs](https://water.buildfor.life/docs) · built by [buildfor.life](https://buildfor.life)
+after the [October 2024 flood](https://buildfor.life/blog/chiang-mai-flood-2024/) —
+background in [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
 
-[![CI/CD](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Documentation](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-0.104+-green.svg)](https://fastapi.tiangolo.com) [![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Version](https://img.shields.io/badge/Version-v3.1.3-blue.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/releases)
+[![CI](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
+[![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
+[![Docs](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
+[![Python 3.11](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
 
-## 🌟 Features
+## What it does
 
-### 📊 **Real-time Data Collection**
-- **16 Monitoring Stations** across Thailand
-- **15-minute Collection Frequency** with intelligent scheduling
-- **Automatic Gap Filling** for missing historical data
-- **Data Validation** and error recovery mechanisms
-- **Rate Limiting** to prevent API abuse
+- **Collects** hourly water level and discharge from 16 Royal Irrigation Department
+  (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly
+  rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment
+  rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and
+  its quirks: [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md).
+- **Fills gaps.** The raw RID grid had readings for ~56 % of hours; a full-history
+  re-fetch plus HII cross-fill brought it to ~93 %. `GET /api/stats` reports the
+  current figure.
+- **Forecasts.** Per gauge and horizon, a gradient-boosted model predicts the rise
+  within 6/12/24 h and the probability of crossing the station's warning and danger
+  levels. Trained on the monitor's own history plus catchment rain; evaluated
+  rolling-origin, event by event. On the October 2024 record flood, trained only on
+  data through August 2024, the first alert came **13 hours before** P.1 crossed
+  3.70 m. Everything about the model, including what did not work:
+  [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md).
+- **Shows it.** A Leaflet map with the river drawn as OSM geometry and styled by live
+  discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the
+  forecast card, a replay of the 2024 flood, English/Thai, light/dark.
+- **Notifies.** Public push alerts over a self-hosted [ntfy](https://ntfy.sh): one
+  message when a gauge crosses its warning or danger level, one all-clear on the
+  way down, an opt-in early-warning topic from the model, nothing in between.
+  Subscribe from the free app, no account. Matrix room alerts for a team are
+  also supported.
 
-### 🌐 **Web API Interface (NEW!)**
-- **FastAPI-powered REST API** with interactive documentation
-- **Station Management** - Add, update, and remove monitoring stations
-- **Real-time health monitoring** and system status
-- **Manual data collection triggers** via web interface
-- **Comprehensive metrics** and performance monitoring
-- **CORS support** for web applications
+## Quick start
 
-### 🗄️ **Multi-Database Support**
-- **VictoriaMetrics** (Recommended) - High-performance time-series
-- **InfluxDB** - Purpose-built time-series database
-- **PostgreSQL + TimescaleDB** - Relational with time-series optimization
-- **MySQL** - Traditional relational database
-- **SQLite** - Local development and testing
-
-### 🗺️ **Geolocation Support**
-- **Grafana Geomap** integration ready
-- **GPS coordinates** and geohash support
-- **Interactive mapping** of water stations
-
-### 📈 **Visualization & Monitoring**
-- **Pre-built Grafana dashboards**
-- **Real-time alerts** and notifications
-- **Historical trend analysis**
-- **Built-in metrics collection** (counters, gauges, histograms)
-- **Health checks** for database, API, and system resources
-
-### 🚀 **Production Ready**
-- **Docker containerization** with multi-service support
-- **Systemd service** configuration
-- **HTTPS support** with SSL certificates
-- **Comprehensive logging** with rotation and colored output
-- **Type safety** with Pydantic models and type hints
-- **Custom exception handling** for better error management
-
-## 🚀 Quick Start
-
-### Prerequisites
-- Python 3.9 or higher
-- Internet connection for data fetching
-- Database server (optional - SQLite works out of the box)
-
-### Installation
+Python **3.11** (3.13 breaks the pinned `psycopg2-binary`), PostgreSQL for anything
+beyond a quick look, [uv](https://docs.astral.sh/uv/).
 
 ```bash
-# Clone the repository
 git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 cd Northern-Thailand-Ping-River-Monitor
-
-# Quick setup with Make
-make dev-setup
-
-# Or manual setup:
-python -m venv venv
-source venv/bin/activate  # Windows: venv\Scripts\activate
-pip install -r requirements.txt
-cp .env.example .env
+uv sync --python 3.11
+cp .env.example .env            # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
+uv run python run.py --web-api  # dashboard + API on http://localhost:8000
 ```
 
-### Basic Usage
+`DB_TYPE=sqlite` works for the dashboard and API; the forecasting path expects the
+PostgreSQL history.
 
 ```bash
-# Test run with SQLite (default)
-make run-test
-# or: python run.py --test
-
-# Run continuous monitoring
-make run
-# or: python run.py
-
-# Start web API server (NEW!)
-make run-api
-# or: python run.py --web-api
-
-# Run all tests
-make test
-
-# Demo different databases
-python src/demo_databases.py
+uv run python run.py --status          # collector status
+uv run python run.py --test            # one collection cycle
+uv run python run.py --fill-gaps 7     # re-fetch the last 7 days from RID
+uv run python run.py --collect-hii     # one ThaiWater/HII collection cycle
+uv run python run.py --alert-check     # evaluate thresholds, notify Matrix
+uv run python scripts/train_flood_model.py --stations all   # retrain (~12 min)
+make test                              # pytest, synthetic data, no network
+make format                            # black + isort (the CI contract)
 ```
 
-### 🌐 Web API Interface (NEW!)
+## API
 
-The system now includes a comprehensive FastAPI web interface:
+Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
+Asia/Bangkok wall-clock without an offset suffix.
+
+| Endpoint | Returns |
+| --- | --- |
+| `GET /stations` | The 16 RID gauges: code, Thai/English names, coordinates |
+| `GET /measurements/latest?limit=N` | Newest reading per station |
+| `GET /measurements/history/{code}?hours=N` | Hourly history; or `?start=YYYY-MM-DD&end=YYYY-MM-DD`; `limit` ≤ 100000 |
+| `GET /forecast` | Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
+| `GET /api/forecast/history/{code}?hours=N&horizon=24` | Forecasts as issued, for auditing lead time after the fact |
+| `GET /api/hii/rainfall/latest`, `/api/hii/waterlevel/latest` | Latest ThaiWater/HII gauge readings |
+| `GET /api/hii/rainfall/catchment?days=N` | HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
+| `GET /api/forecast/skill?station_code=P.1` | Issued forecasts vs what happened, per deployed model version |
+| `GET /api/notifications` | ntfy server and topic names for the subscribe panel |
+| `GET /api/stats` | Row counts per source, date range, coverage |
+| `GET /health` | DB / upstream / memory checks |
+
+Interactive reference with schemas: [water.buildfor.life/docs](https://water.buildfor.life/docs).
+Responses are cached briefly server-side; poll no faster than once a minute — the data
+changes hourly.
+
+## Deployment
+
+Production is a systemd unit on a small VPS behind Cloudflare, updated by `git pull`.
+`scripts/install.sh` (run as root from a checkout) creates the `water-monitor` user,
+deploys to `/opt/thailand-water-monitor`, runs `uv sync` into `.venv`, installs
+`water-monitor.service` and the monthly `water-monitor-retrain.timer`.
 
 ```bash
-# Start the web API
-python run.py --web-api
-
-# Access the API at:
-# - Dashboard: http://localhost:8000
-# - Interactive docs: http://localhost:8000/docs
-# - Health check: http://localhost:8000/health
-# - Latest data: http://localhost:8000/measurements/latest
-```
-
-**Key API Endpoints:**
-- `GET /` - Web dashboard
-- `GET /health` - System health status
-- `GET /metrics` - Application metrics
-- `GET /stations` - List all monitoring stations
-- `POST /stations` - Add new monitoring station
-- `PUT /stations/{id}` - Update station information
-- `DELETE /stations/{id}` - Remove monitoring station
-- `GET /measurements/latest` - Latest measurements
-- `GET /measurements/station/{code}` - Station-specific data
-- `POST /scrape/trigger` - Trigger manual data collection
-
-## 📊 Station Information
-
-The system monitors **16 water stations** along the Ping River Basin in Northern Thailand:
-
-| Station | Thai Name | English Name | Location |
-|---------|-----------|--------------|----------|
-| P.1 | สะพานนวรัฐ | Nawarat Bridge | Nakhon Sawan |
-| P.5 | สะพานท่านาง | Tha Nang Bridge | - |
-| P.20 | บ้านเชียงดาว | Ban Chiang Dao | Chiang Mai |
-| P.21 | บ้านริมใต้ | Ban Rim Tai | - |
-| P.4A | บ้านแม่แตง | Ban Mae Taeng | Chiang Mai |
-| P.67 | บ้านแม่แต | Ban Tae | - |
-| P.75 | บ้านช่อแล | Ban Chai Lat | - |
-| P.76 | บ้านแม่อีไฮ | Banb Mae I Hai | - |
-| P.77 | บ้านสบแม่สะป๊วด | Baan Sop Mae Sapuord | - |
-| P.81 | บ้านโป่ง | Ban Pong | - |
-| P.82 | บ้านสบวิน | Ban Sob win | - |
-| P.84 | บ้านพันตน | Ban Panton | - |
-| P.85 | บ้านหล่ายแก้ว | Baan Lai Kaew | - |
-| P.87 | บ้านป่าซาง | Ban Pa Sang | - |
-| P.92 | บ้านเมืองกึ๊ด | Ban Muang Aut | - |
-| P.103 | สะพานวงแหวนรอบ 3 | Ring Bridge 3 | Bangkok |
-
-### Data Metrics
-- **Water Level**: Measured in meters (m)
-- **Discharge**: Flow rate in cubic meters per second (cms)
-- **Discharge Percentage**: Relative to station capacity
-- **Timestamp**: Thai time (UTC+7) with Buddhist calendar support
-
-## 🗄️ Database Configuration
-
-### VictoriaMetrics (Recommended)
-
-**High-performance time-series database with excellent compression and query speed.**
-
-```bash
-# Environment variables
-export DB_TYPE=victoriametrics
-export VM_HOST=localhost
-export VM_PORT=8428
-
-# Quick start with Docker
-docker run -d \
-  --name victoriametrics \
-  -p 8428:8428 \
-  -v victoria-metrics-data:/victoria-metrics-data \
-  victoriametrics/victoria-metrics:latest \
-  --storageDataPath=/victoria-metrics-data \
-  --retentionPeriod=2y \
-  --httpListenAddr=:8428
-```
-
-### Complete Stack with Grafana
-
-```bash
-# Start the complete monitoring stack
-docker-compose -f docker-compose.victoriametrics.yml up -d
-
-# Access Grafana at http://localhost:3000
-# Username: admin, Password: admin_password
-```
-
-### Other Database Options
-
-<details>
-<summary>InfluxDB Configuration</summary>
-
-```bash
-export DB_TYPE=influxdb
-export INFLUX_HOST=localhost
-export INFLUX_PORT=8086
-export INFLUX_DATABASE=water_monitoring
-export INFLUX_USERNAME=water_user
-export INFLUX_PASSWORD=your_password
-```
-</details>
-
-<details>
-<summary>PostgreSQL Configuration</summary>
-
-```bash
-export DB_TYPE=postgresql
-export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring
-```
-</details>
-
-<details>
-<summary>MySQL Configuration</summary>
-
-```bash
-export DB_TYPE=mysql
-export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring
-```
-</details>
-
-## 📈 Grafana Dashboards
-
-### Pre-built Dashboard Features
-- **Real-time water levels** across all stations
-- **Historical trends** and patterns
-- **Discharge monitoring** with percentage indicators
-- **Station status** and health monitoring
-- **Geomap visualization** of station locations
-- **Alert thresholds** for critical water levels
-
-### Sample Queries
-
-**VictoriaMetrics/Prometheus:**
-```promql
-# Current water levels
-water_level
-
-# High discharge alerts
-water_discharge_percent > 80
-
-# Station-specific data
-water_level{station_code="P.1"}
-```
-
-**SQL Databases:**
-```sql
--- Latest readings from all stations
-SELECT s.station_code, s.english_name, m.water_level, m.discharge
-FROM stations s
-JOIN water_measurements m ON s.id = m.station_id
-WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);
-```
-
-## 🚀 Production Deployment
-
-### Docker Deployment
-
-```bash
-# Build the image
-docker build -t thailand-water-monitor .
-
-# Run with environment variables
-docker run -d \
-  --name water-monitor \
-  -e DB_TYPE=victoriametrics \
-  -e VM_HOST=victoriametrics \
-  thailand-water-monitor
-```
-
-### Systemd Service (Linux)
-
-The install script sets everything up: a dedicated `water-monitor` system user,
-a deploy to `/opt/thailand-water-monitor`, a uv-managed virtualenv, and the
-enabled systemd unit.
-
-```bash
-# From a checkout of the repo, as root:
 sudo bash scripts/install.sh
-
-# Then start and check:
 sudo systemctl start water-monitor.service
-systemctl status water-monitor.service
+systemctl list-timers water-monitor-retrain.timer
 ```
 
-Fill in `/opt/thailand-water-monitor/.env` (Matrix token/room, DB settings)
-before starting if the script reports it is missing.
+The retrain timer runs `scripts/retrain.sh`, which trains into `models/.staging`,
+refuses to promote anything that is not a rain-enabled (`hgb-v3+`) set covering the
+expected stations, and renames the bundles into place. Details and the operations
+runbook: [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) sections 6–8.
 
-<details>
-<summary>Manual setup (if you prefer not to use the script)</summary>
-
-```bash
-sudo useradd --system --no-create-home --shell /usr/sbin/nologin water-monitor
-uv sync --python 3.11            # creates .venv, the interpreter both units run
-sudo cp scripts/water-monitor.service scripts/water-monitor-retrain.service scripts/water-monitor-retrain.timer /etc/systemd/system/
-sudo systemctl enable --now water-monitor.service water-monitor-retrain.timer
-```
-</details>
-
-
-## 🔧 Command Line Tools
-
-### Main Application
-```bash
-python src/water_scraper_v3.py                    # Run continuous monitoring
-python src/water_scraper_v3.py --test             # Single test cycle
-python src/water_scraper_v3.py --help             # Show help
-```
-
-### Data Management
-```bash
-python src/water_scraper_v3.py --check-gaps 7     # Check for missing data (7 days)
-python src/water_scraper_v3.py --fill-gaps 7      # Fill missing data gaps
-python src/water_scraper_v3.py --update-data 2    # Update existing data (2 days)
-```
-
-### Database Testing
-```bash
-python src/demo_databases.py                      # SQLite demo
-python src/demo_databases.py victoriametrics      # VictoriaMetrics demo
-python src/demo_databases.py all                  # Test all databases
-```
-
-## 📚 Documentation
-
-### Core Documentation
-- **[Data Sources & API Catalog](docs/DATA_SOURCES.md)** - Every ingested and available data source (RID, ThaiWater/HII, dams, rainfall, forecasts)
-- **[Installation Guide](docs/DATABASE_DEPLOYMENT_GUIDE.md)** - Complete setup instructions
-- **[Gap Filling Guide](docs/GAP_FILLING_GUIDE.md)** - Data integrity management
-
-### Deployment Guides
-- **[VictoriaMetrics Setup](docs/VICTORIAMETRICS_SETUP.md)** - High-performance deployment
-- **[Debian Troubleshooting](docs/DEBIAN_TROUBLESHOOTING.md)** - Linux deployment issues
-
-### References
-- **[Notable Documents](docs/references/NOTABLE_DOCUMENTS.md)** - Official Thai government resources
-
-## 🔍 Troubleshooting
-
-### Common Issues
-
-**Database Connection Errors:**
-```bash
-# Check database status
-python src/demo_databases.py
-
-# Test specific database
-python src/demo_databases.py victoriametrics
-```
-
-**Missing Data:**
-```bash
-# Check for gaps
-python src/water_scraper_v3.py --check-gaps 7
-
-# Fill missing data
-python src/water_scraper_v3.py --fill-gaps 7
-```
-
-**Service Issues:**
-```bash
-# Check service status
-sudo systemctl status water-monitor
-
-# View logs
-sudo journalctl -u water-monitor -f
-```
-
-### Health Checks
-
-```bash
-# VictoriaMetrics health
-curl http://localhost:8428/health
-
-# Check latest data
-curl "http://localhost:8428/api/v1/query?query=water_level"
-
-# Application logs
-tail -f water_monitor.log
-```
-
-## 🌐 API Integration
-
-### VictoriaMetrics API Examples
-
-```bash
-# Query current water levels
-curl "http://localhost:8428/api/v1/query?query=water_level"
-
-# Query discharge rates for last hour
-curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"
-
-# Query specific station
-curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
-
-# High discharge alerts
-curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
-```
-
-## 📊 Performance
-
-### System Requirements
-- **CPU**: 1-2 cores (minimal load)
-- **RAM**: 512MB - 2GB (depending on database)
-- **Storage**: 1GB+ (for historical data)
-- **Network**: Stable internet connection
-
-### Performance Metrics
-- **Data Collection**: ~300 data points every 15 minutes
-- **Database Write Speed**: 1000+ points/second (VictoriaMetrics)
-- **Query Response**: <100ms for recent data
-- **Storage Efficiency**: 70x compression vs. raw data
-
-## 🤝 Contributing
-
-Contributions are welcome! Please:
-
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Add tests if applicable
-5. Submit a pull request
-
-### Development Setup
-
-```bash
-# Clone your fork
-git clone https://github.com/your-username/thailand-water-monitor.git
-cd thailand-water-monitor
-
-# Install development dependencies
-pip install -r requirements.txt
-pip install pytest black flake8
-
-# Run tests
-pytest
-
-# Format code
-black src/
-```
-
-## 📄 License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## 🙏 Acknowledgments
-
-- **Royal Irrigation Department (RID)** of Thailand for providing the data API
-- **VictoriaMetrics** team for the excellent time-series database
-- **Grafana** team for the visualization platform
-- **Python community** for the amazing libraries and tools
-
-## 📞 Support
-
-- **Issues**: [GitHub Issues](https://github.com/your-username/thailand-water-monitor/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/your-username/thailand-water-monitor/discussions)
-- **Documentation**: [Project Wiki](https://github.com/your-username/thailand-water-monitor/wiki)
-
----
-
-## 📁 Project Structure
+## Repository layout
 
 ```
-Northern-Thailand-Ping-River-Monitor/
-├── src/                    # Main application code
-├── tests/                  # Test suite
-├── docs/                   # Documentation
-├── grafana/               # Grafana dashboards
-├── scripts/               # Utility scripts
-├── docker-compose.yml     # Docker deployment
-├── Makefile              # Development tasks
-└── requirements.txt      # Dependencies
+src/                 collector, API (web_api.py), dashboard (static/dashboard.html)
+src/ml/              features, training, evaluation harness, prediction, rain/dam/HII loaders
+scripts/             train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
+tests/               pytest suite (synthetic data; no DB or network)
+docs/                FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
+models/              trained bundles + metrics.json (gitignored) and evaluation results (tracked)
+.gitea/workflows/    ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
 ```
 
-See [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) for the forecasting architecture and [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) for the data pipeline.
+## Documentation
 
-## 🔄 CI/CD & Automation
+- [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
+- [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) — every ingested and candidate source, endpoints, quirks
+- [docs/STATION_MANAGEMENT_GUIDE.md](docs/STATION_MANAGEMENT_GUIDE.md) — adding/editing gauges
+- [docs/DATABASE_DEPLOYMENT_GUIDE.md](docs/DATABASE_DEPLOYMENT_GUIDE.md), [POSTGRESQL_SETUP.md](POSTGRESQL_SETUP.md) — database setup
+- [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md) — public push alerts: topics, semantics, ntfy deployment
+- [docs/MATRIX_QUICK_START.md](docs/MATRIX_QUICK_START.md) — Matrix room alerts for a team
+- [docs/GAP_FILLING_GUIDE.md](docs/GAP_FILLING_GUIDE.md) — data integrity tooling
+- [docs/references/NOTABLE_DOCUMENTS.md](docs/references/NOTABLE_DOCUMENTS.md) — official Thai government resources
+- Public overview: [buildfor.life/docs/tooling/ping-river-monitor](https://buildfor.life/docs/tooling/ping-river-monitor/)
 
-The project includes comprehensive Gitea Actions workflows:
+Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
+dashboards under `grafana/` are supported by the adapters but not what production
+runs; see [docs/VICTORIAMETRICS_SETUP.md](docs/VICTORIAMETRICS_SETUP.md) if you want them.
 
-- **🧪 CI/CD Pipeline** - Automated testing, building, and deployment
-- **🔒 Security Scanning** - Daily vulnerability and dependency checks
-- **📚 Documentation** - Automated API docs and validation
-- **🚀 Release Management** - Automated releases with multi-arch Docker builds
+## Contributing
 
-See [docs/GITEA_WORKFLOWS.md](docs/GITEA_WORKFLOWS.md) for detailed workflow documentation.
+`make format` before committing (black 88 columns, isort black profile — the CI gate),
+`make test` must stay green, tests use synthetic data only. See
+[CONTRIBUTING.md](CONTRIBUTING.md). Issues and merge requests on
+[git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
 
-## 🔗 Repository
+## Data sources and thanks
 
-- **Main Repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
-- **Issues**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
-- **Actions**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
-- **Documentation**: [docs/](docs/)
+Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) /
+ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry;
+Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All
+instruments are theirs; we aggregate, store, fill gaps and forecast.
 
-**Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin**
+## License
+
+MIT — see [LICENSE](LICENSE).

docs/NOTIFICATIONS.md

@@ -0,0 +1,132 @@
+# Flood notifications (ntfy)
+
+Public push notifications for threshold crossings, without accounts, mailing
+lists or app-store review: the monitor publishes to a self-hosted
+[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
+about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
+
+ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
+runs on the same VPS as the monitor.
+
+## What subscribers get
+
+Every message is a **transition**, never a state. Crossing up into a level sends
+one message; dropping back below it (with 0.10 m hysteresis) sends one
+all-clear. A river that sits at 3.9 m for three days produces two messages, not
+seventy-two. In a quiet season a subscriber hears nothing.
+
+| Topic | Trigger | Priority |
+|---|---|---|
+| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
+| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
+| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
+| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
+| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
+| `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
+
+Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
+Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
+4.20 m, P.103 5.95 / 6.75 m, and so on.
+
+The outlook topic is opt-in for a reason: it is model output, and the message
+says so. Observed-crossing topics only ever report a gauge reading.
+
+Each message carries a click-through and an "Open dashboard" action button to
+the public dashboard.
+
+## How it runs
+
+`src/notify.py` is called once per collection cycle inside the API process
+(leader only), right after the forecast precompute, so it sees exactly the
+readings and forecasts the dashboard shows. Per-key last-sent state is stored
+in the `notification_state` table of the monitor's own database, so a restart
+or redeploy never re-sends and never misses a crossing that happened while
+the service was down (the next cycle compares against the persisted state).
+
+If ntfy is unreachable the transition is **not** recorded, so it is retried
+on the next cycle rather than silently lost. Any other failure in the notify
+step is logged and never reaches the collection loop.
+
+The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
+set; it reads `GET /api/notifications` and renders subscribe links
+(`ntfy://` deep links for the app, https links for the web UI).
+
+## Deployment
+
+On the monitor VPS, as root:
+
+```bash
+cd /opt/thailand-water-monitor
+NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
+```
+
+This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
+host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
+message cache, signup/login/metrics off, tight visitor limits), enables the
+systemd unit,
+creates the `monitor` user with **write-only access to `ping-*`**, mints a
+token, and appends `NTFY_SERVER` (public URL for subscribers),
+`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
+and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
+
+```bash
+systemctl restart water-monitor
+journalctl -u water-monitor -n 20 | grep ntfy      # "ntfy notifications: https://... topics ping-*"
+curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1'   # anonymous read works
+```
+
+The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
+the monitor host's Tailscale address and nothing is exposed on a public
+interface. On the Caddy machine:
+
+```caddyfile
+ntfy.buildfor.life {
+    reverse_proxy <monitor tailscale ip>:2586
+}
+```
+
+Caddy proxies websockets and keeps long-poll connections open by default;
+subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
+`X-Forwarded-For` rather than treating every subscriber as the proxy.
+
+Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
+monitor at the Tailscale address directly, so a DNS or proxy problem never
+holds back an alert. Test the pipeline before the domain is live with
+`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
+
+## Configuration
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
+| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
+| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
+| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
+| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
+
+Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
+(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
+
+## Testing
+
+`tests/test_notify.py` covers the state machine: quiet river sends nothing;
+crossing once, then silence while above, then all-clear; hysteresis on the way
+down; escalation to danger and back; basin digest grouping; outlook on/off;
+heuristic forecasts ignored; stale feed and recovery; state survives a restart
+through sqlite; a failed publish is retried next cycle.
+
+To exercise the real path against a real ntfy locally: run `ntfy serve` (any
+platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
+poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
+template.
+
+## Why ntfy and not …
+
+- **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
+  subscriber and a room invite; fine for a team, wrong for the public.
+- **Gotify**: also self-hosted and light, but Android-only client and one
+  account per subscriber.
+- **Email / SMS**: deliverability work, cost per message, no priority
+  semantics; ntfy can forward to email per subscription if someone wants it.
+- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
+  added later as ntfy→webhook fan-out without touching the monitor.

pyproject.toml

@@ -34,23 +34,23 @@ classifiers = [
     "Environment :: Web Environment",
     "Framework :: FastAPI"
 ]
-requires-python = ">=3.11"
+requires-python = ">=3.11,<3.12"
 dependencies = [
     # Core dependencies
-    "requests==2.31.0",
+    "requests==2.34.2",
     "schedule==1.2.0",
     "pandas==2.0.3",
     "numpy>=1.24,<2",
     # Flood forecasting (ML)
     "scikit-learn==1.9.0",
     # Web API framework
-    "fastapi==0.104.1",
-    "uvicorn[standard]==0.24.0",
-    "pydantic==2.5.0",
+    "fastapi==0.141.1",
+    "uvicorn[standard]==0.52.4",
+    "pydantic==2.13.5",
     # Database adapters
     "sqlalchemy==2.0.23",
     "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
     "psycopg2-binary==2.9.9",
     # Monitoring and metrics
     "psutil==5.9.6"
@@ -59,11 +59,11 @@ dependencies = [
 [project.optional-dependencies]
 dev = [
     # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
     "pytest-cov==4.1.0",
     "pytest-asyncio==0.21.1",
     # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
     "flake8==6.1.0",
     "isort==5.12.0",
     "mypy==1.7.1",
@@ -73,7 +73,7 @@ dev = [
     "ipython==8.17.2",
     "jupyter==1.0.0",
     # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
     "types-python-dateutil==2.8.19.14"
 ]
 docs = [
@@ -83,7 +83,7 @@ docs = [
 ]
 all = [
     "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
     "psycopg2-binary==2.9.9"
 ]
 
@@ -100,11 +100,11 @@ Documentation = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/
 [dependency-groups]
 dev = [
     # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
     "pytest-cov==4.1.0",
     "pytest-asyncio==0.21.1",
     # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
     "flake8==6.1.0",
     "isort==5.12.0",
     "mypy==1.7.1",
@@ -114,7 +114,7 @@ dev = [
     "ipython==8.17.2",
     "jupyter==1.0.0",
     # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
     "types-python-dateutil==2.8.19.14",
     # Documentation
     "sphinx==7.2.6",

requirements-dev.txt

@@ -2,12 +2,12 @@
 -r requirements.txt
 
 # Testing
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
 pytest-asyncio==0.21.1
 
 # Code formatting and linting
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 isort==5.12.0
 mypy==1.7.1
@@ -25,5 +25,5 @@ ipython==8.17.2
 jupyter==1.0.0
 
 # Type stubs
-types-requests==2.31.0.10
+types-requests==2.33.0.20260906
 types-python-dateutil==2.8.19.14

requirements.txt

@@ -1,5 +1,5 @@
 # Core dependencies
-requests==2.31.0
+requests==2.34.2
 schedule==1.2.0
 pandas==2.0.3
 numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
@@ -8,23 +8,23 @@ numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
 scikit-learn==1.9.0
 
 # Web API framework
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-pydantic==2.5.0
+fastapi==0.141.1
+uvicorn[standard]==0.52.4
+pydantic==2.13.5
 
 # Database adapters
 sqlalchemy==2.0.23
 influxdb==5.3.1
-pymysql==1.1.0
+pymysql==1.2.0
 psycopg2-binary==2.9.9
 
 # Monitoring and metrics
 psutil==5.9.6
 
 # Development dependencies (optional)
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 mypy==1.7.1
 pre-commit==3.5.0

scripts/dev_proxy.py

@@ -3,6 +3,7 @@ live server, so browser-side changes can be checked against real data
 before deploy. Usage: python scripts/dev_proxy.py [port]"""
 
 import http.server
+import os
 import sys
 import urllib.request
 from pathlib import Path
@@ -17,6 +18,12 @@ class Handler(http.server.BaseHTTPRequestHandler):
             body = (STATIC / "dashboard.html").read_bytes()
             self._send(200, "text/html; charset=utf-8", body)
             return
+        # Local overrides for endpoints not yet deployed: DEV_PROXY_LOCAL=/api/x=file.json,...
+        for pair in filter(None, os.environ.get("DEV_PROXY_LOCAL", "").split(",")):
+            prefix, file = pair.split("=", 1)
+            if self.path.split("?")[0] == prefix:
+                self._send(200, "application/json", Path(file).read_bytes())
+                return
         if self.path.startswith("/static/"):
             f = STATIC / self.path[len("/static/"):].split("?")[0]
             if f.is_file():

scripts/e2e_notify.py

@@ -0,0 +1,132 @@
+"""Drive the production notify path in-process: startup init -> seeded readings
+-> forecast cache -> _notify_transitions -> sqlite state -> real ntfy."""
+
+import asyncio
+import datetime
+import json
+import os
+import sys
+
+import requests
+
+os.environ.update(
+    DB_TYPE="sqlite",
+    WATER_DB_PATH=os.path.join(os.environ["LOCALAPPDATA"], "Temp", "smoke3.db"),
+    NTFY_SERVER="http://127.0.0.1:2586",
+    NTFY_TOKEN=os.environ.get("NTFY_TOKEN", ""),
+    NTFY_TOPIC_PREFIX="ping",
+)
+for f in ("smoke3.db",):
+    p = os.path.join(os.environ["LOCALAPPDATA"], "Temp", f)
+    if os.path.exists(p):
+        os.remove(p)
+
+from src import web_api  # noqa: E402
+from src.config import Config  # noqa: E402
+
+assert Config.NTFY_SERVER
+
+
+async def main():
+    # what the lifespan does at startup, minus the scheduler
+    from src import notify as notify_mod
+    from src.forecast_history import ForecastHistoryStore
+    from src.water_scraper_v3 import EnhancedWaterMonitorScraper
+
+    db_config = Config.get_database_config()
+    web_api.app_state["scraper"] = EnhancedWaterMonitorScraper(db_config)
+    store = ForecastHistoryStore(db_config["connection_string"], db_config["type"])
+    store.connect()
+    web_api.app_state["forecast_store"] = store
+    state = notify_mod.NotificationState(store.engine, store.db_type)
+    pub = notify_mod.NtfyPublisher(
+        Config.NTFY_SERVER, prefix=Config.NTFY_TOPIC_PREFIX, token=Config.NTFY_TOKEN
+    )
+    web_api.app_state["notify"] = (pub, state)
+
+    scraper = web_api.app_state["scraper"]
+    now = datetime.datetime.now().replace(minute=0, second=0, microsecond=0)
+
+    def seed(level_p1, level_p103, ts):
+        rows = [
+            {
+                "station_code": "P.1",
+                "station_id": 1,
+                "timestamp": ts,
+                "water_level": level_p1,
+                "discharge": 400.0,
+                "station_name_en": "Nawarat Bridge",
+                "station_name_th": "สะพานนวรัฐ",
+                "discharge_percent": 30.0,
+                "status": "active",
+            },
+            {
+                "station_code": "P.103",
+                "station_id": 2,
+                "timestamp": ts,
+                "water_level": level_p103,
+                "discharge": 300.0,
+                "station_name_en": "Ring Road 3",
+                "station_name_th": "วงแหวน 3",
+                "discharge_percent": 20.0,
+                "status": "active",
+            },
+        ]
+        scraper.db_adapter.save_measurements(rows)
+
+    def forecast(p):
+        with web_api.FORECAST_CACHE_LOCK:
+            web_api.FORECAST_CACHE["all"] = (
+                0,
+                [
+                    {
+                        "station_code": "P.1",
+                        "horizon_hours": 24,
+                        "p_warning": p,
+                        "predicted_max_level": 3.9,
+                        "source": "model",
+                    }
+                ],
+            )
+
+    def poll(topic):
+        out = []
+        for line in (
+            requests.get(f"{Config.NTFY_SERVER}/{topic}/json?poll=1", timeout=5)
+            .text.strip()
+            .splitlines()
+        ):
+            m = json.loads(line)
+            if m.get("event") == "message":
+                out.append(m.get("title") or m.get("message", "")[:40])
+        return out
+
+    # cycle 1: quiet
+    seed(1.6, 3.2, now - datetime.timedelta(hours=2))
+    forecast(0.02)
+    await web_api._notify_transitions()
+    # cycle 2: P.1 crosses warning, model outlook on
+    seed(3.75, 3.3, now - datetime.timedelta(hours=1))
+    forecast(0.7)
+    await web_api._notify_transitions()
+    # cycle 3: same state -> silence
+    seed(3.80, 3.3, now)
+    forecast(0.65)
+    await web_api._notify_transitions()
+
+    print("ping-p1-warning:", poll("ping-p1-warning"))
+    print("ping-warning:   ", poll("ping-warning"))
+    print("ping-p1-outlook:", poll("ping-p1-outlook"))
+    print("ping-p103-warning:", poll("ping-p103-warning"))
+    from sqlalchemy import text
+
+    with store.engine.connect() as c:
+        print(
+            "state table:",
+            c.execute(
+                text("SELECT key, state, value FROM notification_state ORDER BY key")
+            ).fetchall(),
+        )
+
+
+asyncio.run(main())

scripts/install_ntfy.sh

@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+# Install ntfy (https://ntfy.sh) as the public notification server for the
+# Ping River Monitor. Run as root on the monitor VPS. Idempotent.
+#
+#   NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
+#
+# What it does:
+#   - installs the ntfy .deb from the official GitHub release (single Go
+#     binary, ~30 MB RSS, sqlite message cache)
+#   - writes /etc/ntfy/server.yml: listens on the Tailscale address only
+#     (the reverse proxy is another VPS on the tailnet; nothing is exposed
+#     on a public interface), anonymous READ on all topics, WRITE only with
+#     a token. Override with NTFY_LISTEN=host:port.
+#   - creates the `monitor` publishing user + token, writes NTFY_SERVER /
+#     NTFY_TOKEN into /opt/thailand-water-monitor/.env if not present
+#
+# Reverse proxy (on the Caddy VPS, over Tailscale):
+#     ntfy.buildfor.life {
+#         reverse_proxy <this host's tailscale ip>:2586
+#     }
+# Caddy passes websockets and keeps long-poll connections open by default;
+# subscribers hold one open. ntfy runs with behind-proxy: true so rate
+# limits key on X-Forwarded-For, not on the proxy's address.
+set -euo pipefail
+
+NTFY_DOMAIN="${NTFY_DOMAIN:?set NTFY_DOMAIN, e.g. ntfy.buildfor.life}"
+NTFY_VERSION="${NTFY_VERSION:-2.28.0}"
+MONITOR_DIR="${MONITOR_DIR:-/opt/thailand-water-monitor}"
+TS_IP="$(tailscale ip -4 2>/dev/null | head -1 || true)"
+LISTEN="${NTFY_LISTEN:-${TS_IP:-127.0.0.1}:2586}"
+echo "ntfy will listen on ${LISTEN}"
+
+if ! command -v ntfy >/dev/null || [[ "$(ntfy --version 2>/dev/null | awk '{print $3}')" != "$NTFY_VERSION" ]]; then
+    tmp=$(mktemp -d)
+    curl -fsSL -o "$tmp/ntfy.deb" \
+        "https://github.com/binwiederhier/ntfy/releases/download/v${NTFY_VERSION}/ntfy_${NTFY_VERSION}_linux_amd64.deb"
+    dpkg -i "$tmp/ntfy.deb"
+    rm -rf "$tmp"
+fi
+
+install -d -m 755 /var/cache/ntfy /var/lib/ntfy
+cat > /etc/ntfy/server.yml <<EOF
+# Ping River Monitor notification server. Managed by scripts/install_ntfy.sh.
+base-url: "https://${NTFY_DOMAIN}"
+listen-http: "${LISTEN}"
+behind-proxy: true
+
+# Messages are kept so a phone that was offline still gets the crossing.
+cache-file: "/var/cache/ntfy/cache.db"
+cache-duration: "72h"
+
+# Everyone may subscribe; only the monitor (token) may publish.
+auth-file: "/var/lib/ntfy/user.db"
+auth-default-access: "read-only"
+
+# The monitor publishes a handful of messages per flood; be strict with
+# everything else so the box cannot be used as a free relay.
+visitor-request-limit-burst: 30
+visitor-request-limit-replenish: "10s"
+visitor-subscription-limit: 60
+visitor-message-daily-limit: 200
+attachment-cache-dir: ""
+enable-signup: false
+enable-login: false
+enable-metrics: false
+EOF
+
+systemctl enable --now ntfy
+systemctl restart ntfy
+sleep 1
+curl -fsS "http://${LISTEN}/v1/health" >/dev/null && echo "ntfy up on ${LISTEN}"
+
+# Publishing identity for the monitor
+if ! ntfy user list 2>/dev/null | grep -q '^user monitor (role'; then
+    NTFY_PASSWORD="$(openssl rand -base64 24)" ntfy user add --role=user monitor
+fi
+ntfy access monitor 'ping-*' write-only >/dev/null
+# 'ping-*' read stays anonymous via auth-default-access
+
+token=$(ntfy token list monitor 2>/dev/null | awk '/^- tk_/{print $2; exit}')  # '- tk_xxx (label), ...'
+if [[ -z "$token" ]]; then
+    token=$(ntfy token add --label "water-monitor" monitor | grep -oE 'tk_[A-Za-z0-9]+' | head -1)  # 'token tk_xxx created for user monitor'
+fi
+
+env_file="${MONITOR_DIR}/.env"
+if [[ -f "$env_file" ]] && ! grep -q '^NTFY_SERVER=' "$env_file"; then
+    {
+        echo ""
+        echo "# ntfy public notifications (scripts/install_ntfy.sh)"
+        echo "NTFY_SERVER=https://${NTFY_DOMAIN}"
+        echo "NTFY_PUBLISH_URL=http://${LISTEN}"
+        echo "NTFY_TOPIC_PREFIX=ping"
+        echo "NTFY_TOKEN=${token}"
+    } >> "$env_file"
+    echo "wrote NTFY_* to ${env_file}; restart water-monitor to enable"
+else
+    echo "NTFY_TOKEN=${token}"
+fi
+
+echo
+echo "Subscribe test (anonymous read):  curl -s 'http://${LISTEN}/ping-status/json?poll=1'"
+echo "Publish test (needs token):       curl -s -H 'Authorization: Bearer ${token}' -d 'hello' http://${LISTEN}/ping-status"

src/config.py

@@ -38,6 +38,17 @@ class Config:
     TARGET_URL = "https://hyd-app-db.rid.go.th/hydro1h.html"
     API_URL = "https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx"
     THAIWATER_API_KEY = os.getenv("THAIWATER_API_KEY")
+
+    # Public flood notifications (ntfy). Off unless NTFY_SERVER is set.
+    # NTFY_SERVER is what subscribers use (public https URL, shown on the
+    # dashboard). NTFY_PUBLISH_URL is where the monitor POSTs; defaults to
+    # NTFY_SERVER, set it to http://127.0.0.1:2586 when ntfy runs on the same
+    # host so publishing never depends on DNS/proxy/tunnel being up.
+    NTFY_SERVER = os.getenv("NTFY_SERVER", "").strip()
+    NTFY_PUBLISH_URL = os.getenv("NTFY_PUBLISH_URL", "").strip() or NTFY_SERVER
+    NTFY_TOPIC_PREFIX = os.getenv("NTFY_TOPIC_PREFIX", "ping").strip()
+    NTFY_TOKEN = os.getenv("NTFY_TOKEN", "").strip()  # publish token if ACL enabled
+    PUBLIC_URL = os.getenv("PUBLIC_URL", "https://water.buildfor.life/").strip()
     REQUEST_TIMEOUT = int(os.getenv("REQUEST_TIMEOUT", "30"))
     USER_AGENT = (
         "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "

src/database_adapters.py

@@ -139,12 +139,16 @@ class InfluxDBAdapter(DatabaseAdapter):
                     "time": measurement["timestamp"].isoformat(),
                     "fields": {
                         "water_level": float(measurement["water_level"]),
-                        "discharge": float(measurement["discharge"])
-                        if measurement.get("discharge") is not None
-                        else None,
-                        "discharge_percent": float(measurement["discharge_percent"])
-                        if measurement.get("discharge_percent")
-                        else None,
+                        "discharge": (
+                            float(measurement["discharge"])
+                            if measurement.get("discharge") is not None
+                            else None
+                        ),
+                        "discharge_percent": (
+                            float(measurement["discharge_percent"])
+                            if measurement.get("discharge_percent")
+                            else None
+                        ),
                     },
                 }
                 points.append(point)
@@ -551,13 +555,13 @@ class SQLAdapter(DatabaseAdapter):
                             "station_code": row[1],
                             "station_name_en": row[2],
                             "station_name_th": row[3],
-                            "water_level": float(row[4])
-                            if row[4] is not None
-                            else None,
+                            "water_level": (
+                                float(row[4]) if row[4] is not None else None
+                            ),
                             "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
-                            if row[6] is not None
-                            else None,
+                            "discharge_percent": (
+                                float(row[6]) if row[6] is not None else None
+                            ),
                             "status": row[7],
                         }
                     )
@@ -611,13 +615,13 @@ class SQLAdapter(DatabaseAdapter):
                             "station_code": row[1],
                             "station_name_en": row[2],
                             "station_name_th": row[3],
-                            "water_level": float(row[4])
-                            if row[4] is not None
-                            else None,
+                            "water_level": (
+                                float(row[4]) if row[4] is not None else None
+                            ),
                             "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
-                            if row[6] is not None
-                            else None,
+                            "discharge_percent": (
+                                float(row[6]) if row[6] is not None else None
+                            ),
                             "status": row[7],
                         }
                     )
@@ -666,13 +670,13 @@ class SQLAdapter(DatabaseAdapter):
                             "station_id": row[1],
                             "station_code": row[2] or f"Station_{row[1]}",
                             "station_name_th": row[3] or f"Station {row[1]}",
-                            "water_level": float(row[4])
-                            if row[4] is not None
-                            else None,
+                            "water_level": (
+                                float(row[4]) if row[4] is not None else None
+                            ),
                             "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
-                            if row[6] is not None
-                            else None,
+                            "discharge_percent": (
+                                float(row[6]) if row[6] is not None else None
+                            ),
                             "status": row[7],
                         }
                     )

src/health_check.py

@@ -125,9 +125,9 @@ class DatabaseHealthCheck(HealthCheck):
                 "message": "Database connection OK",
                 "details": {
                     "latest_data_count": len(latest_data),
-                    "latest_timestamp": str(latest_data[0].get("timestamp"))
-                    if latest_data
-                    else None,
+                    "latest_timestamp": (
+                        str(latest_data[0].get("timestamp")) if latest_data else None
+                    ),
                 },
             }
 

src/ml/data.py

@@ -207,9 +207,9 @@ def fill_from_hii(
                 "timestamp": missing["timestamp"],
                 "station_code": code,
                 "water_level": missing["wl_msl"] - offset,
-                "discharge": missing["discharge"]
-                if code in _HII_EXACT_MIRRORS
-                else float("nan"),
+                "discharge": (
+                    missing["discharge"] if code in _HII_EXACT_MIRRORS else float("nan")
+                ),
             }
         )
         fills.append(fill)

src/ml/features.py

@@ -37,9 +37,17 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
     "P.4A": (3.40, 3.90),
     "P.5": (4.55, 4.95),
     "P.67": (2.45, 2.90),
-    "P.75": (2.75, 3.50),
+    # P.75: 2024 (the only year with a full flood record, 191% capacity peak)
+    # puts 75-85% at 3.45 m and 95-105% at 3.72 m; 2018/2022 agree within
+    # 0.15 m. The 2026-08 value (2.75) alerted on 15 quiet-season hours.
+    "P.75": (3.20, 3.65),
     "P.76": (5.35, 5.45),
-    "P.77": (2.85, 3.35),
+    # P.77: recalibrated 2026-09-12. The 2026-08 value (2.85) sat below the
+    # gauge's own dry-season baseline (2.6-2.7 m at 8-14% capacity), so the
+    # first ntfy cycle fired a "warning" at 22% capacity. Across 2018-2024,
+    # 75-85% capacity reads 3.35-4.57 m and 95-105% 4.27-5.08 m; 2024 (the
+    # best-sampled flood year) gives 4.57 / 5.08. Slightly conservative:
+    "P.77": (4.30, 4.90),
     "P.81": (5.15, 6.30),
     # P.82 never reached 100% capacity in the record (max level 3.78, max 96.4%);
     # danger sits just below the observed maximum so the head can actually train.

src/ml/skill.py

@@ -0,0 +1,170 @@
+"""Live forecast skill: what the deployed model said versus what the river did.
+
+Every hour the precompute stores the issued 24 h forecast (forecast_history);
+water_measurements holds what actually happened. Joining the two gives a
+verification that needs no retraining and answers the question the dashboard
+is asked most: "is the model getting better?" — per model version, on the
+hours that version was actually serving.
+
+Metrics per version and horizon:
+  n           verified forecasts (issued, and the horizon has since elapsed)
+  mae         |predicted_max - observed_max| over the horizon window, metres
+  bias        mean(predicted - observed): >0 over-predicts the peak
+  persistence MAE of the trivial "peak = current level" forecast on the
+              same rows; a model is only useful if it beats this
+  skill       1 - mae/persistence  (0 = no better than persistence, 1 = perfect)
+  above_2m    same MAE restricted to rows where the observed peak >= 2 m,
+              i.e. the flood-relevant regime
+
+Only the P.1 gauge is verified by default: it is the one the city threshold
+is keyed to, and one station keeps the query cheap enough to run on request.
+"""
+
+import datetime
+import logging
+from typing import Dict, List, Optional
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_STATION = "P.1"
+DEFAULT_HORIZON = 24
+MIN_VERIFIED = 24  # fewer than a day of verified hours is not a number
+
+
+def _sql_for(db_type: str) -> str:
+    """Join each issued forecast to the observed max over (as_of, as_of + h]."""
+    if db_type == "postgresql":
+        window_end = "f.as_of + (f.horizon_hours || ' hours')::interval"
+    elif db_type == "mysql":
+        window_end = "DATE_ADD(f.as_of, INTERVAL f.horizon_hours HOUR)"
+    else:  # sqlite
+        window_end = "datetime(f.as_of, '+' || f.horizon_hours || ' hours')"
+    return f"""
+        SELECT f.as_of, f.model_version, f.predicted_max_level, f.current_level,
+               (SELECT MAX(m.water_level) FROM water_measurements m
+                 JOIN stations s ON s.id = m.station_id
+                WHERE s.station_code = f.station_code
+                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_max,
+               (SELECT COUNT(m.water_level) FROM water_measurements m
+                 JOIN stations s ON s.id = m.station_id
+                WHERE s.station_code = f.station_code
+                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_n
+        FROM forecast_history f
+        WHERE f.station_code = :code AND f.horizon_hours = :horizon
+          AND f.source = 'model' AND f.predicted_max_level IS NOT NULL
+          AND f.as_of <= :verifiable_before
+        ORDER BY f.as_of
+    """
+
+
+def compute_skill(
+    engine,
+    db_type: str,
+    station_code: str = DEFAULT_STATION,
+    horizon_hours: int = DEFAULT_HORIZON,
+    now: Optional[datetime.datetime] = None,
+) -> Dict:
+    """Per-model-version verification of issued forecasts against observations."""
+    from sqlalchemy import text
+
+    now = now or datetime.datetime.now()
+    verifiable_before = now - datetime.timedelta(hours=horizon_hours)
+    with engine.connect() as conn:
+        rows = [
+            dict(r._mapping)
+            for r in conn.execute(
+                text(_sql_for(db_type)),
+                {
+                    "code": station_code,
+                    "horizon": horizon_hours,
+                    "verifiable_before": verifiable_before,
+                },
+            )
+        ]
+
+    def _ts(value):
+        # sqlite hands back strings; postgres/mysql give datetimes
+        if isinstance(value, datetime.datetime):
+            return value
+        return datetime.datetime.fromisoformat(str(value).replace(" ", "T"))
+
+    by_version: Dict[str, List[dict]] = {}
+    for r in rows:
+        r["as_of"] = _ts(r["as_of"])
+        # need most of the window observed, or the "max" is not the peak
+        if r["observed_max"] is None or (r["observed_n"] or 0) < horizon_hours * 0.75:
+            continue
+        by_version.setdefault(r["model_version"] or "unknown", []).append(r)
+
+    versions = []
+    for version, vrows in by_version.items():
+        pred = [float(r["predicted_max_level"]) for r in vrows]
+        obs = [float(r["observed_max"]) for r in vrows]
+        cur = [
+            float(r["current_level"]) if r["current_level"] is not None else None
+            for r in vrows
+        ]
+        err = [p - o for p, o in zip(pred, obs)]
+        mae = sum(abs(e) for e in err) / len(err)
+        bias = sum(err) / len(err)
+        pers_rows = [(c, o) for c, o in zip(cur, obs) if c is not None]
+        persistence = (
+            sum(abs(c - o) for c, o in pers_rows) / len(pers_rows)
+            if pers_rows
+            else None
+        )
+        high = [(p, o) for p, o in zip(pred, obs) if o >= 2.0]
+        versions.append(
+            {
+                "model_version": version,
+                "first_issued": min(r["as_of"] for r in vrows).isoformat(),
+                "last_issued": max(r["as_of"] for r in vrows).isoformat(),
+                "n": len(vrows),
+                "mae_m": round(mae, 3),
+                "bias_m": round(bias, 3),
+                "persistence_mae_m": (
+                    None if persistence is None else round(persistence, 3)
+                ),
+                "skill": (
+                    None if not persistence else round(1.0 - mae / persistence, 3)
+                ),
+                "above_2m_n": len(high),
+                "above_2m_mae_m": (
+                    round(sum(abs(p - o) for p, o in high) / len(high), 3)
+                    if high
+                    else None
+                ),
+                "enough_data": len(vrows) >= MIN_VERIFIED,
+            }
+        )
+    versions.sort(key=lambda v: v["first_issued"])
+
+    # Headline: current version vs the previous one that had enough data
+    current = versions[-1] if versions else None
+    previous = (
+        next((v for v in reversed(versions[:-1]) if v["enough_data"]), None)
+        if versions
+        else None
+    )
+    trend = None
+    if current and previous and current["enough_data"]:
+        trend = {
+            "previous_version": previous["model_version"],
+            "mae_delta_m": round(current["mae_m"] - previous["mae_m"], 3),
+            "skill_delta": (
+                None
+                if current["skill"] is None or previous["skill"] is None
+                else round(current["skill"] - previous["skill"], 3)
+            ),
+            "better": current["mae_m"] < previous["mae_m"],
+        }
+
+    return {
+        "station_code": station_code,
+        "horizon_hours": horizon_hours,
+        "verified_until": verifiable_before.isoformat(),
+        "min_verified": MIN_VERIFIED,
+        "versions": versions,
+        "current": current,
+        "trend": trend,
+    }

src/ml/train.py

@@ -320,9 +320,9 @@ def train_station(
                     skipped_heads,
                 )
             else:
-                skipped_heads[
-                    head_key
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                skipped_heads[head_key] = (
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
             heads[head_key] = clf
 
             if not skip_eval:
@@ -417,9 +417,9 @@ def train_station(
                 if clf is not None:
                     skipped_heads.pop(head_key, None)
             else:
-                skipped_heads[
-                    head_key
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                skipped_heads[head_key] = (
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
                 final_heads[head_key] = None
 
     # v4 = + Mae Ngat dam features; v3 = rise + rain; v2 = rise target only

src/notify.py

@@ -0,0 +1,454 @@
+"""Public flood notifications over ntfy.
+
+Runs once per collection cycle inside the API process (leader only), right
+after the forecast precompute, so it sees the same readings and forecasts the
+dashboard shows. Publishes to a self-hosted ntfy server; anyone subscribes to
+a topic from the free app or a browser, no account needed.
+
+Topics (all under one configurable prefix, default "ping"):
+
+  {prefix}-{station}-warning     observed level crossed the station's warning threshold
+  {prefix}-{station}-danger      observed level crossed the danger threshold
+  {prefix}-warning               any station crossed warning (basin-wide digest)
+  {prefix}-danger                any station crossed danger
+  {prefix}-p1-outlook            model early warning for Chiang Mai city: P.1's 24 h
+                                 warning probability crossed the alert level (opt-in;
+                                 the forecast is experimental and says so)
+  {prefix}-status                feed/monitor health: data stale, recovered
+
+Each notification is a TRANSITION, not a state: crossing UP into a level sends
+one message; dropping back below (with hysteresis) sends an all-clear. While
+the river sits above a threshold nothing is repeated, so a subscriber in a
+flood gets a handful of messages, not one an hour. The per-topic state is
+persisted (notification_state table) so a restart never re-sends.
+
+Everything is fail-safe: ntfy unreachable, table missing, malformed
+reading -> a logged warning, never an exception into the collection loop.
+"""
+
+import datetime
+import logging
+from dataclasses import dataclass
+from typing import Dict, Iterable, List, Optional
+
+import requests
+
+from .ml import features
+
+logger = logging.getLogger(__name__)
+
+# Hysteresis: an all-clear needs the level this far BELOW the threshold, so a
+# river bobbing around 3.70 m does not toggle warning/clear every hour.
+CLEAR_MARGIN_M = 0.10
+# Capacity guard. The level thresholds in features.THRESHOLDS were calibrated
+# from RID's discharge_percent (% of channel capacity); if RID re-rates a
+# gauge or moves its datum, the level crosses while capacity says the channel
+# is nearly empty (P.77, 2026-09: 3.0 m "warning" at 22 %). A crossing is
+# only announced when the reported capacity agrees that the river is high.
+# P.1 is exempt: its stages come from the municipal inundation map, not from
+# capacity. Readings without a capacity figure fall back to level only.
+CAPACITY_GUARD_MIN_PCT = 60.0
+CAPACITY_GUARD_EXEMPT = {"P.1"}
+# Outlook alert fires when p_warning(24h) rises through ON, clears below OFF.
+OUTLOOK_ON = 0.50
+OUTLOOK_OFF = 0.25
+# Below this the outlook is not announced at all (avoid "5 % chance" noise).
+OUTLOOK_HORIZON = 24
+
+STATION_NAMES: Dict[str, str] = {
+    "P.1": "Nawarat Bridge, Chiang Mai city",
+    "P.103": "Ring Road Bridge 3, Chiang Mai",
+    "P.67": "Ban Tae (Mae Taeng)",
+    "P.21": "Ban Rim Tai (Mae Rim)",
+    "P.75": "Ban Chai Lat",
+    "P.92": "Ban Muang Aut",
+    "P.20": "Ban Chiang Dao",
+    "P.4A": "Ban Mae Taeng",
+    "P.5": "Tha Nang Bridge (downstream)",
+    "P.81": "Ban Pong (downstream)",
+    "P.82": "Ban Sob Win",
+    "P.84": "Ban Panton",
+    "P.87": "Ban Pa Sang",
+    "P.77": "Ban Sop Mae Sapuat",
+    "P.85": "Ban Lai Kaew",
+    "P.76": "Ban Mae I Hai",
+}
+
+
+def _slug(code: str) -> str:
+    return code.lower().replace(".", "")
+
+
+@dataclass
+class Notification:
+    topic: str
+    title: str
+    message: str
+    priority: int = 3  # ntfy: 1 min .. 5 max
+    tags: Optional[List[str]] = None
+    click: Optional[str] = None
+
+
+class NtfyPublisher:
+    def __init__(
+        self,
+        server: str,
+        prefix: str = "ping",
+        token: Optional[str] = None,
+        dashboard_url: str = "https://water.buildfor.life/",
+        timeout: int = 10,
+    ):
+        self.server = server.rstrip("/")
+        self.prefix = prefix
+        self.token = token
+        self.dashboard_url = dashboard_url
+        self.timeout = timeout
+
+    def topic(self, *parts: str) -> str:
+        return "-".join([self.prefix, *parts])
+
+    def publish(self, n: Notification) -> bool:
+        headers = {
+            "Title": n.title,
+            "Priority": str(n.priority),
+            "Click": n.click or self.dashboard_url,
+            "Actions": f"view, Open dashboard, {n.click or self.dashboard_url}",
+        }
+        if n.tags:
+            headers["Tags"] = ",".join(n.tags)
+        if self.token:
+            headers["Authorization"] = f"Bearer {self.token}"
+        try:
+            r = requests.post(
+                f"{self.server}/{n.topic}",
+                data=n.message.encode("utf-8"),
+                headers=headers,
+                timeout=self.timeout,
+            )
+            if r.status_code >= 300:
+                logger.warning(f"ntfy {n.topic}: HTTP {r.status_code} {r.text[:120]}")
+                return False
+            return True
+        except Exception as error:
+            logger.warning(f"ntfy {n.topic}: {error}")
+            return False
+
+
+class NotificationState:
+    """Per-key last-sent state, in the monitor's own SQL database."""
+
+    def __init__(self, engine, db_type: str):
+        self.engine = engine
+        self.db_type = db_type
+        self._ensure()
+
+    def _ensure(self) -> None:
+        from sqlalchemy import text
+
+        ddl = (
+            "CREATE TABLE IF NOT EXISTS notification_state ("
+            "key VARCHAR(64) PRIMARY KEY, state VARCHAR(16) NOT NULL, "
+            "value NUMERIC(8,3), updated_at TIMESTAMP NOT NULL)"
+        )
+        try:
+            with self.engine.begin() as conn:
+                conn.execute(text(ddl))
+        except Exception as error:
+            # Postgres: two sessions racing CREATE TABLE IF NOT EXISTS can
+            # both pass the existence check; the loser fails with a unique
+            # violation on pg_type. The table exists either way; verify.
+            with self.engine.connect() as conn:
+                conn.execute(text("SELECT 1 FROM notification_state WHERE 1=0"))
+            logger.debug(f"notification_state DDL raced, table present: {error}")
+
+    def get(self, key: str) -> Optional[str]:
+        from sqlalchemy import text
+
+        with self.engine.connect() as conn:
+            row = conn.execute(
+                text("SELECT state FROM notification_state WHERE key = :k"), {"k": key}
+            ).fetchone()
+        return row[0] if row else None
+
+    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
+        from sqlalchemy import text
+
+        now = datetime.datetime.now()
+        with self.engine.begin() as conn:
+            if self.db_type == "mysql":
+                sql = (
+                    "INSERT INTO notification_state (key, state, value, updated_at) "
+                    "VALUES (:k, :s, :v, :t) ON DUPLICATE KEY UPDATE "
+                    "state = VALUES(state), value = VALUES(value), updated_at = VALUES(updated_at)"
+                )
+            else:
+                sql = (
+                    "INSERT INTO notification_state (key, state, value, updated_at) "
+                    "VALUES (:k, :s, :v, :t) ON CONFLICT (key) DO UPDATE SET "
+                    "state = EXCLUDED.state, value = EXCLUDED.value, updated_at = EXCLUDED.updated_at"
+                )
+            conn.execute(text(sql), {"k": key, "s": state, "v": value, "t": now})
+
+
+class InMemoryState(NotificationState):
+    """For tests and when no SQL engine is available (loses state on restart)."""
+
+    def __init__(self):  # noqa: D107 - intentionally skips the SQL parent
+        self._d: Dict[str, str] = {}
+
+    def get(self, key: str) -> Optional[str]:
+        return self._d.get(key)
+
+    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
+        self._d[key] = state
+
+
+def _level_state(level: float, warn: float, danger: float, prev: Optional[str]) -> str:
+    """'clear' | 'warning' | 'danger', with hysteresis on the way down."""
+    if level >= danger:
+        return "danger"
+    if level >= warn:
+        # from danger: stay 'danger' until below danger - margin
+        if prev == "danger" and level >= danger - CLEAR_MARGIN_M:
+            return "danger"
+        return "warning"
+    if prev in ("warning", "danger") and level >= warn - CLEAR_MARGIN_M:
+        return "warning"
+    return "clear"
+
+
+def evaluate(
+    readings: Iterable[dict],
+    forecasts: Iterable[dict],
+    state: NotificationState,
+    publisher: NtfyPublisher,
+    stale_after_h: float = 3.0,
+    now: Optional[datetime.datetime] = None,
+) -> List[Notification]:
+    """Compare current readings/forecasts with last-sent state; publish transitions.
+
+    readings: rows with station_code, water_level, timestamp (latest per station)
+    forecasts: /forecast rows (station_code, horizon_hours, p_warning, predicted_max_level)
+    Returns the notifications that were published (for logs/tests).
+    """
+    now = now or datetime.datetime.now()
+    sent: List[Notification] = []
+
+    def emit(n: Notification) -> bool:
+        ok = publisher.publish(n)
+        if ok:
+            sent.append(n)
+        return ok
+
+    # ---- observed levels, per station, plus basin-wide fan-out
+    basin_changes: Dict[str, List[str]] = {"warning": [], "danger": [], "clear": []}
+    latest_ts: Optional[datetime.datetime] = None
+    for r in readings:
+        code = r.get("station_code")
+        level = r.get("water_level")
+        if not code or level is None:
+            continue
+        try:
+            level = float(level)
+        except (TypeError, ValueError):
+            continue
+        ts = r.get("timestamp")
+        if isinstance(ts, str):
+            try:
+                ts = datetime.datetime.fromisoformat(ts)
+            except ValueError:
+                ts = None
+        if isinstance(ts, datetime.datetime) and (latest_ts is None or ts > latest_ts):
+            latest_ts = ts
+        warn, danger = features.get_thresholds(code)
+        key = f"level:{code}"
+        prev = state.get(key) or "clear"
+        cur = _level_state(level, warn, danger, prev)
+        pct = r.get("discharge_percent")
+        if (
+            cur != "clear"
+            and prev == "clear"
+            and code not in CAPACITY_GUARD_EXEMPT
+            and pct is not None
+        ):
+            try:
+                if float(pct) < CAPACITY_GUARD_MIN_PCT:
+                    logger.info(
+                        f"{code}: level {level:.2f} m >= {warn:.2f} but only "
+                        f"{float(pct):.0f}% capacity; threshold looks stale, not alerting"
+                    )
+                    continue
+            except (TypeError, ValueError):
+                pass
+        if cur == prev:
+            continue
+        name = STATION_NAMES.get(code, code)
+        slug = _slug(code)
+        when = (
+            ts.strftime("%d %b %H:%M") if isinstance(ts, datetime.datetime) else "now"
+        )
+        if cur == "danger":
+            ok = emit(
+                Notification(
+                    publisher.topic(slug, "danger"),
+                    f"DANGER level at {code}",
+                    f"{name}: {level:.2f} m at {when}, above the danger level of {danger:.2f} m.",
+                    priority=5,
+                    tags=["rotating_light", code],
+                )
+            )
+            basin_changes["danger"].append(f"{code} {level:.2f} m")
+        elif cur == "warning":
+            if prev == "danger":
+                ok = emit(
+                    Notification(
+                        publisher.topic(slug, "danger"),
+                        f"{code} back below danger level",
+                        f"{name}: {level:.2f} m at {when}; still above the warning level of {warn:.2f} m.",
+                        priority=3,
+                        tags=["arrow_down", code],
+                    )
+                )
+                basin_changes["clear"].append(f"{code} below danger ({level:.2f} m)")
+            else:
+                ok = emit(
+                    Notification(
+                        publisher.topic(slug, "warning"),
+                        f"Warning level at {code}",
+                        f"{name}: {level:.2f} m at {when}, above the warning level of {warn:.2f} m.",
+                        priority=4,
+                        tags=["warning", code],
+                    )
+                )
+                basin_changes["warning"].append(f"{code} {level:.2f} m")
+        else:  # clear
+            ok = emit(
+                Notification(
+                    publisher.topic(slug, "warning"),
+                    f"{code} back to normal",
+                    f"{name}: {level:.2f} m at {when}, below the warning level of {warn:.2f} m.",
+                    priority=2,
+                    tags=["white_check_mark", code],
+                )
+            )
+            basin_changes["clear"].append(f"{code} normal ({level:.2f} m)")
+        # Only remember the transition once it was actually delivered: if ntfy
+        # was down, the next cycle retries instead of silently swallowing a
+        # flood crossing.
+        if ok:
+            state.set(key, cur, level)
+
+    if basin_changes["danger"]:
+        emit(
+            Notification(
+                publisher.topic("danger"),
+                "Ping River: danger level reached",
+                "; ".join(basin_changes["danger"]),
+                priority=5,
+                tags=["rotating_light"],
+            )
+        )
+    if basin_changes["warning"]:
+        emit(
+            Notification(
+                publisher.topic("warning"),
+                "Ping River: warning level reached",
+                "; ".join(basin_changes["warning"]),
+                priority=4,
+                tags=["warning"],
+            )
+        )
+    if basin_changes["clear"]:
+        emit(
+            Notification(
+                publisher.topic("warning"),
+                "Ping River: levels falling",
+                "; ".join(basin_changes["clear"]),
+                priority=2,
+                tags=["white_check_mark"],
+            )
+        )
+
+    # ---- model outlook for the city gauge (opt-in topic, experimental)
+    p1 = next(
+        (
+            f
+            for f in forecasts
+            if f.get("station_code") == "P.1"
+            and f.get("horizon_hours") == OUTLOOK_HORIZON
+            and f.get("source") == "model"
+        ),
+        None,
+    )
+    if p1 and p1.get("p_warning") is not None:
+        p = float(p1["p_warning"])
+        key = "outlook:P.1"
+        prev = state.get(key) or "off"
+        cur = (
+            "on" if (p >= OUTLOOK_ON or (prev == "on" and p >= OUTLOOK_OFF)) else "off"
+        )
+        if cur != prev:
+            peak = p1.get("predicted_max_level")
+            warn, _ = features.get_thresholds("P.1")
+            if cur == "on":
+                ok = emit(
+                    Notification(
+                        publisher.topic("p1-outlook"),
+                        "Early warning: Chiang Mai flood risk rising",
+                        f"The forecast model gives a {p * 100:.0f}% chance that Nawarat Bridge (P.1) "
+                        f"reaches {warn:.2f} m within 24 h"
+                        + (
+                            f" (expected peak {float(peak):.2f} m)"
+                            if peak is not None
+                            else ""
+                        )
+                        + ". Experimental model output, not an official warning; "
+                        "follow ThaiWater/TMD for official alerts.",
+                        priority=4,
+                        tags=["crystal_ball"],
+                    )
+                )
+            else:
+                ok = emit(
+                    Notification(
+                        publisher.topic("p1-outlook"),
+                        "Chiang Mai flood risk easing",
+                        f"The model's 24 h probability of reaching {warn:.2f} m at P.1 has dropped to {p * 100:.0f}%.",
+                        priority=2,
+                        tags=["crystal_ball"],
+                    )
+                )
+            if ok:
+                state.set(key, cur, p)
+
+    # ---- feed health
+    if latest_ts is not None:
+        age_h = (now - latest_ts).total_seconds() / 3600.0
+        key = "feed"
+        prev = state.get(key) or "ok"
+        cur = "stale" if age_h >= stale_after_h else "ok"
+        if cur != prev:
+            if cur == "stale":
+                ok = emit(
+                    Notification(
+                        publisher.topic("status"),
+                        "Ping River monitor: gauge feed stale",
+                        f"No new readings for {age_h:.0f} h (last {latest_ts:%d %b %H:%M}). "
+                        "Levels and forecasts on the dashboard are not current.",
+                        priority=3,
+                        tags=["hourglass"],
+                    )
+                )
+            else:
+                ok = emit(
+                    Notification(
+                        publisher.topic("status"),
+                        "Ping River monitor: feed recovered",
+                        f"Readings are current again (latest {latest_ts:%d %b %H:%M}).",
+                        priority=2,
+                        tags=["white_check_mark"],
+                    )
+                )
+            if ok:
+                state.set(key, cur, age_h)
+    return sent

src/postgres_history.py

@@ -51,8 +51,7 @@ class PostgresHistory:
         if start >= end:
             raise ValueError("start must be before end")
 
-        query = text(
-            """
+        query = text("""
             SELECT m.timestamp, s.station_code, m.water_level,
                    m.discharge, m.discharge_percent
             FROM water_measurements m
@@ -62,8 +61,7 @@ class PostgresHistory:
               AND m.timestamp <= :end_time
             ORDER BY m.timestamp ASC
             LIMIT :limit
-            """
-        )
+            """)
         with self.engine.connect() as connection:
             rows = connection.execute(
                 query,
@@ -91,9 +89,9 @@ class PostgresHistory:
                         "station_code": station_code,
                         "water_level": water_level,
                         "discharge": discharge,
-                        "discharge_percent": float(row[4])
-                        if row[4] is not None
-                        else None,
+                        "discharge_percent": (
+                            float(row[4]) if row[4] is not None else None
+                        ),
                     }
                 )
             return result

src/rate_limiter.py

@@ -173,8 +173,10 @@ class RequestTracker:
                 "failed_requests": self.failed_requests,
                 "success_rate": self.successful_requests / self.total_requests,
                 "average_response_time": self.total_response_time / self.total_requests,
-                "last_request_time": self.last_request_time.isoformat()
-                if self.last_request_time
-                else None,
+                "last_request_time": (
+                    self.last_request_time.isoformat()
+                    if self.last_request_time
+                    else None
+                ),
                 "error_breakdown": dict(self.error_count_by_type),
             }

src/rid_reservoir.py

@@ -323,9 +323,11 @@ class RidReservoirStore:
             if not preserve_cols:
                 return f"INSERT OR REPLACE INTO {table} ({col_list}) VALUES ({params})"
             updates = ", ".join(
-                f"{c} = COALESCE(excluded.{c}, {table}.{c})"
-                if c in preserve_cols
-                else f"{c} = excluded.{c}"
+                (
+                    f"{c} = COALESCE(excluded.{c}, {table}.{c})"
+                    if c in preserve_cols
+                    else f"{c} = excluded.{c}"
+                )
                 for c in value_cols
             )
             return (
@@ -334,9 +336,11 @@ class RidReservoirStore:
             )
         if self.db_type == "postgresql":
             updates = ", ".join(
-                f"{c} = COALESCE(EXCLUDED.{c}, {table}.{c})"
-                if c in preserve_cols
-                else f"{c} = EXCLUDED.{c}"
+                (
+                    f"{c} = COALESCE(EXCLUDED.{c}, {table}.{c})"
+                    if c in preserve_cols
+                    else f"{c} = EXCLUDED.{c}"
+                )
                 for c in value_cols
             )
             return (
@@ -344,9 +348,11 @@ class RidReservoirStore:
                 f"ON CONFLICT ({conflict}) DO UPDATE SET {updates}"
             )
         updates = ", ".join(
-            f"{c} = COALESCE(VALUES({c}), {c})"
-            if c in preserve_cols
-            else f"{c} = VALUES({c})"
+            (
+                f"{c} = COALESCE(VALUES({c}), {c})"
+                if c in preserve_cols
+                else f"{c} = VALUES({c})"
+            )
             for c in value_cols
         )
         return (

src/static/dashboard.html

@@ -113,6 +113,25 @@
         .lang-toggle { padding: 9px 12px; font-size: .78rem; font-weight: 800; white-space: nowrap; }
         .lang-toggle[data-active-lang="th"] { background: var(--mint); border-color: var(--mint-border); color: var(--mint-ink); }
         .theme-toggle { padding: 9px 11px; font-size: .95rem; line-height: 1; }
+        .alerts-panel { margin-bottom: 14px; padding: 18px 20px; border-radius: 16px; background: var(--card); border: 1px solid var(--border); box-shadow: var(--shadow); }
+        .alerts-head { display: flex; justify-content: space-between; align-items: center; gap: 12px; }
+        .alerts-head h2 { margin: 0; font-size: 1.15rem; }
+        .alerts-close { background: transparent; border: 0; color: var(--muted); font-size: 1.1rem; cursor: pointer; padding: 4px 8px; }
+        .alerts-intro { color: var(--muted); font-size: .92rem; line-height: 1.5; margin: 8px 0 12px; }
+        .alerts-server { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; font-size: .9rem; margin-bottom: 12px; }
+        .alerts-server code { background: var(--surface); border: 1px solid var(--border); padding: 4px 8px; border-radius: 8px; font-size: .9rem; }
+        .alerts-server button { padding: 4px 10px; font-size: .8rem; }
+        .alerts-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 10px; }
+        .alerts-topic { border: 1px solid var(--border); border-radius: 12px; padding: 10px 12px; background: var(--surface); display: flex; flex-direction: column; gap: 4px; }
+        .alerts-topic .name { font-weight: 600; font-size: .95rem; }
+        .alerts-topic .desc { color: var(--muted); font-size: .82rem; line-height: 1.4; }
+        .alerts-topic .row { display: flex; align-items: center; gap: 8px; margin-top: 4px; flex-wrap: wrap; }
+        .alerts-topic code { font-size: .82rem; background: var(--card); border: 1px solid var(--border); padding: 2px 6px; border-radius: 6px; }
+        .alerts-topic a { font-size: .82rem; }
+        .alerts-topic.danger { border-color: rgba(220, 38, 38, .45); }
+        .alerts-topic.outlook { border-style: dashed; }
+        .alerts-foot { color: var(--muted); font-size: .82rem; margin: 12px 0 0; line-height: 1.6; }
+        .alerts-disclaimer { display: block; margin-top: 4px; }
         .leaflet-popup-content-wrapper, .leaflet-popup-tip { background: var(--card); color: var(--ink); }
         .leaflet-container a.leaflet-popup-close-button { color: var(--muted); }
         .leaflet-bar a, .leaflet-control-attribution { background: var(--surface); color: var(--ink); border-color: var(--border); }
@@ -155,6 +174,18 @@
         .stat-value { margin-top: 9px; font-size: 1.65rem; font-weight: 800; letter-spacing: -.04em; white-space: nowrap; }
         .stat-note { color: var(--muted); margin-top: 3px; font-size: .77rem; }
         .stat.stale { border-color: var(--red); background: rgba(204,75,55,.08); }
+        .skill-panel { border: 1px solid var(--border); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: var(--surface-3); }
+        .skill-head { display: flex; justify-content: space-between; align-items: baseline; gap: 12px; flex-wrap: wrap; }
+        .skill-headline { margin: 8px 0 10px; font-weight: 700; font-size: .9rem; }
+        .skill-headline.better { color: var(--green); }
+        .skill-headline.worse { color: var(--amber); }
+        .skill-table-wrap { overflow-x: auto; }
+        .skill-table { border-collapse: collapse; font-size: .76rem; width: 100%; min-width: 560px; }
+        .skill-table th { text-align: left; color: var(--muted); font-weight: 700; font-size: .66rem; text-transform: uppercase; letter-spacing: .06em; padding: 4px 8px; border-bottom: 1px solid var(--border); }
+        .skill-table td { padding: 5px 8px; border-bottom: 1px solid var(--border); white-space: nowrap; }
+        .skill-table tr.current td { font-weight: 700; }
+        .skill-table td.num { text-align: right; font-variant-numeric: tabular-nums; }
+        .skill-table td.dim { color: var(--muted); }
         .stat.stale .stat-value, .stat.stale .stat-note { color: var(--red); }
         .workspace { display: grid; grid-template-columns: minmax(0, 1fr) 330px; gap: 14px; min-height: 640px; }
         .map-card, .side-card { background: var(--card); border: 1px solid var(--border); border-radius: 19px; box-shadow: var(--shadow); overflow: hidden; }
@@ -336,10 +367,33 @@
             <button id="refresh-button" type="button" data-i18n="action.refresh">↻ Refresh</button>
             <button id="lang-toggle" class="lang-toggle" type="button" data-active-lang="en" aria-label="Switch to Thai">ไทย</button>
             <button id="theme-toggle" class="theme-toggle" type="button" data-i18n-aria="theme.toggle" aria-label="Switch to dark mode" title="Switch to dark mode">🌙</button>
+            <button id="alerts-button" type="button" data-i18n="alerts.button" style="display:none">🔔 Get alerts</button>
             <button id="replay-2024" type="button" data-i18n="replay.start">▶ Replay Oct 2024 flood</button>
         </div>
     </header>
 
+    <section id="alerts-panel" class="alerts-panel" style="display:none" aria-labelledby="alerts-title">
+        <div class="alerts-head">
+            <h2 id="alerts-title" data-i18n="alerts.title">Flood alerts on your phone</h2>
+            <button type="button" class="alerts-close" id="alerts-close" data-i18n-aria="alerts.close" aria-label="Close">✕</button>
+        </div>
+        <p class="alerts-intro" data-i18n="alerts.intro">Free push notifications when a gauge crosses its warning or danger level, and an all-clear when it drops back. No account: install the ntfy app (iOS / Android / any browser), add the server, subscribe to the topics you want. You get a message only when something changes: a few per flood, none in a quiet season.</p>
+        <div class="alerts-server">
+            <span data-i18n="alerts.server">Server</span>
+            <code id="alerts-server-url"></code>
+            <button type="button" id="alerts-copy" data-i18n="alerts.copy">Copy</button>
+        </div>
+        <div class="alerts-grid" id="alerts-topics"></div>
+        <p class="alerts-foot">
+            <span data-i18n="alerts.apps">Apps:</span>
+            <a href="https://apps.apple.com/us/app/ntfy/id1625396347" target="_blank" rel="noopener">iOS</a> ·
+            <a href="https://play.google.com/store/apps/details?id=io.heckel.ntfy" target="_blank" rel="noopener">Android</a> ·
+            <a href="https://f-droid.org/en/packages/io.heckel.ntfy/" target="_blank" rel="noopener">F-Droid</a> ·
+            <a id="alerts-web-link" href="#" target="_blank" rel="noopener" data-i18n="alerts.web">Web (no install)</a>
+            <span class="alerts-disclaimer" data-i18n="alerts.disclaimer">Unofficial community service, best effort. For official warnings follow ThaiWater / TMD / your district office.</span>
+        </p>
+    </section>
+
     <section id="flood-verdict" role="status" aria-live="polite" style="display:none;margin-bottom:14px;padding:15px 18px;border-radius:16px;border:1px solid;display:none">
         <div style="display:flex;gap:12px;align-items:baseline;flex-wrap:wrap">
             <strong id="verdict-icon" style="font-size:1.2rem"></strong>
@@ -417,6 +471,16 @@
             <div class="p1-peak" style="margin-top:7px" data-i18n="outlook.explainer">Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.</div>
         </div>
         <button type="button" class="zones-button" id="forecast-expand" style="display:none;margin-top:12px">Show all station forecasts ▾</button>
+        <div class="skill-panel" id="skill-panel" style="display:none">
+            <div class="skill-head">
+                <strong data-i18n="skill.title">Is the model getting better?</strong>
+                <span class="subtitle" id="skill-sub"></span>
+            </div>
+            <div class="skill-headline" id="skill-headline"></div>
+            <p class="subtitle skill-caveat" id="skill-caveat" style="margin:-4px 0 10px"></p>
+            <div class="skill-table-wrap"><table class="skill-table" id="skill-table"></table></div>
+            <p class="subtitle" style="margin:8px 0 0" data-i18n="skill.explain">Every hour the deployed model's 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.</p>
+        </div>
         <div class="forecast-grid" id="forecast-grid" style="display:none"></div>
     </section>
 
@@ -556,6 +620,51 @@
             'forecast.chip.peak': (lvl) => ` · peak ~${lvl} m`,
             'forecast.chip.heuristic': ' · heuristic fallback',
             'forecast.expand': (n) => `Show all ${n} station forecasts ▾`,
+            'skill.title': 'Is the model getting better?',
+            'skill.sub': (n, since) => `${n} verified 24 h forecasts for P.1 since ${since}`,
+            'skill.explain': 'Every hour the deployed model\'s 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.',
+            'skill.better': (v, prev, d) => `Current model ${v} is more accurate than ${prev}: peak error ${d} cm lower on the hours it has served.`,
+            'skill.worse': (v, prev, d) => `Current model ${v} has a higher peak error than ${prev} so far (+${d} cm).`,
+            'skill.caveat.quiet': 'All verified hours so far were below 2 m: this measures quiet-river accuracy only. The model is built and judged for flood onset (lead time before 3.70 m), which no quiet week can test — see the backtests in the documentation.',
+            'skill.caveat.regime': 'Versions served different weeks; the ≥ 2 m column compares them on the hours that matter.',
+            'alerts.button': '🔔 Get alerts',
+            'alerts.title': 'Flood alerts on your phone',
+            'alerts.intro': 'Free push notifications when a gauge crosses its warning or danger level, and an all-clear when it drops back. No account: install the ntfy app (iOS / Android / any browser), add the server, subscribe to the topics you want. You get a message only when something changes: a few per flood, none in a quiet season.',
+            'alerts.server': 'Server',
+            'alerts.copy': 'Copy',
+            'alerts.copied': 'Copied',
+            'alerts.close': 'Close',
+            'alerts.apps': 'Apps:',
+            'alerts.web': 'Web (no install)',
+            'alerts.disclaimer': 'Unofficial community service, best effort. For official warnings follow ThaiWater / TMD / your district office.',
+            'alerts.subscribe': 'Subscribe in app',
+            'alerts.t.warning': 'Any gauge: warning level',
+            'alerts.t.warning.d': 'One message when any Ping River gauge crosses its warning level, and when levels fall back. The one to pick if unsure.',
+            'alerts.t.danger': 'Any gauge: danger level',
+            'alerts.t.danger.d': 'Only the serious crossings, basin-wide. Highest priority: rings through Do Not Disturb on most phones.',
+            'alerts.t.p1.warning': 'Chiang Mai city (P.1) warning',
+            'alerts.t.p1.warning.d': 'Nawarat Bridge crosses 3.70 m (stage 1: low-lying riverside areas), and the all-clear.',
+            'alerts.t.p1.danger': 'Chiang Mai city (P.1) danger',
+            'alerts.t.p1.danger.d': 'Nawarat Bridge crosses 4.20 m (stage 5: inner city districts).',
+            'alerts.t.p103.warning': 'Ring Road 3 (P.103) warning',
+            'alerts.t.p103.warning.d': 'Downstream city gauge crosses 5.95 m.',
+            'alerts.t.outlook': 'Early warning (model forecast)',
+            'alerts.t.outlook.d': 'Experimental: the forecast model gives a ≥ 50 % chance that P.1 reaches its warning level within 24 h. Up to ~13 h earlier than the gauge, but it can be wrong.',
+            'alerts.t.status': 'Monitor status',
+            'alerts.t.status.d': 'Gauge feed stale / recovered. For people who rely on the dashboard.',
+            'skill.single': (v) => `Only ${v} has enough verified hours yet; the next retrain adds a row to compare.`,
+            'skill.young': (v, n, min) => `${v} has ${n} verified hours; a comparison needs ${min}.`,
+            'skill.none': 'No verified forecasts yet — the first appear 24 h after a model starts serving.',
+            'skill.col.version': 'Model',
+            'skill.col.period': 'Served',
+            'skill.col.n': 'Hours',
+            'skill.col.mae': 'Peak error',
+            'skill.col.bias': 'Bias',
+            'skill.col.pers': 'Persistence',
+            'skill.col.skill': 'Skill',
+            'skill.col.high': '≥ 2 m error',
+            'skill.cm': (v) => `${v} cm`,
+            'skill.na': '—',
             'forecast.collapse': 'Hide station forecasts ▴',
             'outlook.title': 'Chiang Mai city flood outlook · P.1 Nawarat Bridge',
             'outlook.explainer': 'Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.',
@@ -732,6 +841,51 @@
             'forecast.chip.peak': (lvl) => ` · ระดับสูงสุดประมาณ ${lvl} ม.`,
             'forecast.chip.heuristic': ' · ใช้การประมาณอย่างง่าย',
             'forecast.expand': (n) => `แสดงพยากรณ์ทั้ง ${n} สถานี ▾`,
+            'skill.title': 'โมเดลแม่นยำขึ้นหรือไม่?',
+            'skill.sub': (n, since) => `พยากรณ์ 24 ชม. ของ P.1 ที่ตรวจสอบแล้ว ${n} ครั้ง ตั้งแต่ ${since}`,
+            'skill.explain': 'ทุกชั่วโมงระบบบันทึกค่าพยากรณ์ระดับน้ำสูงสุดใน 24 ชม. ของ P.1 ไว้ เมื่อครบ 24 ชม. จึงนำมาเทียบกับระดับน้ำจริง "ทักษะ" คือโมเดลดีกว่าการสมมติว่าระดับน้ำคงที่มากเพียงใด (0 = ไม่ดีกว่า, 1 = สมบูรณ์แบบ) โมเดลที่ฝึกใหม่ด้วยข้อมูลมากขึ้นจะปรากฏเป็นแถวใหม่ จึงเห็นได้ว่าดีขึ้นจริงหรือไม่',
+            'skill.better': (v, prev, d) => `โมเดลปัจจุบัน ${v} แม่นยำกว่า ${prev}: ค่าคลาดเคลื่อนต่ำกว่า ${d} ซม. ในช่วงที่ให้บริการ`,
+            'skill.worse': (v, prev, d) => `โมเดลปัจจุบัน ${v} มีค่าคลาดเคลื่อนสูงกว่า ${prev} (+${d} ซม.)`,
+            'skill.caveat.quiet': 'ชั่วโมงที่ตรวจสอบทั้งหมดอยู่ต่ำกว่า 2 ม.: วัดได้เพียงความแม่นยำช่วงน้ำปกติ โมเดลถูกสร้างและประเมินสำหรับช่วงน้ำเริ่มท่วม (เวลาเตือนล่วงหน้าก่อน 3.70 ม.) ซึ่งสัปดาห์ปกติทดสอบไม่ได้ — ดูผลทดสอบย้อนหลังในเอกสาร',
+            'skill.caveat.regime': 'แต่ละเวอร์ชันให้บริการคนละช่วงเวลา คอลัมน์ ≥ 2 ม. เปรียบเทียบเฉพาะชั่วโมงที่สำคัญ',
+            'alerts.button': '🔔 รับการแจ้งเตือน',
+            'alerts.title': 'แจ้งเตือนน้ำท่วมบนมือถือของคุณ',
+            'alerts.intro': 'การแจ้งเตือนฟรีเมื่อระดับน้ำที่สถานีใดข้ามระดับเฝ้าระวังหรือระดับอันตราย และแจ้งเมื่อกลับสู่ปกติ ไม่ต้องสมัครสมาชิก: ติดตั้งแอป ntfy (iOS / Android / เบราว์เซอร์) เพิ่มเซิร์ฟเวอร์ แล้วเลือกหัวข้อที่ต้องการ คุณจะได้รับข้อความเฉพาะเมื่อมีการเปลี่ยนแปลง: ไม่กี่ข้อความต่อเหตุการณ์น้ำท่วม และไม่มีเลยในช่วงปกติ',
+            'alerts.server': 'เซิร์ฟเวอร์',
+            'alerts.copy': 'คัดลอก',
+            'alerts.copied': 'คัดลอกแล้ว',
+            'alerts.close': 'ปิด',
+            'alerts.apps': 'แอป:',
+            'alerts.web': 'เว็บ (ไม่ต้องติดตั้ง)',
+            'alerts.disclaimer': 'บริการชุมชนอย่างไม่เป็นทางการ พยายามอย่างดีที่สุด สำหรับคำเตือนอย่างเป็นทางการโปรดติดตาม ThaiWater / กรมอุตุนิยมวิทยา / สำนักงานอำเภอของคุณ',
+            'alerts.subscribe': 'สมัครในแอป',
+            'alerts.t.warning': 'สถานีใดก็ได้: ระดับเฝ้าระวัง',
+            'alerts.t.warning.d': 'หนึ่งข้อความเมื่อสถานีใดในแม่น้ำปิงข้ามระดับเฝ้าระวัง และเมื่อระดับน้ำลดลง หากไม่แน่ใจให้เลือกอันนี้',
+            'alerts.t.danger': 'สถานีใดก็ได้: ระดับอันตราย',
+            'alerts.t.danger.d': 'เฉพาะการข้ามระดับที่ร้ายแรง ทั้งลุ่มน้ำ ความสำคัญสูงสุด: ดังผ่านโหมดห้ามรบกวนในโทรศัพท์ส่วนใหญ่',
+            'alerts.t.p1.warning': 'เมืองเชียงใหม่ (P.1) ระดับเฝ้าระวัง',
+            'alerts.t.p1.warning.d': 'สะพานนวรัฐข้าม 3.70 ม. (ระยะที่ 1: พื้นที่ริมน้ำที่ต่ำ) และแจ้งเมื่อกลับสู่ปกติ',
+            'alerts.t.p1.danger': 'เมืองเชียงใหม่ (P.1) ระดับอันตราย',
+            'alerts.t.p1.danger.d': 'สะพานนวรัฐข้าม 4.20 ม. (ระยะที่ 5: ย่านใจกลางเมือง)',
+            'alerts.t.p103.warning': 'ถนนวงแหวน 3 (P.103) ระดับเฝ้าระวัง',
+            'alerts.t.p103.warning.d': 'สถานีท้ายเมืองข้าม 5.95 ม.',
+            'alerts.t.outlook': 'เตือนล่วงหน้า (แบบจำลองพยากรณ์)',
+            'alerts.t.outlook.d': 'ทดลอง: แบบจำลองพยากรณ์ให้โอกาส ≥ 50% ที่ P.1 จะถึงระดับเฝ้าระวังภายใน 24 ชม. เร็วกว่าสถานีวัดได้ถึง ~13 ชม. แต่อาจผิดพลาดได้',
+            'alerts.t.status': 'สถานะระบบ',
+            'alerts.t.status.d': 'ข้อมูลสถานีล่าช้า / กลับมาปกติ สำหรับผู้ที่พึ่งพาแดชบอร์ด',
+            'skill.single': (v) => `มีเพียง ${v} ที่มีข้อมูลตรวจสอบเพียงพอ การฝึกครั้งถัดไปจะเพิ่มแถวให้เปรียบเทียบ`,
+            'skill.young': (v, n, min) => `${v} มีข้อมูลตรวจสอบ ${n} ชั่วโมง ต้องการอย่างน้อย ${min} เพื่อเปรียบเทียบ`,
+            'skill.none': 'ยังไม่มีพยากรณ์ที่ตรวจสอบได้ — จะเริ่มมี 24 ชม. หลังโมเดลเริ่มทำงาน',
+            'skill.col.version': 'โมเดล',
+            'skill.col.period': 'ช่วงเวลา',
+            'skill.col.n': 'ชั่วโมง',
+            'skill.col.mae': 'คลาดเคลื่อน',
+            'skill.col.bias': 'อคติ',
+            'skill.col.pers': 'ระดับคงที่',
+            'skill.col.skill': 'ทักษะ',
+            'skill.col.high': 'คลาดเคลื่อน ≥ 2 ม.',
+            'skill.cm': (v) => `${v} ซม.`,
+            'skill.na': '—',
             'forecast.collapse': 'ซ่อนพยากรณ์รายสถานี ▴',
             'outlook.title': 'แนวโน้มน้ำท่วมเมืองเชียงใหม่ · P.1 สะพานนวรัฐ',
             'outlook.explainer': 'โอกาสที่ระดับน้ำจะถึงแต่ละระดับการท่วมตามประกาศทางการภายใน 24 ชม. — น้ำเริ่มท่วมเมืองที่ระดับ 1 (3.70 ม.) และแต่ละระดับจะท่วมพื้นที่เพิ่มขึ้น',
@@ -843,6 +997,60 @@
         return typeof value === 'function' ? value(...args) : value;
     }
 
+    // ---- public push notifications (ntfy) -------------------------------------
+    let ALERTS_CFG = null;
+    const ALERT_TOPICS = [
+        { key: 'warning',     topic: 'warning',     cls: '' },
+        { key: 'danger',      topic: 'danger',      cls: 'danger' },
+        { key: 'p1.warning',  topic: 'p1-warning',  cls: '' },
+        { key: 'p1.danger',   topic: 'p1-danger',   cls: 'danger' },
+        { key: 'p103.warning', topic: 'p103-warning', cls: '' },
+        { key: 'outlook',     topic: 'p1-outlook',  cls: 'outlook' },
+        { key: 'status',      topic: 'status',      cls: '' },
+    ];
+    async function loadAlertsConfig() {
+        try {
+            const r = await fetch('/api/notifications');
+            if (!r.ok) return;
+            const cfg = await r.json();
+            if (!cfg.enabled || !cfg.server) return;
+            ALERTS_CFG = cfg;
+            $('alerts-button').style.display = '';
+            renderAlertsPanel();
+        } catch (e) { /* no notifications configured */ }
+    }
+    function renderAlertsPanel() {
+        if (!ALERTS_CFG) return;
+        const server = ALERTS_CFG.server.replace(/\/$/, '');
+        const host = server.replace(/^https?:\/\//, '');
+        $('alerts-server-url').textContent = host;
+        $('alerts-web-link').href = server + '/' + ALERTS_CFG.prefix + '-warning';
+        $('alerts-topics').innerHTML = ALERT_TOPICS.map(tp => {
+            const full = ALERTS_CFG.prefix + '-' + tp.topic;
+            const url = server + '/' + full;
+            return `<div class="alerts-topic ${tp.cls}">
+                <div class="name">${esc(t('alerts.t.' + tp.key))}</div>
+                <div class="desc">${esc(t('alerts.t.' + tp.key + '.d'))}</div>
+                <div class="row"><code>${esc(full)}</code> <a href="ntfy://${esc(host)}/${esc(full)}">${esc(t('alerts.subscribe'))}</a> · <a href="${esc(url)}" target="_blank" rel="noopener">web</a></div>
+            </div>`;
+        }).join('');
+    }
+    function esc(x) { return String(x).replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c])); }
+    $('alerts-button').addEventListener('click', () => {
+        const p = $('alerts-panel');
+        const open = p.style.display === 'none';
+        p.style.display = open ? '' : 'none';
+        if (open) { renderAlertsPanel(); p.scrollIntoView({ behavior: 'smooth', block: 'start' }); }
+    });
+    $('alerts-close').addEventListener('click', () => { $('alerts-panel').style.display = 'none'; });
+    $('alerts-copy').addEventListener('click', async () => {
+        try {
+            await navigator.clipboard.writeText(ALERTS_CFG ? ALERTS_CFG.server : '');
+            $('alerts-copy').textContent = t('alerts.copied');
+            setTimeout(() => { $('alerts-copy').textContent = t('alerts.copy'); }, 1500);
+        } catch (e) { /* clipboard blocked */ }
+    });
+
     function applyTranslations() {
         document.documentElement.lang = state.lang;
         document.querySelectorAll('[data-i18n]').forEach((el) => {
@@ -897,6 +1105,7 @@
     function cssVar(name) { return getComputedStyle(document.documentElement).getPropertyValue(name).trim(); }
 
     function setLang(lang) {
+        setTimeout(renderAlertsPanel, 0);
         state.lang = lang;
         try { localStorage.setItem(LANG_KEY, lang); } catch (e) { /* private mode */ }
         applyTranslations();
@@ -1989,11 +2198,60 @@
                 ? t('forecast.collapse')
                 : t('forecast.expand', stations.length);
             card.style.display = 'block';
+            loadSkill(); // non-blocking; panel stays hidden until there is verified data
+            loadAlertsConfig(); // shows the "Get alerts" button only when ntfy is configured
         } catch (error) {
             card.style.display = 'none';
         }
     }
 
+    async function loadSkill() {
+        const panel = $('skill-panel');
+        try {
+            const response = await fetch('/api/forecast/skill?station_code=P.1&horizon=24');
+            if (!response.ok) throw new Error(`HTTP ${response.status}`);
+            const data = await response.json();
+            const versions = (data.versions || []).filter((v) => v.n > 0);
+            if (!versions.length) { panel.style.display = 'none'; return; }
+            const cm = (m) => m == null ? t('skill.na') : t('skill.cm', (m * 100).toFixed(1));
+            const fmtDay = (v) => parseTs(v).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' });
+            const total = versions.reduce((a, v) => a + v.n, 0);
+            $('skill-sub').textContent = t('skill.sub', total.toLocaleString(loc()), fmtDay(versions[0].first_issued));
+            const head = $('skill-headline');
+            head.className = 'skill-headline';
+            const cur = data.current;
+            if (data.trend && cur) {
+                const delta = Math.abs(data.trend.mae_delta_m * 100).toFixed(1);
+                head.textContent = data.trend.better
+                    ? t('skill.better', cur.model_version, data.trend.previous_version, delta)
+                    : t('skill.worse', cur.model_version, data.trend.previous_version, delta);
+                head.classList.add(data.trend.better ? 'better' : 'worse');
+            } else if (cur && cur.enough_data) {
+                head.textContent = t('skill.single', cur.model_version);
+            } else if (cur) {
+                head.textContent = t('skill.young', cur.model_version, cur.n, data.min_verified);
+            } else head.textContent = t('skill.none');
+            const anyHigh = versions.some((v) => v.above_2m_n > 0);
+            const compared = versions.filter((v) => v.enough_data).length > 1;
+            $('skill-caveat').textContent = !anyHigh ? t('skill.caveat.quiet') : compared ? t('skill.caveat.regime') : '';
+            const cols = ['version', 'period', 'n', 'mae', 'bias', 'pers', 'skill', 'high'];
+            const rows = versions.map((v) => `<tr class="${v === cur ? 'current' : ''}${v.enough_data ? '' : ' young'}">`
+                + `<td>${escapeHtml(v.model_version)}</td>`
+                + `<td class="dim">${fmtDay(v.first_issued)} – ${fmtDay(v.last_issued)}</td>`
+                + `<td class="num">${v.n.toLocaleString(loc())}</td>`
+                + `<td class="num">${cm(v.mae_m)}</td>`
+                + `<td class="num">${v.bias_m == null ? t('skill.na') : (v.bias_m >= 0 ? '+' : '') + (v.bias_m * 100).toFixed(1)}</td>`
+                + `<td class="num dim">${cm(v.persistence_mae_m)}</td>`
+                + `<td class="num">${v.skill == null ? t('skill.na') : v.skill.toFixed(2)}</td>`
+                + `<td class="num">${v.above_2m_n ? `${cm(v.above_2m_mae_m)} <span class="dim">(${v.above_2m_n})</span>` : t('skill.na')}</td>`
+                + '</tr>').join('');
+            $('skill-table').innerHTML = `<thead><tr>${cols.map((c) => `<th>${escapeHtml(t('skill.col.' + c))}</th>`).join('')}</tr></thead><tbody>${rows}</tbody>`;
+            panel.style.display = 'block';
+        } catch (error) {
+            panel.style.display = 'none';
+        }
+    }
+
     async function loadDbStats() {
         const strip = $('db-stats');
         try {

src/web_api.py

@@ -210,7 +210,9 @@ async def lifespan(app: FastAPI):
     app_state["leader_lock"] = _acquire_collection_leadership(
         Config.COLLECTION_LEADER_PORT
     )
+    app_state["notify"] = None
     if app_state["leader_lock"]:
+        app_state["notify"] = _init_notifications()
         app_state["scraping_task"] = asyncio.create_task(background_scraping_task())
         logger.info("This worker is the background-collection leader")
     else:
@@ -296,6 +298,73 @@ async def _persist_rain():
         logger.warning(f"rain persistence failed: {e}")
 
 
+def _init_notifications():
+    """Publisher + persisted state for ntfy, or None if off/unavailable.
+
+    Called only by the collection leader: it is the one process that
+    publishes, so the notification_state DDL runs exactly once per host.
+    """
+    if not Config.NTFY_SERVER:
+        return None
+    try:
+        from . import notify as notify_mod
+
+        store = app_state.get("forecast_store")
+        if store and not store.engine:
+            store.connect()
+        state = (
+            notify_mod.NotificationState(store.engine, store.db_type)
+            if store and store.engine
+            else notify_mod.InMemoryState()
+        )
+        if isinstance(state, notify_mod.InMemoryState):
+            logger.warning(
+                "ntfy: no SQL store; notification state is in-memory "
+                "(a restart may re-send the current level)"
+            )
+        publisher = notify_mod.NtfyPublisher(
+            Config.NTFY_PUBLISH_URL,
+            prefix=Config.NTFY_TOPIC_PREFIX,
+            token=Config.NTFY_TOKEN or None,
+            dashboard_url=Config.PUBLIC_URL,
+        )
+        logger.info(
+            f"ntfy notifications: publish to {Config.NTFY_PUBLISH_URL}, "
+            f"subscribers use {Config.NTFY_SERVER}, topics {Config.NTFY_TOPIC_PREFIX}-*"
+        )
+        return publisher, state
+    except Exception as e:
+        logger.error(f"ntfy init failed (notifications off): {e}")
+        return None
+
+
+async def _notify_transitions():
+    """Publish flood/outlook/feed transitions to ntfy (leader only, fail-safe)."""
+    cfg = app_state.get("notify")
+    if not cfg:
+        return
+    publisher, state = cfg
+    try:
+        from . import notify as notify_mod
+
+        scraper = app_state["scraper"]
+        readings = await asyncio.to_thread(
+            scraper.db_adapter.get_latest_measurements, 200
+        )
+        with FORECAST_CACHE_LOCK:
+            cached = FORECAST_CACHE.get("all")
+        forecasts = cached[1] if cached else []
+        sent = await asyncio.to_thread(
+            notify_mod.evaluate, readings, forecasts, state, publisher
+        )
+        if sent:
+            logger.info(
+                "ntfy: published " + ", ".join(f"{n.topic}: {n.title}" for n in sent)
+            )
+    except Exception as e:
+        logger.warning(f"ntfy notify cycle failed: {e}")
+
+
 async def _precompute_forecasts():
     """Refresh the forecast cache and persist the issued forecasts (leader only)."""
     try:
@@ -402,6 +471,10 @@ async def background_scraping_task():
                     # evaluation.
                     await _precompute_forecasts()
 
+                    # Push notifications for threshold crossings (uses the
+                    # forecasts just computed; no-op unless NTFY_SERVER set).
+                    await _notify_transitions()
+
                 app_state["is_scraping"] = False
 
             # Calculate next run time
@@ -864,7 +937,7 @@ def _hii_rows(sql: str, params: Dict[str, Any]) -> List[Dict[str, Any]]:
 # flood, slightly stale readings with a visible timestamp beat an error page.
 HII_CACHE: Dict[str, Any] = {}
 HII_CACHE_LOCK = Lock()
-_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock()}
+_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock(), "skill": Lock()}
 LATEST_CACHE: Dict[str, Any] = {}
 LATEST_CACHE_LOCK = Lock()
 _LATEST_COMPUTE_LOCK = Lock()
@@ -1230,6 +1303,78 @@ async def get_forecast_history(
     return await asyncio.to_thread(store.fetch, station_code, start_dt, end_dt, horizon)
 
 
+@app.get("/api/notifications")
+async def get_notifications_config():
+    """Public ntfy settings so the dashboard can offer subscribe links."""
+    server = Config.NTFY_SERVER
+    if not server:
+        return {"enabled": False}
+    prefix = Config.NTFY_TOPIC_PREFIX
+    return {
+        "enabled": True,
+        "server": server,
+        "prefix": prefix,
+        "topics": {
+            "warning": f"{prefix}-warning",
+            "danger": f"{prefix}-danger",
+            "p1_outlook": f"{prefix}-p1-outlook",
+            "status": f"{prefix}-status",
+            "station_pattern": f"{prefix}-<station>-warning | {prefix}-<station>-danger (station code lowercase, no dot: p1, p103)",
+        },
+        "semantics": "transitions only: one message on crossing up, one all-clear on the way down (0.10 m hysteresis)",
+    }
+
+
+@app.get("/api/forecast/skill")
+async def get_forecast_skill(
+    response: Response,
+    station_code: str = Query("P.1"),
+    horizon: int = Query(24, ge=1, le=48),
+):
+    """Is the model getting better? Issued forecasts verified against what the
+    river then did, per model version, with a persistence baseline.
+
+    Read from forecast_history (what each deployed version predicted, hourly)
+    joined to water_measurements; no retraining involved. Cached like the HII
+    feeds because the join is a few hundred correlated subqueries.
+    """
+    increment_counter("api_requests", labels={"endpoint": "forecast_skill"})
+    store = app_state.get("forecast_store")
+    if not store:
+        return {
+            "station_code": station_code,
+            "horizon_hours": horizon,
+            "versions": [],
+            "current": None,
+            "trend": None,
+        }
+
+    def compute():
+        from .ml import skill
+
+        if not store.engine and not store.connect():
+            return {
+                "station_code": station_code,
+                "horizon_hours": horizon,
+                "versions": [],
+                "current": None,
+                "trend": None,
+            }
+        return skill.compute_skill(store.engine, store.db_type, station_code, horizon)
+
+    payload, stale = await _cached_swr(
+        HII_CACHE,
+        HII_CACHE_LOCK,
+        _HII_COMPUTE_LOCKS["skill"],
+        f"skill:{station_code}:{horizon}",
+        max(Config.HII_CACHE_TTL_SECONDS, 900),
+        compute,
+    )
+    if stale:
+        response.headers["X-Data-Stale"] = "true"
+    return payload
+
+
 @app.get("/measurements/latest", response_model=List[MeasurementResponse])
 async def get_latest_measurements(response: Response, limit: int = 100):
     """Get latest measurements from all stations"""
@@ -1313,9 +1458,7 @@ async def get_database_stats():
         from sqlalchemy import text
 
         with engine.connect() as conn:
-            return conn.execute(
-                text(
-                    """
+            return conn.execute(text("""
                     SELECT (SELECT COUNT(*) FROM hii_rainfall) AS rain_n,
                            (SELECT COUNT(*) FROM hii_waterlevel) AS wl_n,
                            (SELECT COUNT(*) FROM hii_rain_stations) AS rain_s,
@@ -1324,9 +1467,7 @@ async def get_database_stats():
                            (SELECT MAX(timestamp) FROM hii_rainfall) AS rain_hi,
                            (SELECT MIN(timestamp) FROM hii_waterlevel) AS wl_lo,
                            (SELECT MAX(timestamp) FROM hii_waterlevel) AS wl_hi
-                    """
-                )
-            ).one()
+                    """)).one()
 
     def compute():
         # Heavy: full-table counts and coverage over ~1.7M rows. Runs at most

tests/test_forecast_skill.py

@@ -0,0 +1,93 @@
+"""Forecast skill verification: issued forecasts vs observed peaks (sqlite)."""
+
+import datetime
+
+import pytest
+from sqlalchemy import create_engine, text
+
+from src.ml import skill
+
+
+@pytest.fixture
+def engine(tmp_path):
+    eng = create_engine(f"sqlite:///{tmp_path / 'skill.db'}")
+    with eng.begin() as c:
+        c.execute(text("CREATE TABLE stations (id INTEGER PRIMARY KEY, station_code TEXT)"))
+        c.execute(text("INSERT INTO stations VALUES (1, 'P.1')"))
+        c.execute(
+            text(
+                "CREATE TABLE water_measurements (timestamp DATETIME, station_id INTEGER, water_level REAL)"
+            )
+        )
+        c.execute(
+            text(
+                "CREATE TABLE forecast_history (as_of TIMESTAMP, station_code TEXT, horizon_hours INTEGER, "
+                "predicted_max_level REAL, p_warning REAL, p_danger REAL, current_level REAL, "
+                "model_version TEXT, source TEXT)"
+            )
+        )
+    return eng
+
+
+def _fill(engine, start, hours, level_fn, forecasts):
+    """hours of hourly observations from `start`, plus (as_of_offset_h, version, pred) rows."""
+    with engine.begin() as c:
+        for h in range(hours):
+            ts = start + datetime.timedelta(hours=h)
+            c.execute(
+                text("INSERT INTO water_measurements VALUES (:t, 1, :l)"),
+                {"t": ts, "l": level_fn(h)},
+            )
+        for off, version, pred in forecasts:
+            ts = start + datetime.timedelta(hours=off)
+            c.execute(
+                text(
+                    "INSERT INTO forecast_history VALUES (:t, 'P.1', 24, :p, 0, 0, :cur, :v, 'model')"
+                ),
+                {"t": ts, "p": pred, "cur": level_fn(off), "v": version},
+            )
+
+
+def test_skill_per_version_and_trend(engine):
+    start = datetime.datetime(2026, 8, 1)
+    # river: flat 1.5 m, with a bump to 2.4 m around hour 100
+    level = lambda h: 2.4 if 96 <= h <= 104 else 1.5
+    forecasts = []
+    # old version: always predicts 1.5 (persistence-like, misses the bump)
+    for off in range(0, 60):
+        forecasts.append((off, "hgb-v2+aaaaaaa", 1.5))
+    # new version: predicts 1.5 normally and 2.3 ahead of the bump
+    for off in range(60, 200):
+        pred = 2.3 if 72 <= off <= 104 else 1.5
+        forecasts.append((off, "hgb-v3+bbbbbbb", pred))
+    _fill(engine, start, 260, level, forecasts)
+
+    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=300))
+    assert [v["model_version"] for v in out["versions"]] == ["hgb-v2+aaaaaaa", "hgb-v3+bbbbbbb"]
+    old, new = out["versions"]
+    assert old["n"] == 60 and old["enough_data"]
+    assert new["n"] == 140 and new["enough_data"]
+    # the old version issued only on flat hours: perfect there, no bump rows
+    assert old["mae_m"] == 0.0 and old["above_2m_n"] == 0
+    # the new version saw the bump: nonzero MAE but positive skill vs persistence
+    assert new["above_2m_n"] > 0
+    assert new["skill"] is not None and new["skill"] > 0
+    assert out["current"]["model_version"] == "hgb-v3+bbbbbbb"
+    assert out["trend"]["previous_version"] == "hgb-v2+aaaaaaa"
+    assert out["trend"]["better"] is False  # honest: old had an easier period
+
+
+def test_skill_requires_full_window(engine):
+    start = datetime.datetime(2026, 8, 1)
+    # forecasts issued at the very end have no observed window yet
+    _fill(engine, start, 30, lambda h: 1.5, [(o, "hgb-v3+ccccccc", 1.5) for o in range(0, 30)])
+    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=30))
+    # only as_of <= now-24h AND with >= 18 observed hours in the window count
+    assert out["versions"] and out["versions"][0]["n"] == 7  # as_of 0..6 h: <= now-24h with >= 18 observed hours
+    assert out["versions"][0]["enough_data"] is False
+    assert out["trend"] is None
+
+
+def test_skill_empty(engine):
+    out = skill.compute_skill(engine, "sqlite", "P.1", 24)
+    assert out["versions"] == [] and out["current"] is None and out["trend"] is None

tests/test_notify.py

@@ -0,0 +1,285 @@
+"""ntfy notification state machine: transitions only, hysteresis, restart-safe."""
+
+import datetime
+
+import pytest
+
+from src import notify
+
+
+class FakePublisher(notify.NtfyPublisher):
+    def __init__(self):
+        super().__init__("http://ntfy.test", prefix="ping")
+        self.sent = []
+
+    def publish(self, n):
+        self.sent.append(n)
+        return True
+
+
+@pytest.fixture
+def pub():
+    return FakePublisher()
+
+
+def _reading(code, level, ts="2026-09-24T12:00:00"):
+    return {"station_code": code, "water_level": level, "timestamp": ts}
+
+
+def _fc(p, peak=None):
+    return [
+        {
+            "station_code": "P.1",
+            "horizon_hours": 24,
+            "p_warning": p,
+            "predicted_max_level": peak,
+            "source": "model",
+        }
+    ]
+
+
+NOW = datetime.datetime(2026, 9, 24, 12, 30)
+
+
+def topics(pub):
+    return [n.topic for n in pub.sent]
+
+
+def test_quiet_river_sends_nothing(pub):
+    state = notify.InMemoryState()
+    for h in range(48):
+        notify.evaluate(
+            [_reading("P.1", 1.6), _reading("P.103", 3.2)],
+            _fc(0.01),
+            state,
+            pub,
+            now=NOW,
+        )
+    assert pub.sent == []
+
+
+def test_warning_crossing_once_then_silence_then_clear(pub):
+    state = notify.InMemoryState()
+    # rising through 3.70 (P.1 warning)
+    notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
+    assert pub.sent == []
+    notify.evaluate([_reading("P.1", 3.72)], [], state, pub, now=NOW)
+    assert topics(pub) == ["ping-p1-warning", "ping-warning"]
+    assert pub.sent[0].priority == 4 and "3.72 m" in pub.sent[0].message
+    # stays above: no repeats for many hours
+    for level in (3.80, 3.95, 4.05, 3.90, 3.75):
+        notify.evaluate([_reading("P.1", level)], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    # dips to 3.65: within hysteresis, still no message
+    notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    # 3.55: clear
+    notify.evaluate([_reading("P.1", 3.55)], [], state, pub, now=NOW)
+    assert topics(pub)[2:] == ["ping-p1-warning", "ping-warning"]
+    assert "back to normal" in pub.sent[2].title
+
+
+def test_danger_escalation_and_deescalation(pub):
+    state = notify.InMemoryState()
+    notify.evaluate([_reading("P.1", 3.9)], [], state, pub, now=NOW)  # warning
+    notify.evaluate(
+        [_reading("P.1", 4.25)], [], state, pub, now=NOW
+    )  # danger (>= 4.20)
+    assert topics(pub) == [
+        "ping-p1-warning",
+        "ping-warning",
+        "ping-p1-danger",
+        "ping-danger",
+    ]
+    assert pub.sent[2].priority == 5
+    notify.evaluate(
+        [_reading("P.1", 4.15)], [], state, pub, now=NOW
+    )  # hysteresis: still danger
+    assert len(pub.sent) == 4
+    notify.evaluate([_reading("P.1", 4.05)], [], state, pub, now=NOW)  # back to warning
+    assert topics(pub)[4:] == ["ping-p1-danger", "ping-warning"]
+    assert "below danger" in pub.sent[4].title
+
+
+def test_jump_straight_to_danger(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [_reading("P.103", 7.0)], [], state, pub, now=NOW
+    )  # P.103 danger 6.75
+    assert topics(pub) == ["ping-p103-danger", "ping-danger"]
+
+
+def test_basin_digest_groups_stations(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [_reading("P.1", 3.8), _reading("P.103", 6.0), _reading("P.67", 1.0)],
+        [],
+        state,
+        pub,
+        now=NOW,
+    )
+    basin = [n for n in pub.sent if n.topic == "ping-warning"]
+    assert len(basin) == 1 and "P.1" in basin[0].message and "P.103" in basin[0].message
+
+
+def test_outlook_on_off_with_hysteresis(pub):
+    state = notify.InMemoryState()
+    r = [_reading("P.1", 2.9)]
+    notify.evaluate(r, _fc(0.30), state, pub, now=NOW)
+    assert pub.sent == []
+    notify.evaluate(r, _fc(0.55, 3.9), state, pub, now=NOW)
+    assert topics(pub) == ["ping-p1-outlook"]
+    assert "55%" in pub.sent[0].message and "3.90 m" in pub.sent[0].message
+    assert "not an official warning" in pub.sent[0].message
+    notify.evaluate(
+        r, _fc(0.40), state, pub, now=NOW
+    )  # between OFF and ON: stays on, silent
+    assert len(pub.sent) == 1
+    notify.evaluate(r, _fc(0.20), state, pub, now=NOW)
+    assert len(pub.sent) == 2 and "easing" in pub.sent[1].title
+
+
+def test_heuristic_forecast_ignored(pub):
+    state = notify.InMemoryState()
+    fc = [
+        {
+            "station_code": "P.1",
+            "horizon_hours": 24,
+            "p_warning": 0.9,
+            "source": "heuristic",
+        }
+    ]
+    notify.evaluate([_reading("P.1", 2.0)], fc, state, pub, now=NOW)
+    assert pub.sent == []
+
+
+def test_stale_feed_and_recovery(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=NOW
+    )
+    assert pub.sent == []
+    later = NOW + datetime.timedelta(hours=4)
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=later
+    )
+    assert topics(pub) == ["ping-status"] and "stale" in pub.sent[0].title
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T12:00:00")],
+        [],
+        state,
+        pub,
+        now=later + datetime.timedelta(hours=1),
+    )
+    assert len(pub.sent) == 1  # still stale, no repeat
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T17:00:00")],
+        [],
+        state,
+        pub,
+        now=later + datetime.timedelta(hours=1),
+    )
+    assert len(pub.sent) == 2 and "recovered" in pub.sent[1].title
+
+
+def test_capacity_guard_blocks_stale_threshold(pub):
+    """P.77 2026-09: 3.02 m >= 2.85 m 'warning' at 22 % capacity -> not a flood."""
+    state = notify.InMemoryState()
+    r = {
+        "station_code": "P.77",
+        "water_level": 4.40,
+        "timestamp": "2026-09-24T12:00:00",
+        "discharge_percent": 10.3,
+    }
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert pub.sent == [] and state.get("level:P.77") is None
+    # same level with capacity agreeing -> alert
+    r["discharge_percent"] = 82.0
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert topics(pub) == ["ping-p77-warning", "ping-warning"]
+
+
+def test_capacity_guard_exempts_p1_and_missing_pct(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [
+            {
+                "station_code": "P.1",
+                "water_level": 3.75,
+                "timestamp": "2026-09-24T12:00:00",
+                "discharge_percent": 40.0,
+            }
+        ],
+        [],
+        state,
+        pub,
+        now=NOW,
+    )
+    assert topics(pub) == ["ping-p1-warning", "ping-warning"]
+    pub.sent.clear()
+    notify.evaluate(
+        [
+            {
+                "station_code": "P.103",
+                "water_level": 6.0,
+                "timestamp": "2026-09-24T12:00:00",
+            }
+        ],
+        [],
+        state,
+        pub,
+        now=NOW,
+    )
+    assert topics(pub) == ["ping-p103-warning", "ping-warning"]
+
+
+def test_capacity_guard_does_not_block_clearing(pub):
+    """Guard applies only to the clear->alert edge; the all-clear always goes out."""
+    state = notify.InMemoryState()
+    r = {
+        "station_code": "P.67",
+        "water_level": 2.6,
+        "timestamp": "2026-09-24T12:00:00",
+        "discharge_percent": 90.0,
+    }
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    r.update(water_level=2.2, discharge_percent=30.0)
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert "back to normal" in pub.sent[2].title
+
+
+def test_state_survives_restart_via_sql(tmp_path, pub):
+    from sqlalchemy import create_engine
+
+    eng = create_engine(f"sqlite:///{tmp_path / 'n.db'}")
+    state = notify.NotificationState(eng, "sqlite")
+    notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    # "restart": new state object on the same DB, same reading -> nothing re-sent
+    state2 = notify.NotificationState(eng, "sqlite")
+    notify.evaluate([_reading("P.1", 3.8)], [], state2, pub, now=NOW)
+    assert len(pub.sent) == 2
+
+
+def test_publish_failure_does_not_advance_state():
+    """If ntfy is down the transition must be retried next cycle, not lost."""
+
+    class Down(notify.NtfyPublisher):
+        def __init__(self):
+            super().__init__("http://ntfy.test")
+            self.calls = 0
+
+        def publish(self, n):
+            self.calls += 1
+            return False
+
+    pub = Down()
+    state = notify.InMemoryState()
+    notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
+    assert pub.calls == 2 and state.get("level:P.1") is None
+    # next cycle, ntfy back: the crossing is delivered
+    good = FakePublisher()
+    notify.evaluate([_reading("P.1", 3.8)], [], state, good, now=NOW)
+    assert topics(good) == ["ping-p1-warning", "ping-warning"]
+    assert state.get("level:P.1") == "warning"