feat: recalibrate P.77/P.75 thresholds; capacity guard on alerts

The first ntfy cycle announced "Warning level at P.77" at 3.02 m. That gauge's 2.85 m threshold sat below its own dry-season baseline (2.6-2.7 m at 8-14 % channel capacity): P.77 had been "above warning" for 761 of the last 2 146 hours, at 22 % capacity. Across 2018-2024, 75-85 % capacity reads 3.35-4.57 m and 95-105 % reads 4.27-5.08 m; set 4.30 / 4.90. P.75 moved 2.75/3.50 -> 3.20/3.65 on the same evidence (2024: 3.45 / 3.72). The predictor already handles changed thresholds (regression-derived probabilities until the Oct 1 retrain). Second line of defence in notify.py: a clear->alert transition is only announced when RID's discharge_percent for the reading is >= 60 %, so a re-rated or datum-shifted gauge cannot page subscribers again. P.1 is exempt (its stages come from the inundation map, not capacity); readings without a capacity figure fall back to level only; the all-clear edge is never blocked. 3 tests.
fix: ntfy listens on the Tailscale address; monitor publishes to it directly
2026-09-12 00:34:59 +02:00 · 2026-09-12 00:28:29 +02:00 · 2026-09-12 00:22:03 +02:00 · 2026-09-12 00:18:38 +02:00 · 2026-09-11 23:54:44 +02:00 · 2026-09-11 23:44:44 +02:00
55 changed files with 4690 additions and 2379 deletions
@@ -84,6 +84,19 @@ SMTP_PORT=587
 SMTP_USERNAME=
 SMTP_PASSWORD=
 # Public push notifications via self-hosted ntfy (https://ntfy.sh, single binary).
 # Leave NTFY_SERVER empty to disable. Topics published: <prefix>-<station>-warning,
 # <prefix>-<station>-danger, <prefix>-warning, <prefix>-danger, <prefix>-p1-outlook,
 # <prefix>-status. See docs/NOTIFICATIONS.md.
 NTFY_SERVER=
 # Where the monitor POSTs (defaults to NTFY_SERVER). Use the local ntfy
 # address (loopback or Tailscale IP) so publishing does not depend on
 # DNS / the reverse proxy being up.
 NTFY_PUBLISH_URL=
 NTFY_TOPIC_PREFIX=ping
 NTFY_TOKEN=
 PUBLIC_URL=https://water.buildfor.life/
 # Matrix Alerting Configuration
 MATRIX_HOMESERVER=https://matrix.org
 MATRIX_ACCESS_TOKEN=
@@ -1,4 +1,13 @@
-name: CI/CD Pipeline - Northern Thailand Ping River Monitor
+name: CI
 # What this checks, on every push and PR to master:
 #   1. formatting contract (black + isort, config in pyproject.toml)
 #   2. flake8 hard-error gate (syntax, undefined names)
 #   3. the pytest suite (synthetic data, no DB/network; ~1 min)
 # Docker build / staging / production / perf jobs from the original template
 # were removed: there is no registry, no staging host, and production is a
 # systemd unit deployed by `git pull` on the server (docs/FLOOD_FORECASTING.md
 # section 6, scripts/install.sh). Re-add a job when the thing it deploys exists.
 on:
  push:
@@ -6,337 +15,61 @@ on:
  pull_request:
    branches: [master]
  schedule:
-    # Run tests daily at 2 AM UTC
+    # daily, catches dependency drift / upstream API changes in the tests
-    - cron: '0 2 * * *'
+    - cron: "0 2 * * *"
  workflow_dispatch:
 env:
-  PYTHON_VERSION: '3.11'
+  PYTHON_VERSION: "3.11" # pandas 2.0.3 ships no 3.12 wheels; psycopg2-binary 2.9.9 breaks on 3.13
  REGISTRY: git.b4l.co.th
  IMAGE_NAME: b4l/northern-thailand-ping-river-monitor
  # GitHub token for better rate limits and authentication
  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 jobs:
-  # Test job
+  lint:
    name: Format & lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
          cache-dependency-path: requirements-dev.txt
      - name: Install tools
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore black==26.5.1 isort==5.12.0 flake8==6.1.0
      - name: black
        run: black --check --diff src/ *.py
      - name: isort
        run: isort --check-only --diff src/ *.py
      - name: flake8 (errors only)
        run: flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
  test:
-    name: Test Suite
+    name: Test suite
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.11']  # pandas 2.0.3 ships no 3.12 wheels; widen after upgrading pandas
    steps:
-    - name: Checkout code
+      - uses: actions/checkout@v4
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
-    - name: Set up Python ${{ matrix.python-version }}
+      - uses: actions/setup-python@v5
      uses: actions/setup-python@v4
        with:
-        python-version: ${{ matrix.python-version }}
+          python-version: ${{ env.PYTHON_VERSION }}
-        
+          cache: pip
-    - name: Cache pip dependencies
+          cache-dependency-path: |
-      uses: actions/cache@v3
+            requirements.txt
-      with:
+            requirements-dev.txt
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore -r requirements.txt
-        pip install --root-user-action=ignore -r requirements-dev.txt
+          pip install --root-user-action=ignore pytest==9.1.1 pytest-asyncio==0.21.1
-    - name: Lint with flake8
+      - name: pytest
      run: |
        flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 src/ --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
    - name: Type check with mypy (advisory)
      run: |
        # 86 pre-existing errors; blocking typing gate deferred until the debt is paid down
        mypy src/ --ignore-missing-imports || true
    - name: Format check with black
      run: |
        black --check src/ *.py
    - name: Import sort check
      run: |
        isort --check-only src/ *.py
    - name: Run integration tests
      run: |
        python tests/test_integration.py
    - name: Run station management tests
      run: |
        python tests/test_station_management.py
    - name: Test application startup
      run: |
        timeout 10s python run.py --test || true
    - name: Security scan with bandit
      run: |
        bandit -r src/ -f json -o bandit-report.json || true
    - name: Upload test artifacts
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: test-results-${{ matrix.python-version }}
        path: |
          bandit-report.json
          *.log
  # Code quality job
  code-quality:
    name: Code Quality
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip --root-user-action=ignore
        pip install --root-user-action=ignore -r requirements-dev.txt
    - name: Run safety check
      run: |
        safety check -r requirements.txt --json --output safety-report.json || true
    - name: Run bandit security scan
      run: |
        bandit -r src/ -f json -o bandit-report.json || true
    - name: Upload security reports
      uses: actions/upload-artifact@v3
      with:
        name: security-reports
        path: |
          safety-report.json
          bandit-report.json
  # Build Docker image
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    needs: test
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
    - name: Log in to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITEA_TOKEN }}
    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=ref,event=branch
          type=ref,event=pr
          type=sha,prefix={{branch}}-
          type=raw,value=latest,enable={{is_default_branch}}
    - name: Build and push Docker image
      uses: docker/build-push-action@v5
      with:
        context: .
        platforms: linux/amd64,linux/arm64
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max
        env:
-        GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
+          DB_TYPE: sqlite
-        
+        run: pytest -q -p no:cacheprovider
    - name: Test Docker image
      run: |
        docker run --rm ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} python run.py --test
  # Integration test with services
  integration-test:
    name: Integration Test with Services
    runs-on: ubuntu-latest
    needs: build
    services:
      victoriametrics:
        image: victoriametrics/victoria-metrics:latest
        ports:
          - 8428:8428
        options: >-
          --health-cmd "wget --quiet --tries=1 --spider http://localhost:8428/health"
          --health-interval 30s
          --health-timeout 10s
          --health-retries 3
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Wait for VictoriaMetrics
      run: |
        timeout 60s bash -c 'until curl -f http://localhost:8428/health; do sleep 2; done'
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip --root-user-action=ignore
        pip install --root-user-action=ignore -r requirements.txt
    - name: Test with VictoriaMetrics
      env:
        DB_TYPE: victoriametrics
        VM_HOST: localhost
        VM_PORT: 8428
      run: |
        python run.py --test
    - name: Start API server
      env:
        DB_TYPE: victoriametrics
        VM_HOST: localhost
        VM_PORT: 8428
      run: |
        python run.py --web-api &
        sleep 10
    - name: Test API endpoints
      run: |
        curl -f http://localhost:8000/health
        curl -f http://localhost:8000/stations
        curl -f http://localhost:8000/metrics
  # Deploy to staging (only on develop branch)
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: [test, build, integration-test]
    if: github.ref == 'refs/heads/develop'
    environment:
      name: staging
      url: https://staging.ping-river-monitor.b4l.co.th
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Deploy to staging
      run: |
        echo "Deploying to staging environment..."
        # Add your staging deployment commands here
        # Example: kubectl, docker-compose, or webhook call
    - name: Health check staging
      run: |
        sleep 30
        curl -f https://staging.ping-river-monitor.b4l.co.th/health
  # Deploy to production (only on main branch, manual approval)
  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: [test, build, integration-test]
    if: github.ref == 'refs/heads/master'
    environment:
      name: production
      url: https://ping-river-monitor.b4l.co.th
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Deploy to production
      run: |
        echo "Deploying to production environment..."
        # Add your production deployment commands here
    - name: Health check production
      run: |
        sleep 30
        curl -f https://ping-river-monitor.b4l.co.th/health
    - name: Notify deployment
      run: |
        echo "✅ Production deployment successful!"
        echo "🌐 URL: https://ping-river-monitor.b4l.co.th"
        echo "📊 Grafana: https://grafana.ping-river-monitor.b4l.co.th"
  # Performance test (only on main branch)
  performance-test:
    name: Performance Test
    runs-on: ubuntu-latest
    needs: deploy-production
    if: github.ref == 'refs/heads/master'
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Install Apache Bench
      run: |
        sudo apt-get update
        sudo apt-get install -y apache2-utils
    - name: Performance test API endpoints
      run: |
        # Test health endpoint
        ab -n 100 -c 10 https://ping-river-monitor.b4l.co.th/health
        # Test stations endpoint
        ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/stations
        # Test metrics endpoint
        ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/metrics
  # Cleanup old artifacts
  cleanup:
    name: Cleanup
    runs-on: ubuntu-latest
    if: always()
    needs: [test, build, integration-test]
    steps:
    - name: Clean up old Docker images
      run: |
        echo "Cleaning up old Docker images..."
        # Add cleanup commands for old images/artifacts
@@ -1,367 +1,99 @@
-name: Documentation
+name: Docs
 # Checks that the documentation the project actually ships stays consistent:
 #   - every relative link / image path in docs/*.md and README.md resolves
 #     inside the repo (external URLs are NOT fetched: localhost examples,
 #     rate-limited hosts and the Tailscale-era links made that gate permanently
 #     red, and a 200 on a curl --head proves nothing about a doc anyway)
 #   - the FastAPI app imports and its OpenAPI schema is exportable (that is
 #     the reference at https://water.buildfor.life/docs)
 # The previous Sphinx/apidoc jobs produced artifacts nobody read and were
 # removed. Reference docs live in docs/*.md; the public overview is at
 # https://buildfor.life/docs/tooling/ping-river-monitor/.
 on:
  push:
    branches: [master, develop]
    paths:
-      - 'docs/**'
+      - "docs/**"
-      - 'README.md'
+      - "README.md"
-      - 'CONTRIBUTING.md'
+      - "CONTRIBUTING.md"
-      - 'src/**/*.py'
+      - "src/web_api.py"
      - "src/schemas.py"
      - ".gitea/workflows/docs.yml"
  pull_request:
    paths:
-      - 'docs/**'
+      - "docs/**"
-      - 'README.md'
+      - "README.md"
-      - 'CONTRIBUTING.md'
+      - "CONTRIBUTING.md"
  workflow_dispatch:
 env:
-  PYTHON_VERSION: '3.11'
+  PYTHON_VERSION: "3.11"
 jobs:
-  # Validate documentation
+  docs:
-  validate-docs:
+    name: Validate documentation
    name: Validate Documentation
    runs-on: ubuntu-latest
    steps:
-    - name: Checkout code
+      - uses: actions/checkout@v4
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
-    - name: Set up Python
+      - name: Relative links and images resolve
-      uses: actions/setup-python@v4
+        run: |
-      with:
+          python3 - <<'PY'
-        python-version: ${{ env.PYTHON_VERSION }}
+          import re, sys, pathlib
-        
+          root = pathlib.Path(".")
-    - name: Install documentation tools
+          files = [root / "README.md", root / "CONTRIBUTING.md", *root.glob("docs/**/*.md")]
-      run: |
+          link = re.compile(r"!?\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
-        python -m pip install --upgrade pip
+          bad = []
-        pip install -r requirements.txt
+          for md in files:
-        pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
+              if not md.exists():
-        pip install markdown-link-check || true
+                  continue
-        
+              for m in link.finditer(md.read_text(encoding="utf-8")):
-    - name: Check markdown links
+                  target = m.group(1)
-      run: |
+                  if target.startswith(("http://", "https://", "mailto:", "#")):
-        echo "🔗 Checking markdown links..."
+                      continue
-        find . -name "*.md" -not -path "./.git/*" -not -path "./node_modules/*" | while read file; do
+                  path = target.split("#", 1)[0]
-          echo "Checking $file"
+                  if not path:
-          # Basic link validation (you can enhance this)
+                      continue
-          grep -o 'http[s]*://[^)]*' "$file" | while read url; do
+                  resolved = (md.parent / path).resolve()
-            if curl -s --head "$url" | head -n 1 | grep -q "200 OK"; then
+                  if not resolved.exists():
-              echo "✅ $url"
+                      bad.append(f"{md}: {target}")
-            else
+          if bad:
-              echo "❌ $url (in $file)"
+              print("Broken relative links:")
-            fi
+              print("\n".join("  " + b for b in bad))
-          done
+              sys.exit(1)
-        done
+          print(f"checked {len(files)} files, all relative links resolve")
-        
+          PY
-    - name: Validate README structure
+
-      run: |
+      - uses: actions/setup-python@v5
        echo "📋 Validating README structure..."
        required_sections=(
          "# Northern Thailand Ping River Monitor"
          "## Features"
          "## Quick Start"
          "## Installation"
          "## Usage"
          "## API Endpoints"
          "## Docker"
          "## Contributing"
          "## License"
        )
        for section in "${required_sections[@]}"; do
          if grep -q "$section" README.md; then
            echo "✅ Found: $section"
          else
            echo "❌ Missing: $section"
          fi
        done
    - name: Check documentation completeness
      run: |
        echo "📚 Checking documentation completeness..."
        # Check if all Python modules have docstrings
        python -c "
        import ast
        import os
        def check_docstrings(filepath):
            with open(filepath, 'r', encoding='utf-8') as f:
                tree = ast.parse(f.read())
            missing_docstrings = []
            for node in ast.walk(tree):
                if isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef)):
                    if not ast.get_docstring(node):
                        missing_docstrings.append(f'{node.name} in {filepath}')
            return missing_docstrings
        all_missing = []
        for root, dirs, files in os.walk('src'):
            for file in files:
                if file.endswith('.py') and not file.startswith('__'):
                    filepath = os.path.join(root, file)
                    missing = check_docstrings(filepath)
                    all_missing.extend(missing)
        if all_missing:
            print('⚠️ Missing docstrings:')
            for item in all_missing[:10]:  # Show first 10
                print(f'  - {item}')
            if len(all_missing) > 10:
                print(f'  ... and {len(all_missing) - 10} more')
        else:
            print('✅ All functions and classes have docstrings')
        "
  # Generate API documentation
  generate-api-docs:
    name: Generate API Documentation
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Python
      uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
          cache-dependency-path: requirements.txt
      - name: Install dependencies
        run: |
-        python -m pip install --upgrade pip
+          python -m pip install --upgrade pip --root-user-action=ignore
-        pip install -r requirements.txt
+          pip install --root-user-action=ignore -r requirements.txt
-    - name: Generate OpenAPI spec
+      - name: OpenAPI schema exports
        env:
          DB_TYPE: sqlite
        run: |
-        echo "📝 Generating OpenAPI specification..."
+          python - <<'PY'
        python -c "
          import json
-        import sys
+          from src.web_api import app
-        sys.path.insert(0, 'src')
+          spec = app.openapi()
          paths = sorted(spec["paths"])
          required = {"/forecast", "/measurements/latest", "/measurements/history/{station_code}", "/stations", "/api/stats", "/health"}
          missing = required - set(paths)
          assert not missing, f"documented endpoints missing from the app: {missing}"
          json.dump(spec, open("openapi.json", "w"), indent=1)
          print(f"{len(paths)} paths; schema written to openapi.json")
          PY
-        try:
+      - uses: actions/upload-artifact@v3
            from web_api import app
            openapi_spec = app.openapi()
            with open('openapi.json', 'w') as f:
                json.dump(openapi_spec, f, indent=2)
            print('✅ OpenAPI spec generated: openapi.json')
        except Exception as e:
            print(f'❌ Failed to generate OpenAPI spec: {e}')
        "
    - name: Generate API documentation
      run: |
        echo "📖 Generating API documentation..."
        # Create API documentation from OpenAPI spec
        if [ -f openapi.json ]; then
          cat > api-docs.md << 'EOF'
        # API Documentation
        This document describes the REST API endpoints for the Northern Thailand Ping River Monitor.
        ## Base URL
        - Production: `https://ping-river-monitor.b4l.co.th`
        - Staging: `https://staging.ping-river-monitor.b4l.co.th`
        - Development: `http://localhost:8000`
        ## Authentication
        Currently, the API does not require authentication. This may change in future versions.
        ## Endpoints
        EOF
          # Extract endpoints from OpenAPI spec
          python -c "
        import json
        with open('openapi.json', 'r') as f:
            spec = json.load(f)
        for path, methods in spec.get('paths', {}).items():
            for method, details in methods.items():
                print(f'### {method.upper()} {path}')
                print()
                print(details.get('summary', 'No description available'))
                print()
                if 'parameters' in details:
                    print('**Parameters:**')
                    for param in details['parameters']:
                        print(f'- `{param[\"name\"]}` ({param.get(\"in\", \"query\")}): {param.get(\"description\", \"No description\")}')
                    print()
                print('---')
                print()
          " >> api-docs.md
          echo "✅ API documentation generated: api-docs.md"
        fi
    - name: Upload documentation artifacts
      uses: actions/upload-artifact@v3
        with:
-        name: documentation-${{ github.run_number }}
+          name: openapi-${{ github.run_number }}
-        path: |
+          path: openapi.json
          openapi.json
          api-docs.md
  # Build Sphinx documentation
  build-sphinx-docs:
    name: Build Sphinx Documentation
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
    - name: Create Sphinx configuration
      run: |
        mkdir -p docs/sphinx
        cat > docs/sphinx/conf.py << 'EOF'
        import os
        import sys
        sys.path.insert(0, os.path.abspath('../../src'))
        project = 'Northern Thailand Ping River Monitor'
        copyright = '2025, Ping River Monitor Team'
        author = 'Ping River Monitor Team'
        version = '3.1.3'
        release = '3.1.3'
        extensions = [
            'sphinx.ext.autodoc',
            'sphinx.ext.viewcode',
            'sphinx.ext.napoleon',
            'sphinx_autodoc_typehints',
        ]
        templates_path = ['_templates']
        exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
        html_theme = 'sphinx_rtd_theme'
        html_static_path = ['_static']
        autodoc_default_options = {
            'members': True,
            'member-order': 'bysource',
            'special-members': '__init__',
            'undoc-members': True,
            'exclude-members': '__weakref__'
        }
        EOF
        cat > docs/sphinx/index.rst << 'EOF'
        Northern Thailand Ping River Monitor Documentation
        ================================================
        .. toctree::
           :maxdepth: 2
           :caption: Contents:
           modules
        Indices and tables
        ==================
        * :ref:`genindex`
        * :ref:`modindex`
        * :ref:`search`
        EOF
    - name: Generate module documentation
      run: |
        cd docs/sphinx
        sphinx-apidoc -o . ../../src
    - name: Build documentation
      run: |
        cd docs/sphinx
        sphinx-build -b html . _build/html
    - name: Upload Sphinx documentation
      uses: actions/upload-artifact@v3
      with:
        name: sphinx-docs-${{ github.run_number }}
        path: docs/sphinx/_build/html/
  # Documentation summary
  docs-summary:
    name: Documentation Summary
    runs-on: ubuntu-latest
    needs: [validate-docs, generate-api-docs, build-sphinx-docs]
    if: always()
    steps:
    - name: Generate documentation summary
      run: |
        echo "# 📚 Documentation Build Summary" > docs-summary.md
        echo "" >> docs-summary.md
        echo "**Build Date:** $(date -u)" >> docs-summary.md
        echo "**Repository:** ${{ github.repository }}" >> docs-summary.md
        echo "**Commit:** ${{ github.sha }}" >> docs-summary.md
        echo "" >> docs-summary.md
        echo "## 📊 Results" >> docs-summary.md
        echo "" >> docs-summary.md
        if [ "${{ needs.validate-docs.result }}" = "success" ]; then
          echo "- ✅ **Documentation Validation**: Passed" >> docs-summary.md
        else
          echo "- ❌ **Documentation Validation**: Failed" >> docs-summary.md
        fi
        if [ "${{ needs.generate-api-docs.result }}" = "success" ]; then
          echo "- ✅ **API Documentation**: Generated" >> docs-summary.md
        else
          echo "- ❌ **API Documentation**: Failed" >> docs-summary.md
        fi
        if [ "${{ needs.build-sphinx-docs.result }}" = "success" ]; then
          echo "- ✅ **Sphinx Documentation**: Built" >> docs-summary.md
        else
          echo "- ❌ **Sphinx Documentation**: Failed" >> docs-summary.md
        fi
        echo "" >> docs-summary.md
        echo "## 🔗 Available Documentation" >> docs-summary.md
        echo "" >> docs-summary.md
        echo "- [README.md](../README.md)" >> docs-summary.md
        echo "- [API Documentation](../docs/)" >> docs-summary.md
        echo "- [Contributing Guide](../CONTRIBUTING.md)" >> docs-summary.md
        cat docs-summary.md
    - name: Upload documentation summary
      uses: actions/upload-artifact@v3
      with:
        name: docs-summary-${{ github.run_number }}
        path: docs-summary.md
@@ -1,293 +1,109 @@
-name: Security & Dependency Updates
+name: Security
 # Two gates that can actually fail, plus one report:
 #   - pip-audit against requirements.txt: any known vulnerability in a runtime
 #     dependency fails the job (dev-only tools are reported, not gated)
 #   - bandit on src/: HIGH severity findings fail; medium/low are listed.
 #     B104 (bind 0.0.0.0) is skipped: the service is meant to listen on all
 #     interfaces behind Cloudflare/Caddy.
 #   - pip-licenses report as an artifact (informational; the project is MIT
 #     and its runtime deps are MIT/BSD/Apache/PSF)
 # The old file ran safety/bandit/semgrep with `|| true` and could not go red.
 on:
  schedule:
-    # Run security scans daily at 3 AM UTC
+    - cron: "0 3 * * 1" # weekly, Monday 03:00 UTC
    - cron: "0 3 * * *"
  workflow_dispatch:
  push:
    paths:
      - "requirements*.txt"
-      - "Dockerfile"
+      - "pyproject.toml"
      - "uv.lock"
      - "src/**/*.py"
      - ".gitea/workflows/security.yml"
  pull_request:
    paths:
      - "requirements*.txt"
      - "pyproject.toml"
      - "src/**/*.py"
 env:
  PYTHON_VERSION: "3.11"
  # GitHub token for better rate limits and authentication
  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 jobs:
-  # Dependency vulnerability scan
+  dependencies:
-  dependency-scan:
+    name: Dependency vulnerabilities
    name: Dependency Security Scan
    runs-on: ubuntu-latest
    steps:
-      - name: Checkout code
+      - uses: actions/checkout@v4
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
-      - name: Set up Python
+      - uses: actions/setup-python@v5
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install dependencies
+      - name: Install pip-audit
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore safety bandit semgrep
+          pip install --root-user-action=ignore pip-audit
-      - name: Run Safety check
+      - name: Runtime dependencies (gate)
-        run: |
+        run: pip-audit -r requirements.txt --strict --desc on
          safety check -r requirements.txt --json --output safety-report.json || true
          safety check -r requirements-dev.txt --json --output safety-dev-report.json || true
-      - name: Run Bandit security scan
+      - name: Dev dependencies (report only)
-        run: |
+        run: pip-audit -r requirements-dev.txt --desc on || echo "::warning::dev-only dependency advisories above"
          bandit -r src/ -f json -o bandit-report.json || true
-      - name: Run Semgrep security scan
+  code:
-        run: |
+    name: Static analysis
          semgrep --config=auto src/ --json --output=semgrep-report.json || true
      - name: Upload security reports
        uses: actions/upload-artifact@v3
        with:
          name: security-reports-${{ github.run_number }}
          path: |
            safety-report.json
            safety-dev-report.json
            bandit-report.json
            semgrep-report.json
      - name: Check for critical vulnerabilities
        run: |
          echo "Checking for critical vulnerabilities..."
          # Check Safety results
          if [ -f safety-report.json ]; then
            critical_count=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null || echo "0")
            if [ "$critical_count" -gt 0 ]; then
              echo "Found $critical_count dependency vulnerabilities"
              jq '.vulnerabilities[] | "- \(.package_name) \(.installed_version): \(.vulnerability_id)"' safety-report.json
            else
              echo "No dependency vulnerabilities found"
            fi
          fi
          # Check Bandit results
          if [ -f bandit-report.json ]; then
            high_severity=$(jq '.results[] | select(.issue_severity == "HIGH") | length' bandit-report.json 2>/dev/null | wc -l)
            if [ "$high_severity" -gt 0 ]; then
              echo "Found $high_severity high-severity security issues"
            else
              echo "No high-severity security issues found"
            fi
          fi
  # License compliance check
  license-check:
    name: License Compliance
    runs-on: ubuntu-latest
    steps:
-      - name: Checkout code
+      - uses: actions/checkout@v4
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
-      - name: Set up Python
+      - uses: actions/setup-python@v5
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install pip-licenses
+      - name: Install bandit
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-licenses
+          pip install --root-user-action=ignore bandit
          pip install --root-user-action=ignore -r requirements.txt
-      - name: Check licenses
+      - name: bandit (HIGH fails; medium/low listed)
        run: |
-          echo "Checking dependency licenses..."
+          bandit -r src/ -q --skip B104 -ll -ii || true
          bandit -r src/ -q --skip B104 --severity-level high --confidence-level medium
  licenses:
    name: License report
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
          cache-dependency-path: requirements.txt
      # A fresh venv, not the runner's site-packages: the report must list the
      # project's runtime deps, not whatever the runner image or a previous
      # workflow happened to leave installed (semgrep once showed up here).
      - name: Install into a clean venv
        run: |
          python -m venv .lic && . .lic/bin/activate
          pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore -r requirements.txt pip-licenses
      - name: Report
        run: |
          . .lic/bin/activate
          pip-licenses --format=markdown --with-urls --output-file=licenses.md
          pip-licenses --format=json --output-file=licenses.json
-          pip-licenses --format=markdown --output-file=licenses.md
+          echo "Copyleft licenses among runtime deps (informational; LGPL is fine to link from MIT):"
          pip-licenses --format=plain --ignore-packages pip-licenses | grep -iE 'GPL|AGPL|LGPL' || echo "  none"
-          # Check for problematic licenses
+      - uses: actions/upload-artifact@v3
          problematic_licenses=("GPL" "AGPL" "LGPL")
          for license in "${problematic_licenses[@]}"; do
            if grep -i "$license" licenses.json; then
              echo "Found potentially problematic license: $license"
            fi
          done
          echo "License check completed"
      - name: Upload license report
        uses: actions/upload-artifact@v3
        with:
-          name: license-report-${{ github.run_number }}
+          name: licenses-${{ github.run_number }}
          path: |
            licenses.json
            licenses.md
-
+            licenses.json
  # Dependency update check
  dependency-update:
    name: Check for Dependency Updates
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Install pip-check-updates equivalent
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore pip-review
      - name: Check for outdated packages
        run: |
          echo "Checking for outdated packages..."
          pip install --root-user-action=ignore -r requirements.txt
          pip list --outdated --format=json > outdated-packages.json || true
          if [ -s outdated-packages.json ]; then
            echo "Outdated packages found:"
            cat outdated-packages.json | jq -r '.[] | "- \(.name): \(.version) -> \(.latest_version)"'
          else
            echo "All packages are up to date"
          fi
      - name: Upload dependency reports
        uses: actions/upload-artifact@v3
        with:
          name: dependency-reports-${{ github.run_number }}
          path: |
            outdated-packages.json
  # Code quality metrics
  code-quality:
    name: Code Quality Metrics
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Install quality tools
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore radon xenon vulture
          pip install --root-user-action=ignore -r requirements.txt
      - name: Calculate code complexity
        run: |
          echo "Calculating code complexity..."
          radon cc src/ --json > complexity-report.json
          radon mi src/ --json > maintainability-report.json
          echo "Complexity Summary:"
          radon cc src/ --average
          echo "Maintainability Summary:"
          radon mi src/
      - name: Find dead code
        run: |
          echo "Checking for dead code..."
          vulture src/ --json > dead-code-report.json || true
      - name: Check for code smells
        run: |
          echo "Checking for code smells..."
          xenon --max-absolute B --max-modules A --max-average A src/ || true
      - name: Upload quality reports
        uses: actions/upload-artifact@v3
        with:
          name: code-quality-reports-${{ github.run_number }}
          path: |
            complexity-report.json
            maintainability-report.json
            dead-code-report.json
  # Security summary
  security-summary:
    name: Security Summary
    runs-on: ubuntu-latest
    needs: [dependency-scan, license-check, code-quality]
    if: always()
    steps:
      - name: Download all artifacts
        uses: actions/download-artifact@v3
      - name: Generate security summary
        run: |
          echo "# Security Scan Summary" > security-summary.md
          echo "" >> security-summary.md
          echo "**Scan Date:** $(date -u)" >> security-summary.md
          echo "**Repository:** ${{ github.repository }}" >> security-summary.md
          echo "**Commit:** ${{ github.sha }}" >> security-summary.md
          echo "" >> security-summary.md
          echo "## Results" >> security-summary.md
          echo "" >> security-summary.md
          # Dependency scan results
          if [ -f security-reports-*/safety-report.json ]; then
            vuln_count=$(jq '.vulnerabilities | length' security-reports-*/safety-report.json 2>/dev/null || echo "0")
            if [ "$vuln_count" -eq 0 ]; then
              echo "- Dependency Scan: No vulnerabilities found" >> security-summary.md
            else
              echo "- Dependency Scan: $vuln_count vulnerabilities found" >> security-summary.md
            fi
          else
            echo "- Dependency Scan: Results not available" >> security-summary.md
          fi
          # Docker scan results (removed Trivy)
          echo "- Docker Scan: Skipped (Trivy removed)" >> security-summary.md
          # License check results
          if [ -f license-report-*/licenses.json ]; then
            echo "- License Check: Completed" >> security-summary.md
          else
            echo "- License Check: Results not available" >> security-summary.md
          fi
          # Code quality results
          if [ -f code-quality-reports-*/complexity-report.json ]; then
            echo "- Code Quality: Analyzed" >> security-summary.md
          else
            echo "- Code Quality: Results not available" >> security-summary.md
          fi
          echo "" >> security-summary.md
          echo "## Detailed Reports" >> security-summary.md
          echo "" >> security-summary.md
          echo "Detailed reports are available in the workflow artifacts." >> security-summary.md
          cat security-summary.md
      - name: Upload security summary
        uses: actions/upload-artifact@v3
        with:
          name: security-summary-${{ github.run_number }}
          path: security-summary.md
@@ -148,6 +148,9 @@ grafana_data/
 models/*.joblib
 models/cache/
 models/metrics.json
 # scripts/retrain.sh working dirs (staging + one rollback generation)
 models/.staging/
 models/.previous/
 # Playwright MCP browser artifacts (screenshots/snapshots from agent sessions)
 .playwright-mcp/
@@ -19,22 +19,20 @@ repos:
  # Python code formatting with Black
  - repo: https://github.com/psf/black
-    rev: 23.11.0
+    rev: 26.5.1
    hooks:
      - id: black
        language_version: python3
        args: ['--line-length=120']
  # Import sorting with isort
  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
        args: ['--profile', 'black', '--line-length', '120']
  # Linting with flake8
  - repo: https://github.com/pycqa/flake8
    rev: 6.1.0
    hooks:
      - id: flake8
-        args: ['--max-line-length=120', '--extend-ignore=E203,W503']
+        args: ['--max-line-length=100', '--extend-ignore=E203,W503']
@@ -0,0 +1,40 @@
 # CLAUDE.md
 Guidance for AI coding agents working in this repository.
 ## What this is
 Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
 API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
 Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
 `water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
 ## Rules
 - Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
 - `make format` (black 88 / isort black profile, config in pyproject.toml) before
  committing; CI fails on formatting. `make test` must stay green — tests are
  synthetic-data only, never add one that needs the DB or network.
 - Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
  parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
 - Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
  and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive
  or negative, in `docs/FLOOD_FORECASTING.md` section 5. Do not change what is deployed
  (`rise_rain` / hgb-v3) without a harness result that beats it on lead.
 - `train_all()` must never silently produce a gauge-only (v2) model; the guard that
  raises `RainUnavailableError` stays.
 - No `git add -A`: zero-byte shell-accident files (`#`, `$(wc`, ...) have been committed
  before. Stage files by name.
 - Do not add Co-Authored-By trailers.
 - The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
  table: every user-visible string needs both languages.
 ## Where things are
 - `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
  `src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
  `rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
 - `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
  promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
 - `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
  the source catalog.
@@ -1,492 +1,157 @@
-# Northern Thailand Ping River Monitor 🏔️
+# Northern Thailand Ping River Monitor
-A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.
+Live water levels, discharge, rainfall and machine-learning flood forecasts for the
 Ping River basin around Chiang Mai. Collects hourly gauge data from public sources,
 keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST
 API, and 6/12/24-hour flood-risk forecasts per gauge.
-[![CI/CD](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Documentation](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Python](https://img.shields.io/badge/Python-3.9+-blue.svg)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-0.104+-green.svg)](https://fastapi.tiangolo.com) [![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Version](https://img.shields.io/badge/Version-v3.1.3-blue.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/releases)
+**Live: [water.buildfor.life](https://water.buildfor.life/)** · API reference at
 [/docs](https://water.buildfor.life/docs) · built by [buildfor.life](https://buildfor.life)
 after the [October 2024 flood](https://buildfor.life/blog/chiang-mai-flood-2024/) —
 background in [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
-## 🌟 Features
+[![CI](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Docs](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Python 3.11](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
-### 📊 **Real-time Data Collection**
+## What it does
 - **16 Monitoring Stations** across Thailand
 - **15-minute Collection Frequency** with intelligent scheduling
 - **Automatic Gap Filling** for missing historical data
 - **Data Validation** and error recovery mechanisms
 - **Rate Limiting** to prevent API abuse
-### 🌐 **Web API Interface (NEW!)**
+- **Collects** hourly water level and discharge from 16 Royal Irrigation Department
- **FastAPI-powered REST API** with interactive documentation
+  (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly
- **Station Management** - Add, update, and remove monitoring stations
+  rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment
- **Real-time health monitoring** and system status
+  rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and
- **Manual data collection triggers** via web interface
+  its quirks: [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md).
- **Comprehensive metrics** and performance monitoring
+- **Fills gaps.** The raw RID grid had readings for ~56 % of hours; a full-history
- **CORS support** for web applications
+  re-fetch plus HII cross-fill brought it to ~93 %. `GET /api/stats` reports the
  current figure.
 - **Forecasts.** Per gauge and horizon, a gradient-boosted model predicts the rise
  within 6/12/24 h and the probability of crossing the station's warning and danger
  levels. Trained on the monitor's own history plus catchment rain; evaluated
  rolling-origin, event by event. On the October 2024 record flood, trained only on
  data through August 2024, the first alert came **13 hours before** P.1 crossed
  3.70 m. Everything about the model, including what did not work:
  [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md).
 - **Shows it.** A Leaflet map with the river drawn as OSM geometry and styled by live
  discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the
  forecast card, a replay of the 2024 flood, English/Thai, light/dark.
 - **Notifies.** Public push alerts over a self-hosted [ntfy](https://ntfy.sh): one
  message when a gauge crosses its warning or danger level, one all-clear on the
  way down, an opt-in early-warning topic from the model, nothing in between.
  Subscribe from the free app, no account. Matrix room alerts for a team are
  also supported.
-### 🗄️ **Multi-Database Support**
+## Quick start
 - **VictoriaMetrics** (Recommended) - High-performance time-series
 - **InfluxDB** - Purpose-built time-series database
 - **PostgreSQL + TimescaleDB** - Relational with time-series optimization
 - **MySQL** - Traditional relational database
 - **SQLite** - Local development and testing
-### 🗺️ **Geolocation Support**
+Python **3.11** (3.13 breaks the pinned `psycopg2-binary`), PostgreSQL for anything
- **Grafana Geomap** integration ready
+beyond a quick look, [uv](https://docs.astral.sh/uv/).
 - **GPS coordinates** and geohash support
 - **Interactive mapping** of water stations
 ### 📈 **Visualization & Monitoring**
 - **Pre-built Grafana dashboards**
 - **Real-time alerts** and notifications
 - **Historical trend analysis**
 - **Built-in metrics collection** (counters, gauges, histograms)
 - **Health checks** for database, API, and system resources
 ### 🚀 **Production Ready**
 - **Docker containerization** with multi-service support
 - **Systemd service** configuration
 - **HTTPS support** with SSL certificates
 - **Comprehensive logging** with rotation and colored output
 - **Type safety** with Pydantic models and type hints
 - **Custom exception handling** for better error management
 ## 🚀 Quick Start
 ### Prerequisites
 - Python 3.9 or higher
 - Internet connection for data fetching
 - Database server (optional - SQLite works out of the box)
 ### Installation
 ```bash
 # Clone the repository
 git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 cd Northern-Thailand-Ping-River-Monitor
-
+uv sync --python 3.11
-# Quick setup with Make
+cp .env.example .env            # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
-make dev-setup
+uv run python run.py --web-api  # dashboard + API on http://localhost:8000
 # Or manual setup:
 python -m venv venv
 source venv/bin/activate  # Windows: venv\Scripts\activate
 pip install -r requirements.txt
 cp .env.example .env
 ```
-### Basic Usage
+`DB_TYPE=sqlite` works for the dashboard and API; the forecasting path expects the
 PostgreSQL history.
 ```bash
-# Test run with SQLite (default)
+uv run python run.py --status          # collector status
-make run-test
+uv run python run.py --test            # one collection cycle
-# or: python run.py --test
+uv run python run.py --fill-gaps 7     # re-fetch the last 7 days from RID
-
+uv run python run.py --collect-hii     # one ThaiWater/HII collection cycle
-# Run continuous monitoring
+uv run python run.py --alert-check     # evaluate thresholds, notify Matrix
-make run
+uv run python scripts/train_flood_model.py --stations all   # retrain (~12 min)
-# or: python run.py
+make test                              # pytest, synthetic data, no network
-
+make format                            # black + isort (the CI contract)
 # Start web API server (NEW!)
 make run-api
 # or: python run.py --web-api
 # Run all tests
 make test
 # Demo different databases
 python src/demo_databases.py
 ```
-### 🌐 Web API Interface (NEW!)
+## API
-The system now includes a comprehensive FastAPI web interface:
+Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
 Asia/Bangkok wall-clock without an offset suffix.
 | Endpoint | Returns |
 | --- | --- |
 | `GET /stations` | The 16 RID gauges: code, Thai/English names, coordinates |
 | `GET /measurements/latest?limit=N` | Newest reading per station |
 | `GET /measurements/history/{code}?hours=N` | Hourly history; or `?start=YYYY-MM-DD&end=YYYY-MM-DD`; `limit` ≤ 100000 |
 | `GET /forecast` | Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
 | `GET /api/forecast/history/{code}?hours=N&horizon=24` | Forecasts as issued, for auditing lead time after the fact |
 | `GET /api/hii/rainfall/latest`, `/api/hii/waterlevel/latest` | Latest ThaiWater/HII gauge readings |
 | `GET /api/hii/rainfall/catchment?days=N` | HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
 | `GET /api/forecast/skill?station_code=P.1` | Issued forecasts vs what happened, per deployed model version |
 | `GET /api/notifications` | ntfy server and topic names for the subscribe panel |
 | `GET /api/stats` | Row counts per source, date range, coverage |
 | `GET /health` | DB / upstream / memory checks |
 Interactive reference with schemas: [water.buildfor.life/docs](https://water.buildfor.life/docs).
 Responses are cached briefly server-side; poll no faster than once a minute — the data
 changes hourly.
 ## Deployment
 Production is a systemd unit on a small VPS behind Cloudflare, updated by `git pull`.
 `scripts/install.sh` (run as root from a checkout) creates the `water-monitor` user,
 deploys to `/opt/thailand-water-monitor`, runs `uv sync` into `.venv`, installs
 `water-monitor.service` and the monthly `water-monitor-retrain.timer`.
 ```bash
 # Start the web API
 python run.py --web-api
 # Access the API at:
 # - Dashboard: http://localhost:8000
 # - Interactive docs: http://localhost:8000/docs
 # - Health check: http://localhost:8000/health
 # - Latest data: http://localhost:8000/measurements/latest
 ```
 **Key API Endpoints:**
 - `GET /` - Web dashboard
 - `GET /health` - System health status
 - `GET /metrics` - Application metrics
 - `GET /stations` - List all monitoring stations
 - `POST /stations` - Add new monitoring station
 - `PUT /stations/{id}` - Update station information
 - `DELETE /stations/{id}` - Remove monitoring station
 - `GET /measurements/latest` - Latest measurements
 - `GET /measurements/station/{code}` - Station-specific data
 - `POST /scrape/trigger` - Trigger manual data collection
 ## 📊 Station Information
 The system monitors **16 water stations** along the Ping River Basin in Northern Thailand:
 | Station | Thai Name | English Name | Location |
 |---------|-----------|--------------|----------|
 | P.1 | สะพานนวรัฐ | Nawarat Bridge | Nakhon Sawan |
 | P.5 | สะพานท่านาง | Tha Nang Bridge | - |
 | P.20 | บ้านเชียงดาว | Ban Chiang Dao | Chiang Mai |
 | P.21 | บ้านริมใต้ | Ban Rim Tai | - |
 | P.4A | บ้านแม่แตง | Ban Mae Taeng | Chiang Mai |
 | P.67 | บ้านแม่แต | Ban Tae | - |
 | P.75 | บ้านช่อแล | Ban Chai Lat | - |
 | P.76 | บ้านแม่อีไฮ | Banb Mae I Hai | - |
 | P.77 | บ้านสบแม่สะป๊วด | Baan Sop Mae Sapuord | - |
 | P.81 | บ้านโป่ง | Ban Pong | - |
 | P.82 | บ้านสบวิน | Ban Sob win | - |
 | P.84 | บ้านพันตน | Ban Panton | - |
 | P.85 | บ้านหล่ายแก้ว | Baan Lai Kaew | - |
 | P.87 | บ้านป่าซาง | Ban Pa Sang | - |
 | P.92 | บ้านเมืองกึ๊ด | Ban Muang Aut | - |
 | P.103 | สะพานวงแหวนรอบ 3 | Ring Bridge 3 | Bangkok |
 ### Data Metrics
 - **Water Level**: Measured in meters (m)
 - **Discharge**: Flow rate in cubic meters per second (cms)
 - **Discharge Percentage**: Relative to station capacity
 - **Timestamp**: Thai time (UTC+7) with Buddhist calendar support
 ## 🗄️ Database Configuration
 ### VictoriaMetrics (Recommended)
 **High-performance time-series database with excellent compression and query speed.**
 ```bash
 # Environment variables
 export DB_TYPE=victoriametrics
 export VM_HOST=localhost
 export VM_PORT=8428
 # Quick start with Docker
 docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428
 ```
 ### Complete Stack with Grafana
 ```bash
 # Start the complete monitoring stack
 docker-compose -f docker-compose.victoriametrics.yml up -d
 # Access Grafana at http://localhost:3000
 # Username: admin, Password: admin_password
 ```
 ### Other Database Options
 <details>
 <summary>InfluxDB Configuration</summary>
 ```bash
 export DB_TYPE=influxdb
 export INFLUX_HOST=localhost
 export INFLUX_PORT=8086
 export INFLUX_DATABASE=water_monitoring
 export INFLUX_USERNAME=water_user
 export INFLUX_PASSWORD=your_password
 ```
 </details>
 <details>
 <summary>PostgreSQL Configuration</summary>
 ```bash
 export DB_TYPE=postgresql
 export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring
 ```
 </details>
 <details>
 <summary>MySQL Configuration</summary>
 ```bash
 export DB_TYPE=mysql
 export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring
 ```
 </details>
 ## 📈 Grafana Dashboards
 ### Pre-built Dashboard Features
 - **Real-time water levels** across all stations
 - **Historical trends** and patterns
 - **Discharge monitoring** with percentage indicators
 - **Station status** and health monitoring
 - **Geomap visualization** of station locations
 - **Alert thresholds** for critical water levels
 ### Sample Queries
 **VictoriaMetrics/Prometheus:**
 ```promql
 # Current water levels
 water_level
 # High discharge alerts
 water_discharge_percent > 80
 # Station-specific data
 water_level{station_code="P.1"}
 ```
 **SQL Databases:**
 ```sql
 -- Latest readings from all stations
 SELECT s.station_code, s.english_name, m.water_level, m.discharge
 FROM stations s
 JOIN water_measurements m ON s.id = m.station_id
 WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);
 ```
 ## 🚀 Production Deployment
 ### Docker Deployment
 ```bash
 # Build the image
 docker build -t thailand-water-monitor .
 # Run with environment variables
 docker run -d \
  --name water-monitor \
  -e DB_TYPE=victoriametrics \
  -e VM_HOST=victoriametrics \
  thailand-water-monitor
 ```
 ### Systemd Service (Linux)
 The install script sets everything up: a dedicated `water-monitor` system user,
 a deploy to `/opt/thailand-water-monitor`, a uv-managed virtualenv, and the
 enabled systemd unit.
 ```bash
 # From a checkout of the repo, as root:
 sudo bash scripts/install.sh
 # Then start and check:
 sudo systemctl start water-monitor.service
-systemctl status water-monitor.service
+systemctl list-timers water-monitor-retrain.timer
 ```
-Fill in `/opt/thailand-water-monitor/.env` (Matrix token/room, DB settings)
+The retrain timer runs `scripts/retrain.sh`, which trains into `models/.staging`,
-before starting if the script reports it is missing.
+refuses to promote anything that is not a rain-enabled (`hgb-v3+`) set covering the
 expected stations, and renames the bundles into place. Details and the operations
 runbook: [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) sections 6–8.
-<details>
+## Repository layout
 <summary>Manual setup (if you prefer not to use the script)</summary>
 ```bash
 sudo useradd --system --no-create-home --shell /usr/sbin/nologin water-monitor
 sudo cp scripts/water-monitor.service /etc/systemd/system/
 sudo systemctl enable water-monitor.service
 sudo systemctl start water-monitor.service
 ```
 </details>
 ## 🔧 Command Line Tools
 ### Main Application
 ```bash
 python src/water_scraper_v3.py                    # Run continuous monitoring
 python src/water_scraper_v3.py --test             # Single test cycle
 python src/water_scraper_v3.py --help             # Show help
 ```
 ### Data Management
 ```bash
 python src/water_scraper_v3.py --check-gaps 7     # Check for missing data (7 days)
 python src/water_scraper_v3.py --fill-gaps 7      # Fill missing data gaps
 python src/water_scraper_v3.py --update-data 2    # Update existing data (2 days)
 ```
 ### Database Testing
 ```bash
 python src/demo_databases.py                      # SQLite demo
 python src/demo_databases.py victoriametrics      # VictoriaMetrics demo
 python src/demo_databases.py all                  # Test all databases
 ```
 ## 📚 Documentation
 ### Core Documentation
 - **[Data Sources & API Catalog](docs/DATA_SOURCES.md)** - Every ingested and available data source (RID, ThaiWater/HII, dams, rainfall, forecasts)
 - **[Installation Guide](docs/DATABASE_DEPLOYMENT_GUIDE.md)** - Complete setup instructions
 - **[Gap Filling Guide](docs/GAP_FILLING_GUIDE.md)** - Data integrity management
 ### Deployment Guides
 - **[VictoriaMetrics Setup](docs/VICTORIAMETRICS_SETUP.md)** - High-performance deployment
 - **[Debian Troubleshooting](docs/DEBIAN_TROUBLESHOOTING.md)** - Linux deployment issues
 ### References
 - **[Notable Documents](docs/references/NOTABLE_DOCUMENTS.md)** - Official Thai government resources
 ## 🔍 Troubleshooting
 ### Common Issues
 **Database Connection Errors:**
 ```bash
 # Check database status
 python src/demo_databases.py
 # Test specific database
 python src/demo_databases.py victoriametrics
 ```
 **Missing Data:**
 ```bash
 # Check for gaps
 python src/water_scraper_v3.py --check-gaps 7
 # Fill missing data
 python src/water_scraper_v3.py --fill-gaps 7
 ```
 **Service Issues:**
 ```bash
 # Check service status
 sudo systemctl status water-monitor
 # View logs
 sudo journalctl -u water-monitor -f
 ```
 ### Health Checks
 ```bash
 # VictoriaMetrics health
 curl http://localhost:8428/health
 # Check latest data
 curl "http://localhost:8428/api/v1/query?query=water_level"
 # Application logs
 tail -f water_monitor.log
 ```
 ## 🌐 API Integration
 ### VictoriaMetrics API Examples
 ```bash
 # Query current water levels
 curl "http://localhost:8428/api/v1/query?query=water_level"
 # Query discharge rates for last hour
 curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"
 # Query specific station
 curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
 # High discharge alerts
 curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
 ```
 ## 📊 Performance
 ### System Requirements
 - **CPU**: 1-2 cores (minimal load)
 - **RAM**: 512MB - 2GB (depending on database)
 - **Storage**: 1GB+ (for historical data)
 - **Network**: Stable internet connection
 ### Performance Metrics
 - **Data Collection**: ~300 data points every 15 minutes
 - **Database Write Speed**: 1000+ points/second (VictoriaMetrics)
 - **Query Response**: <100ms for recent data
 - **Storage Efficiency**: 70x compression vs. raw data
 ## 🤝 Contributing
 Contributions are welcome! Please:
 1. Fork the repository
 2. Create a feature branch
 3. Make your changes
 4. Add tests if applicable
 5. Submit a pull request
 ### Development Setup
 ```bash
 # Clone your fork
 git clone https://github.com/your-username/thailand-water-monitor.git
 cd thailand-water-monitor
 # Install development dependencies
 pip install -r requirements.txt
 pip install pytest black flake8
 # Run tests
 pytest
 # Format code
 black src/
 ```
 ## 📄 License
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
 ## 🙏 Acknowledgments
 - **Royal Irrigation Department (RID)** of Thailand for providing the data API
 - **VictoriaMetrics** team for the excellent time-series database
 - **Grafana** team for the visualization platform
 - **Python community** for the amazing libraries and tools
 ## 📞 Support
 - **Issues**: [GitHub Issues](https://github.com/your-username/thailand-water-monitor/issues)
 - **Discussions**: [GitHub Discussions](https://github.com/your-username/thailand-water-monitor/discussions)
 - **Documentation**: [Project Wiki](https://github.com/your-username/thailand-water-monitor/wiki)
 ---
 ## 📁 Project Structure
 ```
-Northern-Thailand-Ping-River-Monitor/
+src/                 collector, API (web_api.py), dashboard (static/dashboard.html)
-├── src/                    # Main application code
+src/ml/              features, training, evaluation harness, prediction, rain/dam/HII loaders
-├── tests/                  # Test suite
+scripts/             train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
-├── docs/                   # Documentation
+tests/               pytest suite (synthetic data; no DB or network)
-├── grafana/               # Grafana dashboards
+docs/                FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
-├── scripts/               # Utility scripts
+models/              trained bundles + metrics.json (gitignored) and evaluation results (tracked)
-├── docker-compose.yml     # Docker deployment
+.gitea/workflows/    ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
 ├── Makefile              # Development tasks
 └── requirements.txt      # Dependencies
 ```
-See [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) for the forecasting architecture and [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) for the data pipeline.
+## Documentation
-## 🔄 CI/CD & Automation
+- [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
 - [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) — every ingested and candidate source, endpoints, quirks
 - [docs/STATION_MANAGEMENT_GUIDE.md](docs/STATION_MANAGEMENT_GUIDE.md) — adding/editing gauges
 - [docs/DATABASE_DEPLOYMENT_GUIDE.md](docs/DATABASE_DEPLOYMENT_GUIDE.md), [POSTGRESQL_SETUP.md](POSTGRESQL_SETUP.md) — database setup
 - [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md) — public push alerts: topics, semantics, ntfy deployment
 - [docs/MATRIX_QUICK_START.md](docs/MATRIX_QUICK_START.md) — Matrix room alerts for a team
 - [docs/GAP_FILLING_GUIDE.md](docs/GAP_FILLING_GUIDE.md) — data integrity tooling
 - [docs/references/NOTABLE_DOCUMENTS.md](docs/references/NOTABLE_DOCUMENTS.md) — official Thai government resources
 - Public overview: [buildfor.life/docs/tooling/ping-river-monitor](https://buildfor.life/docs/tooling/ping-river-monitor/)
-The project includes comprehensive Gitea Actions workflows:
+Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
 dashboards under `grafana/` are supported by the adapters but not what production
 runs; see [docs/VICTORIAMETRICS_SETUP.md](docs/VICTORIAMETRICS_SETUP.md) if you want them.
- **🧪 CI/CD Pipeline** - Automated testing, building, and deployment
+## Contributing
 - **🔒 Security Scanning** - Daily vulnerability and dependency checks
 - **📚 Documentation** - Automated API docs and validation
 - **🚀 Release Management** - Automated releases with multi-arch Docker builds
-See [docs/GITEA_WORKFLOWS.md](docs/GITEA_WORKFLOWS.md) for detailed workflow documentation.
+`make format` before committing (black 88 columns, isort black profile — the CI gate),
 `make test` must stay green, tests use synthetic data only. See
 [CONTRIBUTING.md](CONTRIBUTING.md). Issues and merge requests on
 [git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
-## 🔗 Repository
+## Data sources and thanks
- **Main Repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
+Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) /
- **Issues**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
+ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry;
- **Actions**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
+Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All
- **Documentation**: [docs/](docs/)
+instruments are theirs; we aggregate, store, fill gaps and forecast.
-**Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin**
+## License
 MIT — see [LICENSE](LICENSE).
@@ -68,7 +68,7 @@ environment variable, then `Config.get_database_config()` when `DB_TYPE` is
 1. **PostgreSQL** (`_fetch_from_db`) — the primary path. NULL discharge stays
   NULL, which matters because the models must learn from the real missingness
   pattern.
-2. **HTTP API** (`_fetch_from_api`, default `http://100.81.167.42:8000`) — a
+2. **HTTP API** (`_fetch_from_api`, default `https://water.buildfor.life`) — a
   fallback for running off-server. **Caveat:** the public history endpoint
   backfills missing discharge with a synthetic rating-curve estimate, so this
   path is not equivalent to the DB path. It is flagged as
@@ -536,6 +536,54 @@ exists alongside its flood events.
 (`fill_from_hii`, +9,341 h at P.81, +682 h at P.92, +810 h at P.20) is
 lead-neutral — the gate holds at 13 h with fill on — and ships enabled.
 ### 2026-09-12: three candidates on top of hgb-v3 — two rejected, one deferred
 Same rolling-origin harness (`src/ml/evaluate.py`, five monsoon folds
 2021–2025, P.1 and P.103), all variants run from the identical
 `models/cache/` snapshot (`--from-cache`), results in
 `models/eval_2026-09-12*.json`, tables via `scripts/summarize_eval.py`.
 Baseline is `rise_rain`, the deployed configuration.
 **Quantile regression heads (`rise_rain_quantile`, `_uw`) — rejected.** The
 August result that quantile loss beat L2 on MAE held with rain in the model
 (P.1 0.083/0.081 vs 0.087; P.103 0.143/0.124 vs 0.152), and Brier improved a
 hair, but the operational numbers went the wrong way: at P.103 the 2022-08-14
 crossing dropped from +6 h to +1 h lead, 2022-10-02 from +9 h to +5/+3 h, and
 the 2024-09-30 event from +9 h to +4 h; at P.1 2022 dropped +5 → +3/+2 h and
 2025 +2 → +1 h, with one false-alarm episode where the baseline had none. A
 median predicts the *typical* rise, and on the run-up to a crossing the typical
 rise is not the one that matters. MAE is not the objective; lead is.
 **Quantile heads for sigma only (`rise_rain_qsigma`) — no effect.** The
 hybrid keeps the L2 point prediction (so every lead is identical to the
 baseline by construction — p≥0.5 alerts are sigma-independent) and derives a
 per-row sigma from q90−q50. Brier moved 0.0031 → 0.0029 at P.1 and
 0.0061 → 0.0060 at P.103, i.e. within noise, at the cost of three fitted
 heads per horizon instead of one. Per-row uncertainty from this family of
 models is not informative enough here to be worth the training time; the
 0.15 m floor stays.
 **Forward-48 h forecast rain (`rise_rain_fc48`) — deferred.** Adding the
 `(t, t+48]` Open-Meteo sum alongside `rain_fc24` left MAE, Brier and false
 alarms unchanged and every event lead within ±1 h of baseline, *except* the
 2024-10-03 P.1 record crossing, which went from +21 h to +72 h (and +55 → +69 h
 at P.103). That is one event with the highest stakes in the record, on the
 same feature family that already produced the 2024 gain, but n=1 is not
 evidence: the P.103 2025-09-26 event lost 2 h in the same run. Rerun after the
 2026 season adds events; if the 48 h window still moves only the biggest
 onsets, promote it. Serving would need no new data source (`fetch_forecast`
 already pulls `forecast_days=2`).
 **HII gauge rain — not evaluable yet.** `hii_rainfall` (~130 gauges in the
 upper-Ping box, DWR/FOP/HII/RID/TMD) is the obvious independent rain source,
 but the table only exists since 2026-08-11 and the api-v3 archive endpoint
 ignores its date range (see `docs/DATA_SOURCES.md` §2.1), so every training
 row before that is NaN and no fold in the harness has gauge data in its test
 span. `src/ml/hii_rain.py` builds the catchment mean and
 `GET /api/hii/rainfall/catchment` exposes it next to the Open-Meteo series with
 a 24 h-sum bias/MAE/correlation, so the two sources' relationship is on record
 by the time the 2027 fold (train ≤ 2027-04-30, test Jun–Nov 2027) can test it.
 ## 6. Deployment
 ### API
@@ -733,6 +781,32 @@ timestamp in every bundle. Both are echoed in every `/forecast` row, so you can
 tell from the API response alone which code produced a forecast and how old the
 model is.
 **Scheduled retrain (since 2026-09-12).** `scripts/water-monitor-retrain.timer`
 fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
 time (`Persistent=true`, so a missed run catches up at boot). The unit runs
 `scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
 1. trains all stations into `models/.staging/` (the API keeps serving the old
   bundles throughout);
 2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
   least 14 trained stations (exit 3, staging discarded, old models untouched);
 3. renames the new bundles into `models/`, moving the previous generation to
   `models/.previous/` for rollback.
 No API restart: `predict.py` reloads bundles by mtime on the next hourly
 precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
 run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
 flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
 (`scripts/install.sh`) enables the timer.
 **Why the trainer refuses to run without rain (since 2026-09-12).** On
 2026-09-01 the server retrain could not reach the Open-Meteo archive on a
 checkout with no `models/cache/`, logged a warning, and quietly overwrote the
 v3 bundles with gauge-only v2 ones — the 13-hour early warning on the 2024 flood
 became an 18-hour late one and nothing on the dashboard said so. `train_all()`
 now raises `RainUnavailableError` (CLI exit 2) in that situation. Gauge-only
 bundles are still available, but only by asking for them: `--no-rain`.
 ## 8. Operations runbook
 All commands assume the project virtualenv is active (`.venv` locally).
@@ -786,10 +860,12 @@ that went quiet, or a bad backfill) rather than a modelling one.
 python -m pytest tests/test_flood_forecast.py -v
 ```
-Seven tests covering leakage, label alignment, the coverage gate, forward-fill and
+Tests cover leakage, label alignment, the coverage gate, forward-fill and
-staleness, a train/predict round trip, the heuristic fallback, and feature-name
+staleness, a train/predict round trip, the heuristic fallback, feature-name
-stability. The whole suite runs in about 8 seconds, so there is no excuse for
+stability, and the rain-downgrade guard (no rain series → `RainUnavailableError`,
-skipping it before a deploy.
+nothing written; `--no-rain` → v2; rain present → v3 with the rain columns in
 `feature_names`). The file runs in well under a minute, so there is no excuse
 for skipping it before a deploy.
 **Understanding graceful degradation.** Three things can make a forecast row
 non-model-backed, and all of them are visible in the payload:
@@ -0,0 +1,132 @@
 # Flood notifications (ntfy)
 Public push notifications for threshold crossings, without accounts, mailing
 lists or app-store review: the monitor publishes to a self-hosted
 [ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
 about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
 ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
 runs on the same VPS as the monitor.
 ## What subscribers get
 Every message is a **transition**, never a state. Crossing up into a level sends
 one message; dropping back below it (with 0.10 m hysteresis) sends one
 all-clear. A river that sits at 3.9 m for three days produces two messages, not
 seventy-two. In a quiet season a subscriber hears nothing.
 | Topic | Trigger | Priority |
 |---|---|---|
 | `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
 | `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
 | `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
 | `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
 | `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
 | `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
 Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
 Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
 4.20 m, P.103 5.95 / 6.75 m, and so on.
 The outlook topic is opt-in for a reason: it is model output, and the message
 says so. Observed-crossing topics only ever report a gauge reading.
 Each message carries a click-through and an "Open dashboard" action button to
 the public dashboard.
 ## How it runs
 `src/notify.py` is called once per collection cycle inside the API process
 (leader only), right after the forecast precompute, so it sees exactly the
 readings and forecasts the dashboard shows. Per-key last-sent state is stored
 in the `notification_state` table of the monitor's own database, so a restart
 or redeploy never re-sends and never misses a crossing that happened while
 the service was down (the next cycle compares against the persisted state).
 If ntfy is unreachable the transition is **not** recorded, so it is retried
 on the next cycle rather than silently lost. Any other failure in the notify
 step is logged and never reaches the collection loop.
 The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
 set; it reads `GET /api/notifications` and renders subscribe links
 (`ntfy://` deep links for the app, https links for the web UI).
 ## Deployment
 On the monitor VPS, as root:
 ```bash
 cd /opt/thailand-water-monitor
 NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
 ```
 This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
 host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
 message cache, signup/login/metrics off, tight visitor limits), enables the
 systemd unit,
 creates the `monitor` user with **write-only access to `ping-*`**, mints a
 token, and appends `NTFY_SERVER` (public URL for subscribers),
 `NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
 and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
 ```bash
 systemctl restart water-monitor
 journalctl -u water-monitor -n 20 | grep ntfy      # "ntfy notifications: https://... topics ping-*"
 curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1'   # anonymous read works
 ```
 The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
 the monitor host's Tailscale address and nothing is exposed on a public
 interface. On the Caddy machine:
 ```caddyfile
 ntfy.buildfor.life {
    reverse_proxy <monitor tailscale ip>:2586
 }
 ```
 Caddy proxies websockets and keeps long-poll connections open by default;
 subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
 `X-Forwarded-For` rather than treating every subscriber as the proxy.
 Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
 monitor at the Tailscale address directly, so a DNS or proxy problem never
 holds back an alert. Test the pipeline before the domain is live with
 `curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
 ## Configuration
 | Variable | Default | Meaning |
 |---|---|---|
 | `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
 | `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
 | `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
 | `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
 | `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
 Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
 (0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
 ## Testing
 `tests/test_notify.py` covers the state machine: quiet river sends nothing;
 crossing once, then silence while above, then all-clear; hysteresis on the way
 down; escalation to danger and back; basin digest grouping; outlook on/off;
 heuristic forecasts ignored; stale feed and recovery; state survives a restart
 through sqlite; a failed publish is retried next cycle.
 To exercise the real path against a real ntfy locally: run `ntfy serve` (any
 platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
 poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
 template.
 ## Why ntfy and not …
 - **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
  subscriber and a room invite; fine for a team, wrong for the public.
 - **Gotify**: also self-hosted and light, but Android-only client and one
  account per subscriber.
 - **Email / SMS**: deliverability work, cost per message, no priority
  semantics; ntfy can forward to email per subscription if someone wants it.
 - **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
  added later as ntfy→webhook fan-out without touching the monitor.
@@ -0,0 +1,723 @@
 [
 {
  "station": "P.1",
  "warn_thr": 3.7,
  "folds": [
   {
    "year": 2021,
    "n_train": 20024,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07375926701460789,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.07332598842242062,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.07383022350644125,
      "mae_above_2p5": null,
      "brier_warn": 1.0850721383440065e-12,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.06977345537342049,
      "mae_above_2p5": null,
      "brier_warn": 7.128994064266462e-16,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28784,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-10-02T19:00:00",
      "peak_ts": "2022-10-03T15:00:00",
      "peak_level": 4.65
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.08452847354970178,
      "mae_above_2p5": 0.25082252888260664,
      "brier_warn": 0.004300908725927739,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8173954245046406
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.08369773912557228,
      "mae_above_2p5": 0.24702357745371217,
      "brier_warn": 0.004325741658698973,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8114190118860223
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.08230165237819607,
      "mae_above_2p5": 0.2599241552494541,
      "brier_warn": 0.004191550822623699,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 3.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.693734826616603
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.08181109208140971,
      "mae_above_2p5": 0.2639358812546371,
      "brier_warn": 0.004581181754314636,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 2.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.620470606696475
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37539,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07696637416933236,
      "mae_above_2p5": 0.10643655855133666,
      "brier_warn": 1.919860722404625e-15,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.07708101918232377,
      "mae_above_2p5": 0.09974894450126857,
      "brier_warn": 1.7028444765622288e-15,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.06968496247786034,
      "mae_above_2p5": 0.10210094973854984,
      "brier_warn": 1.0987601008721297e-09,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.06726603345661021,
      "mae_above_2p5": 0.09716644572204487,
      "brier_warn": 2.7085590803510675e-09,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46323,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2024-09-24T17:00:00",
      "peak_ts": "2024-09-26T02:00:00",
      "peak_level": 4.93
     },
     {
      "crossing": "2024-10-03T09:00:00",
      "peak_ts": "2024-10-05T12:00:00",
      "peak_level": 5.3
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.0890019157543212,
      "mae_above_2p5": 0.2093245927883516,
      "brier_warn": 0.005826169840715695,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 10.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.817519939833057
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.546588884631041
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.08875916089292855,
      "mae_above_2p5": 0.20975114218621593,
      "brier_warn": 0.006061154593275248,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 11.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.800924141216692
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 72.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.577164984770105
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.08341835741803395,
      "mae_above_2p5": 0.18688839374651373,
      "brier_warn": 0.003188229521816099,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 17.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 5.058029430632501
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.3311787370709975
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.08412336465267245,
      "mae_above_2p5": 0.19173218504592401,
      "brier_warn": 0.0034722638888286116,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 15.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 5.0118284217314
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.3520431553190155
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 55083,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-27T18:00:00",
      "peak_ts": "2025-09-27T22:00:00",
      "peak_level": 3.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.11202835461695825,
      "mae_above_2p5": 0.2456782496767171,
      "brier_warn": 0.005178356039745929,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.11083068059655521,
      "mae_above_2p5": 0.23787013498709667,
      "brier_warn": 0.005228043825289021,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.10372526043263834,
      "mae_above_2p5": 0.23846363852091013,
      "brier_warn": 0.005898259026876986,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 1.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_rain_quantile_uw": {
      "mae": 0.10267267834487567,
      "mae_above_2p5": 0.2507951319537275,
      "brier_warn": 0.005020467408392599,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.2417205711942434
       }
      ],
      "false_alarm_episodes": 1
     }
    }
   }
  ]
 },
 {
  "station": "P.103",
  "warn_thr": 5.95,
  "folds": [
   {
    "year": 2021,
    "n_train": 20009,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.14438683486790602,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.14305613024043953,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.13512418754716052,
      "mae_above_2p5": null,
      "brier_warn": 6.937198832251013e-10,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.10270057685061172,
      "mae_above_2p5": null,
      "brier_warn": 6.565945930999852e-14,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28769,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-08-14T04:00:00",
      "peak_ts": "2022-08-14T08:00:00",
      "peak_level": 6.09
     },
     {
      "crossing": "2022-10-02T16:00:00",
      "peak_ts": "2022-10-03T16:00:00",
      "peak_level": 7.54
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.14392334606606189,
      "mae_above_2p5": 0.41075382386412596,
      "brier_warn": 0.00764679988213082,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.825270553204425
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194064117976157
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.14284722696431765,
      "mae_above_2p5": 0.39401132538096667,
      "brier_warn": 0.007427375799450148,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.845896993132048
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194661123502819
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.1458954690776336,
      "mae_above_2p5": 0.4674051188369517,
      "brier_warn": 0.00899991317044724,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 1.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.82714042795344
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 5.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 6.59871451008115
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.13203958422483053,
      "mae_above_2p5": 0.4902773906613983,
      "brier_warn": 0.008514527559601331,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 4.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.895369771408423
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 3.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 6.294082735853337
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37524,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.16195301512514512,
      "mae_above_2p5": null,
      "brier_warn": 3.0208441147560167e-07,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.16334224973870387,
      "mae_above_2p5": null,
      "brier_warn": 1.1626187140381066e-08,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.14755024879522577,
      "mae_above_2p5": null,
      "brier_warn": 3.850104714388072e-05,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.10589805327026759,
      "mae_above_2p5": null,
      "brier_warn": 1.8896757611081001e-07,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46308,
    "n_test": 4058,
    "events": [
     {
      "crossing": "2024-09-24T10:00:00",
      "peak_ts": "2024-09-26T00:00:00",
      "peak_level": 8.27
     },
     {
      "crossing": "2024-09-30T03:00:00",
      "peak_ts": "2024-09-30T06:00:00",
      "peak_level": 5.99
     },
     {
      "crossing": "2024-10-03T06:00:00",
      "peak_ts": "2024-10-05T07:00:00",
      "peak_level": 9.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.13901842325128816,
      "mae_above_2p5": 0.2887416310966684,
      "brier_warn": 0.0072040857753137046,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.212734363860193
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.47565489914091
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.780278464915938
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.13850733053801131,
      "mae_above_2p5": 0.29817392926900865,
      "brier_warn": 0.007928846574280278,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.166020251020889
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 10.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.524584037259288
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.595120917488185
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.12640546558686208,
      "mae_above_2p5": 0.27944115421093924,
      "brier_warn": 0.007730268539879654,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 20.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.336487732683672
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 4.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.45284915024346
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.575915226697406
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.12480975852168202,
      "mae_above_2p5": 0.2857575557415695,
      "brier_warn": 0.005894928998647136,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 21.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.444720326882825
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.42
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 32.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.781001847167515
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 54577,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-26T06:00:00",
      "peak_ts": "2025-09-27T21:00:00",
      "peak_level": 6.64
     },
     {
      "crossing": "2025-10-03T06:00:00",
      "peak_ts": "2025-10-03T12:00:00",
      "peak_level": 6.14
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.17206685418009837,
      "mae_above_2p5": 0.3840520085986099,
      "brier_warn": 0.015824852891785323,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73043665401637
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.609919760117381
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_fc48": {
      "mae": 0.1722274451362234,
      "mae_above_2p5": 0.3872216974630654,
      "brier_warn": 0.016245727130063177,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 4.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.7312263707793685
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.685511824758637
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_quantile": {
      "mae": 0.15912275848686555,
      "mae_above_2p5": 0.3845695612674703,
      "brier_warn": 0.014797606329811499,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 11.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 10.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.58133012298031
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_quantile_uw": {
      "mae": 0.15385355845103685,
      "mae_above_2p5": 0.41314645854725585,
      "brier_warn": 0.016528116336109958,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 9.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.741254234340573
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.418105405306207
       }
      ],
      "false_alarm_episodes": 2
     }
    }
   }
  ]
 }
 ]
@@ -0,0 +1,439 @@
 [
 {
  "station": "P.1",
  "warn_thr": 3.7,
  "folds": [
   {
    "year": 2021,
    "n_train": 20024,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07375926701460789,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.07375926701460789,
      "mae_above_2p5": null,
      "brier_warn": 7.852802408474157e-14,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28784,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-10-02T19:00:00",
      "peak_ts": "2022-10-03T15:00:00",
      "peak_level": 4.65
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.08452847354970178,
      "mae_above_2p5": 0.25082252888260664,
      "brier_warn": 0.004300908725927739,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8173954245046406
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.08452847354970178,
      "mae_above_2p5": 0.25082252888260664,
      "brier_warn": 0.0039815091186836665,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8173954245046406
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37539,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07696637416933236,
      "mae_above_2p5": 0.10643655855133666,
      "brier_warn": 1.919860722404625e-15,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.07696637416933236,
      "mae_above_2p5": 0.10643655855133666,
      "brier_warn": 4.0844243581898366e-08,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46323,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2024-09-24T17:00:00",
      "peak_ts": "2024-09-26T02:00:00",
      "peak_level": 4.93
     },
     {
      "crossing": "2024-10-03T09:00:00",
      "peak_ts": "2024-10-05T12:00:00",
      "peak_level": 5.3
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.0890019157543212,
      "mae_above_2p5": 0.2093245927883516,
      "brier_warn": 0.005826169840715695,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 10.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.817519939833057
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.546588884631041
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.0890019157543212,
      "mae_above_2p5": 0.2093245927883516,
      "brier_warn": 0.005475787026695418,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 10.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.817519939833057
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.546588884631041
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 55083,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-27T18:00:00",
      "peak_ts": "2025-09-27T22:00:00",
      "peak_level": 3.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.11202835461695825,
      "mae_above_2p5": 0.2456782496767171,
      "brier_warn": 0.005178356039745929,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.11202835461695825,
      "mae_above_2p5": 0.2456782496767171,
      "brier_warn": 0.00491346243876589,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   }
  ]
 },
 {
  "station": "P.103",
  "warn_thr": 5.95,
  "folds": [
   {
    "year": 2021,
    "n_train": 20009,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.14438683486790602,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.14438683486790602,
      "mae_above_2p5": null,
      "brier_warn": 3.5349670949872053e-13,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28769,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-08-14T04:00:00",
      "peak_ts": "2022-08-14T08:00:00",
      "peak_level": 6.09
     },
     {
      "crossing": "2022-10-02T16:00:00",
      "peak_ts": "2022-10-03T16:00:00",
      "peak_level": 7.54
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.14392334606606189,
      "mae_above_2p5": 0.41075382386412596,
      "brier_warn": 0.00764679988213082,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.825270553204425
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194064117976157
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.14392334606606189,
      "mae_above_2p5": 0.41075382386412596,
      "brier_warn": 0.006975493312169236,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.825270553204425
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194064117976157
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37524,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.16195301512514512,
      "mae_above_2p5": null,
      "brier_warn": 3.0208441147560167e-07,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.16195301512514512,
      "mae_above_2p5": null,
      "brier_warn": 3.2176039819636275e-05,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46308,
    "n_test": 4058,
    "events": [
     {
      "crossing": "2024-09-24T10:00:00",
      "peak_ts": "2024-09-26T00:00:00",
      "peak_level": 8.27
     },
     {
      "crossing": "2024-09-30T03:00:00",
      "peak_ts": "2024-09-30T06:00:00",
      "peak_level": 5.99
     },
     {
      "crossing": "2024-10-03T06:00:00",
      "peak_ts": "2024-10-05T07:00:00",
      "peak_level": 9.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.13901842325128816,
      "mae_above_2p5": 0.2887416310966684,
      "brier_warn": 0.0072040857753137046,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.212734363860193
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.47565489914091
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.780278464915938
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.13901842325128816,
      "mae_above_2p5": 0.2887416310966684,
      "brier_warn": 0.006910847607098417,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.212734363860193
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.47565489914091
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.780278464915938
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 54577,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-26T06:00:00",
      "peak_ts": "2025-09-27T21:00:00",
      "peak_level": 6.64
     },
     {
      "crossing": "2025-10-03T06:00:00",
      "peak_ts": "2025-10-03T12:00:00",
      "peak_level": 6.14
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.17206685418009837,
      "mae_above_2p5": 0.3840520085986099,
      "brier_warn": 0.015824852891785323,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73043665401637
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.609919760117381
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_qsigma": {
      "mae": 0.17206685418009837,
      "mae_above_2p5": 0.3840520085986099,
      "brier_warn": 0.015889933586185904,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73043665401637
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.609919760117381
       }
      ],
      "false_alarm_episodes": 2
     }
    }
   }
  ]
 }
 ]
@@ -34,23 +34,23 @@ classifiers = [
    "Environment :: Web Environment",
    "Framework :: FastAPI"
 ]
-requires-python = ">=3.11"
+requires-python = ">=3.11,<3.12"
 dependencies = [
    # Core dependencies
-    "requests==2.31.0",
+    "requests==2.34.2",
    "schedule==1.2.0",
    "pandas==2.0.3",
    "numpy>=1.24,<2",
    # Flood forecasting (ML)
    "scikit-learn==1.9.0",
    # Web API framework
-    "fastapi==0.104.1",
+    "fastapi==0.141.1",
-    "uvicorn[standard]==0.24.0",
+    "uvicorn[standard]==0.52.4",
-    "pydantic==2.5.0",
+    "pydantic==2.13.5",
    # Database adapters
    "sqlalchemy==2.0.23",
    "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
    "psycopg2-binary==2.9.9",
    # Monitoring and metrics
    "psutil==5.9.6"
@@ -59,11 +59,11 @@ dependencies = [
 [project.optional-dependencies]
 dev = [
    # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
    "pytest-cov==4.1.0",
    "pytest-asyncio==0.21.1",
    # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
    "flake8==6.1.0",
    "isort==5.12.0",
    "mypy==1.7.1",
@@ -73,7 +73,7 @@ dev = [
    "ipython==8.17.2",
    "jupyter==1.0.0",
    # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
    "types-python-dateutil==2.8.19.14"
 ]
 docs = [
@@ -83,7 +83,7 @@ docs = [
 ]
 all = [
    "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
    "psycopg2-binary==2.9.9"
 ]
@@ -100,11 +100,11 @@ Documentation = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/
 [dependency-groups]
 dev = [
    # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
    "pytest-cov==4.1.0",
    "pytest-asyncio==0.21.1",
    # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
    "flake8==6.1.0",
    "isort==5.12.0",
    "mypy==1.7.1",
@@ -114,7 +114,7 @@ dev = [
    "ipython==8.17.2",
    "jupyter==1.0.0",
    # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
    "types-python-dateutil==2.8.19.14",
    # Documentation
    "sphinx==7.2.6",
@@ -128,3 +128,16 @@ where = ["src"]
 [tool.setuptools.package-dir]
 "" = "src"
 # One formatting contract for CI, pre-commit and editors. Black's default 88
 # columns; isort in black-compatible mode. Run `make format` before committing.
 [tool.black]
 line-length = 88
 target-version = ["py311"]
 extend-exclude = '/(\.venv|venv|models|\.claude-flow|\.swarm)/'
 [tool.isort]
 profile = "black"
 line_length = 88
 known_first_party = ["src"]
 skip_gitignore = true
@@ -2,12 +2,12 @@
 -r requirements.txt
 # Testing
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
 pytest-asyncio==0.21.1
 # Code formatting and linting
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 isort==5.12.0
 mypy==1.7.1
@@ -25,5 +25,5 @@ ipython==8.17.2
 jupyter==1.0.0
 # Type stubs
-types-requests==2.31.0.10
+types-requests==2.33.0.20260906
 types-python-dateutil==2.8.19.14
@@ -1,5 +1,5 @@
 # Core dependencies
-requests==2.31.0
+requests==2.34.2
 schedule==1.2.0
 pandas==2.0.3
 numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
@@ -8,23 +8,23 @@ numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
 scikit-learn==1.9.0
 # Web API framework
-fastapi==0.104.1
+fastapi==0.141.1
-uvicorn[standard]==0.24.0
+uvicorn[standard]==0.52.4
-pydantic==2.5.0
+pydantic==2.13.5
 # Database adapters
 sqlalchemy==2.0.23
 influxdb==5.3.1
-pymysql==1.1.0
+pymysql==1.2.0
 psycopg2-binary==2.9.9
 # Monitoring and metrics
 psutil==5.9.6
 # Development dependencies (optional)
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 mypy==1.7.1
 pre-commit==3.5.0
@@ -0,0 +1,58 @@
 """Serve the working-copy dashboard locally with API calls proxied to the
 live server, so browser-side changes can be checked against real data
 before deploy. Usage: python scripts/dev_proxy.py [port]"""
 import http.server
 import os
 import sys
 import urllib.request
 from pathlib import Path
 UPSTREAM = "https://water.buildfor.life"
 STATIC = Path(__file__).resolve().parents[1] / "src" / "static"
 class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/" or self.path.startswith("/?"):
            body = (STATIC / "dashboard.html").read_bytes()
            self._send(200, "text/html; charset=utf-8", body)
            return
        # Local overrides for endpoints not yet deployed: DEV_PROXY_LOCAL=/api/x=file.json,...
        for pair in filter(None, os.environ.get("DEV_PROXY_LOCAL", "").split(",")):
            prefix, file = pair.split("=", 1)
            if self.path.split("?")[0] == prefix:
                self._send(200, "application/json", Path(file).read_bytes())
                return
        if self.path.startswith("/static/"):
            f = STATIC / self.path[len("/static/"):].split("?")[0]
            if f.is_file():
                ctype = "application/json" if f.suffix in (".json", ".geojson") else "application/octet-stream"
                self._send(200, ctype, f.read_bytes())
                return
        try:
            req = urllib.request.Request(
                UPSTREAM + self.path,
                headers={"User-Agent": "Mozilla/5.0 (dev_proxy; +https://buildfor.life)", "Accept": "application/json"},
            )
            with urllib.request.urlopen(req, timeout=60) as r:
                self._send(r.status, r.headers.get("Content-Type", "application/json"), r.read())
        except urllib.error.HTTPError as e:
            self._send(e.code, "application/json", e.read())
    def _send(self, code, ctype, body):
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Cache-Control", "no-store")
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a):
        pass
 if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8765
    print(f"http://localhost:{port}/  (API -> {UPSTREAM})")
    http.server.ThreadingHTTPServer(("127.0.0.1", port), Handler).serve_forever()
@@ -0,0 +1,132 @@
 """Drive the production notify path in-process: startup init -> seeded readings
 -> forecast cache -> _notify_transitions -> sqlite state -> real ntfy."""
 import asyncio
 import datetime
 import json
 import os
 import sys
 import requests
 os.environ.update(
    DB_TYPE="sqlite",
    WATER_DB_PATH=os.path.join(os.environ["LOCALAPPDATA"], "Temp", "smoke3.db"),
    NTFY_SERVER="http://127.0.0.1:2586",
    NTFY_TOKEN=os.environ.get("NTFY_TOKEN", ""),
    NTFY_TOPIC_PREFIX="ping",
 )
 for f in ("smoke3.db",):
    p = os.path.join(os.environ["LOCALAPPDATA"], "Temp", f)
    if os.path.exists(p):
        os.remove(p)
 from src import web_api  # noqa: E402
 from src.config import Config  # noqa: E402
 assert Config.NTFY_SERVER
 async def main():
    # what the lifespan does at startup, minus the scheduler
    from src import notify as notify_mod
    from src.forecast_history import ForecastHistoryStore
    from src.water_scraper_v3 import EnhancedWaterMonitorScraper
    db_config = Config.get_database_config()
    web_api.app_state["scraper"] = EnhancedWaterMonitorScraper(db_config)
    store = ForecastHistoryStore(db_config["connection_string"], db_config["type"])
    store.connect()
    web_api.app_state["forecast_store"] = store
    state = notify_mod.NotificationState(store.engine, store.db_type)
    pub = notify_mod.NtfyPublisher(
        Config.NTFY_SERVER, prefix=Config.NTFY_TOPIC_PREFIX, token=Config.NTFY_TOKEN
    )
    web_api.app_state["notify"] = (pub, state)
    scraper = web_api.app_state["scraper"]
    now = datetime.datetime.now().replace(minute=0, second=0, microsecond=0)
    def seed(level_p1, level_p103, ts):
        rows = [
            {
                "station_code": "P.1",
                "station_id": 1,
                "timestamp": ts,
                "water_level": level_p1,
                "discharge": 400.0,
                "station_name_en": "Nawarat Bridge",
                "station_name_th": "สะพานนวรัฐ",
                "discharge_percent": 30.0,
                "status": "active",
            },
            {
                "station_code": "P.103",
                "station_id": 2,
                "timestamp": ts,
                "water_level": level_p103,
                "discharge": 300.0,
                "station_name_en": "Ring Road 3",
                "station_name_th": "วงแหวน 3",
                "discharge_percent": 20.0,
                "status": "active",
            },
        ]
        scraper.db_adapter.save_measurements(rows)
    def forecast(p):
        with web_api.FORECAST_CACHE_LOCK:
            web_api.FORECAST_CACHE["all"] = (
                0,
                [
                    {
                        "station_code": "P.1",
                        "horizon_hours": 24,
                        "p_warning": p,
                        "predicted_max_level": 3.9,
                        "source": "model",
                    }
                ],
            )
    def poll(topic):
        out = []
        for line in (
            requests.get(f"{Config.NTFY_SERVER}/{topic}/json?poll=1", timeout=5)
            .text.strip()
            .splitlines()
        ):
            m = json.loads(line)
            if m.get("event") == "message":
                out.append(m.get("title") or m.get("message", "")[:40])
        return out
    # cycle 1: quiet
    seed(1.6, 3.2, now - datetime.timedelta(hours=2))
    forecast(0.02)
    await web_api._notify_transitions()
    # cycle 2: P.1 crosses warning, model outlook on
    seed(3.75, 3.3, now - datetime.timedelta(hours=1))
    forecast(0.7)
    await web_api._notify_transitions()
    # cycle 3: same state -> silence
    seed(3.80, 3.3, now)
    forecast(0.65)
    await web_api._notify_transitions()
    print("ping-p1-warning:", poll("ping-p1-warning"))
    print("ping-warning:   ", poll("ping-warning"))
    print("ping-p1-outlook:", poll("ping-p1-outlook"))
    print("ping-p103-warning:", poll("ping-p103-warning"))
    from sqlalchemy import text
    with store.engine.connect() as c:
        print(
            "state table:",
            c.execute(
                text("SELECT key, state, value FROM notification_state ORDER BY key")
            ).fetchall(),
        )
 asyncio.run(main())
@@ -18,6 +18,7 @@ APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
 SERVICE_USER="${SERVICE_USER:-water-monitor}"
 SERVICE_GROUP="${SERVICE_GROUP:-${SERVICE_USER}}"
 SERVICE_NAME="water-monitor.service"
 RETRAIN_NAME="water-monitor-retrain"
 # Resolve the repo root (parent of this scripts/ directory).
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -72,11 +73,18 @@ if ! command -v uv >/dev/null 2>&1; then
 fi
 UV="$(command -v uv)"
-log "Creating virtualenv at ${APP_DIR}/venv"
+log "Syncing uv-managed virtualenv at ${APP_DIR}/.venv"
 cd "${APP_DIR}"
-# Named 'venv' (not uv's default .venv) to match the systemd unit's ExecStart.
+# ONE environment: uv sync owns .venv/ (from pyproject.toml + uv.lock, so the
-"${UV}" venv venv
+# ML extras such as scikit-learn/joblib are present) and both systemd units
-"${UV}" pip install --python venv/bin/python -r requirements.txt
+# run its interpreter directly. Never create a second env by another name --
 # a stale 'venv/' once coexisted here and broke manual retrains with
 # ModuleNotFoundError while the service itself ran fine.
 "${UV}" sync --python 3.11 --frozen
 if [ -d "${APP_DIR}/venv" ]; then
    warn "Removing stale ${APP_DIR}/venv (superseded by .venv)"
    rm -rf "${APP_DIR}/venv"
 fi
 # 4. Environment file ----------------------------------------------------------
 if [ ! -f "${APP_DIR}/.env" ]; then
@@ -100,11 +108,14 @@ if [ -f "${APP_DIR}/.env" ]; then
    chmod 0600 "${APP_DIR}/.env"
 fi
-# 6. Install and enable the systemd unit --------------------------------------
+# 6. Install and enable the systemd units -------------------------------------
-log "Installing systemd unit"
+log "Installing systemd units"
 install -m 0644 "${SCRIPT_DIR}/${SERVICE_NAME}" "/etc/systemd/system/${SERVICE_NAME}"
 install -m 0644 "${SCRIPT_DIR}/${RETRAIN_NAME}.service" "/etc/systemd/system/${RETRAIN_NAME}.service"
 install -m 0644 "${SCRIPT_DIR}/${RETRAIN_NAME}.timer" "/etc/systemd/system/${RETRAIN_NAME}.timer"
 systemctl daemon-reload
 systemctl enable "${SERVICE_NAME}"
 systemctl enable --now "${RETRAIN_NAME}.timer"
 log "Done."
 echo
@@ -112,3 +123,5 @@ echo "Next steps:"
 echo "  sudo systemctl start ${SERVICE_NAME}"
 echo "  systemctl status ${SERVICE_NAME}"
 echo "  sudo journalctl -u ${SERVICE_NAME} -f"
 echo "  systemctl list-timers ${RETRAIN_NAME}.timer   # monthly flood-model retrain"
 echo "  sudo systemctl start ${RETRAIN_NAME}.service   # retrain now"
@@ -0,0 +1,102 @@
 #!/usr/bin/env bash
 # Install ntfy (https://ntfy.sh) as the public notification server for the
 # Ping River Monitor. Run as root on the monitor VPS. Idempotent.
 #
 #   NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
 #
 # What it does:
 #   - installs the ntfy .deb from the official GitHub release (single Go
 #     binary, ~30 MB RSS, sqlite message cache)
 #   - writes /etc/ntfy/server.yml: listens on the Tailscale address only
 #     (the reverse proxy is another VPS on the tailnet; nothing is exposed
 #     on a public interface), anonymous READ on all topics, WRITE only with
 #     a token. Override with NTFY_LISTEN=host:port.
 #   - creates the `monitor` publishing user + token, writes NTFY_SERVER /
 #     NTFY_TOKEN into /opt/thailand-water-monitor/.env if not present
 #
 # Reverse proxy (on the Caddy VPS, over Tailscale):
 #     ntfy.buildfor.life {
 #         reverse_proxy <this host's tailscale ip>:2586
 #     }
 # Caddy passes websockets and keeps long-poll connections open by default;
 # subscribers hold one open. ntfy runs with behind-proxy: true so rate
 # limits key on X-Forwarded-For, not on the proxy's address.
 set -euo pipefail
 NTFY_DOMAIN="${NTFY_DOMAIN:?set NTFY_DOMAIN, e.g. ntfy.buildfor.life}"
 NTFY_VERSION="${NTFY_VERSION:-2.28.0}"
 MONITOR_DIR="${MONITOR_DIR:-/opt/thailand-water-monitor}"
 TS_IP="$(tailscale ip -4 2>/dev/null | head -1 || true)"
 LISTEN="${NTFY_LISTEN:-${TS_IP:-127.0.0.1}:2586}"
 echo "ntfy will listen on ${LISTEN}"
 if ! command -v ntfy >/dev/null || [[ "$(ntfy --version 2>/dev/null | awk '{print $3}')" != "$NTFY_VERSION" ]]; then
    tmp=$(mktemp -d)
    curl -fsSL -o "$tmp/ntfy.deb" \
        "https://github.com/binwiederhier/ntfy/releases/download/v${NTFY_VERSION}/ntfy_${NTFY_VERSION}_linux_amd64.deb"
    dpkg -i "$tmp/ntfy.deb"
    rm -rf "$tmp"
 fi
 install -d -m 755 /var/cache/ntfy /var/lib/ntfy
 cat > /etc/ntfy/server.yml <<EOF
 # Ping River Monitor notification server. Managed by scripts/install_ntfy.sh.
 base-url: "https://${NTFY_DOMAIN}"
 listen-http: "${LISTEN}"
 behind-proxy: true
 # Messages are kept so a phone that was offline still gets the crossing.
 cache-file: "/var/cache/ntfy/cache.db"
 cache-duration: "72h"
 # Everyone may subscribe; only the monitor (token) may publish.
 auth-file: "/var/lib/ntfy/user.db"
 auth-default-access: "read-only"
 # The monitor publishes a handful of messages per flood; be strict with
 # everything else so the box cannot be used as a free relay.
 visitor-request-limit-burst: 30
 visitor-request-limit-replenish: "10s"
 visitor-subscription-limit: 60
 visitor-message-daily-limit: 200
 attachment-cache-dir: ""
 enable-signup: false
 enable-login: false
 enable-metrics: false
 EOF
 systemctl enable --now ntfy
 systemctl restart ntfy
 sleep 1
 curl -fsS "http://${LISTEN}/v1/health" >/dev/null && echo "ntfy up on ${LISTEN}"
 # Publishing identity for the monitor
 if ! ntfy user list 2>/dev/null | grep -q '^user monitor (role'; then
    NTFY_PASSWORD="$(openssl rand -base64 24)" ntfy user add --role=user monitor
 fi
 ntfy access monitor 'ping-*' write-only >/dev/null
 # 'ping-*' read stays anonymous via auth-default-access
 token=$(ntfy token list monitor 2>/dev/null | awk '/^- tk_/{print $2; exit}')  # '- tk_xxx (label), ...'
 if [[ -z "$token" ]]; then
    token=$(ntfy token add --label "water-monitor" monitor | grep -oE 'tk_[A-Za-z0-9]+' | head -1)  # 'token tk_xxx created for user monitor'
 fi
 env_file="${MONITOR_DIR}/.env"
 if [[ -f "$env_file" ]] && ! grep -q '^NTFY_SERVER=' "$env_file"; then
    {
        echo ""
        echo "# ntfy public notifications (scripts/install_ntfy.sh)"
        echo "NTFY_SERVER=https://${NTFY_DOMAIN}"
        echo "NTFY_PUBLISH_URL=http://${LISTEN}"
        echo "NTFY_TOPIC_PREFIX=ping"
        echo "NTFY_TOKEN=${token}"
    } >> "$env_file"
    echo "wrote NTFY_* to ${env_file}; restart water-monitor to enable"
 else
    echo "NTFY_TOKEN=${token}"
 fi
 echo
 echo "Subscribe test (anonymous read):  curl -s 'http://${LISTEN}/ping-status/json?poll=1'"
 echo "Publish test (needs token):       curl -s -H 'Authorization: Bearer ${token}' -d 'hello' http://${LISTEN}/ping-status"
@@ -0,0 +1,90 @@
 #!/usr/bin/env bash
 #
 # Retrain the flood forecast models safely. Run by water-monitor-retrain.timer
 # (monthly) or by hand:  sudo systemctl start water-monitor-retrain.service
 #
 # Why a script rather than ExecStart=train_flood_model.py:
 #   * train.py writes each station's bundle straight into models/ over ~12 min,
 #     and the API's hourly precompute reloads bundles by mtime. Training into
 #     a staging dir and mv-ing (atomic on one filesystem) means the API never
 #     sees a half-written joblib file or a mixed old/new set.
 #   * A run that produced gauge-only (v2) bundles, or trained too few stations,
 #     must NOT replace the deployed models. train.py already aborts on a
 #     missing rain series; this script re-checks the written metrics anyway.
 #   * No API restart is needed: predict.py reloads changed bundles on the next
 #     precompute (every scrape cycle, hourly), so the new models are live
 #     within an hour. Restart manually if you want them live immediately.
 #
 # Exit codes: 0 ok, 2 training refused (see log), 3 verification failed.
 set -euo pipefail
 APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
 PYTHON="${PYTHON:-${APP_DIR}/.venv/bin/python}"
 MODELS_DIR="${APP_DIR}/models"
 STAGE_DIR="${MODELS_DIR}/.staging"
 # P.4A is NOT_TRAINABLE by design (17% fill); 15 of 16 is the normal outcome.
 MIN_TRAINED="${MIN_TRAINED:-14}"
 EXPECT_VERSION_PREFIX="${EXPECT_VERSION_PREFIX:-hgb-v3+}"
 log() { printf '%s retrain: %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }
 cd "${APP_DIR}"
 [ -x "${PYTHON}" ] || { log "no interpreter at ${PYTHON} (run uv sync)"; exit 3; }
 rm -rf "${STAGE_DIR}"
 mkdir -p "${STAGE_DIR}"
 log "training into ${STAGE_DIR} (python=${PYTHON}, OMP_NUM_THREADS=${OMP_NUM_THREADS:-unset})"
 # train_flood_model.py exits 2 on a missing rain series (RainUnavailableError)
 # instead of silently writing v2 bundles -- propagate that unchanged.
 set +e
 "${PYTHON}" scripts/train_flood_model.py --stations all --models-dir "${STAGE_DIR}" "$@"
 rc=$?
 set -e
 if [ "${rc}" -ne 0 ]; then
    log "training failed (exit ${rc}); deployed models untouched"
    rm -rf "${STAGE_DIR}"
    exit "${rc}"
 fi
 # Verify before promoting. Reads metrics.json from the stage dir.
 VERSION="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
 import json, sys
 m = json.load(open(sys.argv[1]))
 print(m["model_version"])
 PY
 )"
 TRAINED="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
 import json, sys
 m = json.load(open(sys.argv[1]))
 print(sum(1 for s in m["stations"].values() if s.get("status") == "trained"))
 PY
 )"
 log "staged model_version=${VERSION} trained_stations=${TRAINED}"
 case "${VERSION}" in
    "${EXPECT_VERSION_PREFIX}"*) ;;
    *)
        log "REFUSING to deploy: version '${VERSION}' does not start with '${EXPECT_VERSION_PREFIX}'"
        rm -rf "${STAGE_DIR}"
        exit 3
        ;;
 esac
 if [ "${TRAINED}" -lt "${MIN_TRAINED}" ]; then
    log "REFUSING to deploy: only ${TRAINED} stations trained (< ${MIN_TRAINED})"
    rm -rf "${STAGE_DIR}"
    exit 3
 fi
 # Promote: per-file rename is atomic; readers see either the old or the new
 # bundle, never a partial one. Keep one previous generation for rollback.
 mkdir -p "${MODELS_DIR}/.previous"
 for f in "${STAGE_DIR}"/flood_*.joblib "${STAGE_DIR}/metrics.json"; do
    name="$(basename "${f}")"
    if [ -f "${MODELS_DIR}/${name}" ]; then
        mv -f "${MODELS_DIR}/${name}" "${MODELS_DIR}/.previous/${name}"
    fi
    mv -f "${f}" "${MODELS_DIR}/${name}"
 done
 rm -rf "${STAGE_DIR}"
 log "deployed ${VERSION} (${TRAINED} stations); previous generation in models/.previous. The API picks it up on its next hourly precompute."
@@ -0,0 +1,62 @@
 """Summarise rolling-origin harness output side by side.
 Usage:
    uv run python scripts/summarize_eval.py models/eval_2026-09-12.json [more.json ...]
 Aggregates each (station, variant) across folds: mean MAE, mean flood-regime
 MAE, mean Brier, total false-alarm episodes, and every warning event with its
 first-alert lead and the 24 h-ahead peak error -- the operational numbers that
 decide whether a variant ships.
 """
 import json
 import statistics
 import sys
 from collections import OrderedDict
 def summarize(paths):
    for path in paths:
        results = json.load(open(path, encoding="utf-8"))
        print(f"\n##### {path}")
        for station in results:
            print(f"\n=== {station['station']} (warn {station['warn_thr']:.2f} m) ===")
            agg = OrderedDict()
            for fold in station["folds"]:
                for name, m in fold["variants"].items():
                    a = agg.setdefault(
                        name, {"mae": [], "mae_hi": [], "brier": [], "fa": 0, "events": []}
                    )
                    a["mae"].append(m["mae"])
                    if m.get("mae_above_2p5") is not None:
                        a["mae_hi"].append(m["mae_above_2p5"])
                    if m.get("brier_warn") is not None:
                        a["brier"].append(m["brier_warn"])
                    a["fa"] += m["false_alarm_episodes"]
                    for e in m["events"]:
                        err = (
                            None
                            if e["peak_pred_24h_before"] is None
                            else e["peak_pred_24h_before"] - e["peak_level"]
                        )
                        a["events"].append((fold["year"], e["crossing"][:10], e["lead_h"], e["peak_level"], err))
            print(f"{'variant':22} {'MAE':>6} {'MAE_hi':>7} {'Brier':>7} {'FA':>3}  events: year crossing lead_h peak(err24h)")
            for name, a in agg.items():
                ev = "  ".join(
                    f"{y} {d} {'—' if l is None else format(l, '+.0f')}h {p:.2f}({'—' if err is None else format(err, '+.2f')})"
                    for y, d, l, p, err in a["events"]
                )
                leads = [l for *_, l, _, _ in a["events"] if l is not None]
                print(
                    f"{name:22} {statistics.mean(a['mae']):6.3f} "
                    f"{statistics.mean(a['mae_hi']) if a['mae_hi'] else float('nan'):7.3f} "
                    f"{statistics.mean(a['brier']) if a['brier'] else float('nan'):7.4f} "
                    f"{a['fa']:>3}  {ev}"
                )
                if leads:
                    print(f"{'':22} lead: mean {statistics.mean(leads):+.1f} h, min {min(leads):+.0f} h, "
                          f"missed {sum(1 for *_, l, _, _ in a['events'] if l is None)}/{len(a['events'])}")
 if __name__ == "__main__":
    summarize(sys.argv[1:] or ["models/eval_variants.json"])
@@ -11,7 +11,7 @@ import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
-from src.ml.train import main
+from src.ml.train import cli
 if __name__ == "__main__":
-    main()
+    raise SystemExit(cli())
@@ -0,0 +1,39 @@
 [Unit]
 Description=Retrain the Ping River flood forecast models
 Documentation=https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/-/blob/master/docs/FLOOD_FORECASTING.md
 After=network-online.target
 Wants=network-online.target
 [Service]
 Type=oneshot
 User=water-monitor
 Group=water-monitor
 WorkingDirectory=/opt/thailand-water-monitor
 EnvironmentFile=/opt/thailand-water-monitor/.env
 # Same interpreter as water-monitor.service -- the uv-managed .venv.
 # scripts/retrain.sh trains into models/.staging, refuses to promote anything
 # that is not a rain-enabled (hgb-v3) set covering the expected stations, then
 # renames the bundles into place. The API reloads them on its next hourly
 # precompute; no restart, so a failed run leaves the old models serving.
 ExecStart=/bin/bash /opt/thailand-water-monitor/scripts/retrain.sh
 # HistGradientBoosting is CPU-bound; cap threads so training cannot starve
 # the API (docs/FLOOD_FORECASTING.md section 6 measured 4 as the sweet spot).
 Environment=OMP_NUM_THREADS=4
 Environment=PYTHONPATH=/opt/thailand-water-monitor
 Environment=PYTHONUNBUFFERED=1
 Nice=15
 IOSchedulingClass=idle
 # 15 stations at ~50 s each plus data load: 12 min observed on 2026-09-12.
 TimeoutStartSec=45min
 # Same sandbox as the API unit.
 NoNewPrivileges=true
 PrivateTmp=true
 ProtectSystem=strict
 ProtectHome=true
 ReadWritePaths=/opt/thailand-water-monitor
 CapabilityBoundingSet=
 StandardOutput=journal
 StandardError=journal
 SyslogIdentifier=water-monitor-retrain
@@ -0,0 +1,17 @@
 [Unit]
 Description=Monthly flood-model retrain (docs/FLOOD_FORECASTING.md section 7)
 [Timer]
 # Policy: at minimum once pre-monsoon (May-June), monthly through the season
 # (July-November), and after any major flood. A retrain costs ~12 min and RAM
 # peaks ~300 MB, so running it every month all year is cheaper than remembering
 # which months matter. 1st of the month, 03:30 server-local -- between the
 # hourly scrapes and outside Thai daytime traffic.
 OnCalendar=*-*-01 03:30:00
 # Catch up if the box was off at the scheduled time.
 Persistent=true
 RandomizedDelaySec=20min
 Unit=water-monitor-retrain.service
 [Install]
 WantedBy=timers.target
@@ -9,17 +9,19 @@ Type=simple
 User=water-monitor
 Group=water-monitor
 WorkingDirectory=/opt/thailand-water-monitor
-ExecStart=/opt/thailand-water-monitor/venv/bin/python src/water_scraper_v3.py
+# The uv-managed env (uv sync -> .venv). Same interpreter for water-monitor-retrain.service.
 ExecStart=/opt/thailand-water-monitor/.venv/bin/python run.py --web-api
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=60
 TimeoutStopSec=30
-# Environment variables
+# DB_TYPE / POSTGRES_CONNECTION_STRING / MATRIX_* come from the .env file.
-Environment=DB_TYPE=victoriametrics
+EnvironmentFile=/opt/thailand-water-monitor/.env
 Environment=VM_HOST=localhost
 Environment=VM_PORT=8428
 Environment=PYTHONPATH=/opt/thailand-water-monitor
 # Serving path is latency-bound; single-threaded BLAS is 2.6x faster per call
 # (docs/FLOOD_FORECASTING.md section 6). Training sets its own value.
 Environment=OMP_NUM_THREADS=1
 Environment=PYTHONUNBUFFERED=1
 # Security settings
@@ -12,9 +12,13 @@ __description__ = "Northern Thailand Ping River Monitoring System"
 from .config import Config
 from .database_adapters import DatabaseAdapter, create_database_adapter
-from .exceptions import (APIConnectionError, ConfigurationError,
+from .exceptions import (
-                         DatabaseConnectionError, DataValidationError,
+    APIConnectionError,
-                         WaterMonitorException)
+    ConfigurationError,
    DatabaseConnectionError,
    DataValidationError,
    WaterMonitorException,
 )
 from .models import DatabaseConfig, StationInfo, WaterMeasurement
 from .water_scraper_v3 import EnhancedWaterMonitorScraper
@@ -38,6 +38,17 @@ class Config:
    TARGET_URL = "https://hyd-app-db.rid.go.th/hydro1h.html"
    API_URL = "https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx"
    THAIWATER_API_KEY = os.getenv("THAIWATER_API_KEY")
    # Public flood notifications (ntfy). Off unless NTFY_SERVER is set.
    # NTFY_SERVER is what subscribers use (public https URL, shown on the
    # dashboard). NTFY_PUBLISH_URL is where the monitor POSTs; defaults to
    # NTFY_SERVER, set it to http://127.0.0.1:2586 when ntfy runs on the same
    # host so publishing never depends on DNS/proxy/tunnel being up.
    NTFY_SERVER = os.getenv("NTFY_SERVER", "").strip()
    NTFY_PUBLISH_URL = os.getenv("NTFY_PUBLISH_URL", "").strip() or NTFY_SERVER
    NTFY_TOPIC_PREFIX = os.getenv("NTFY_TOPIC_PREFIX", "ping").strip()
    NTFY_TOKEN = os.getenv("NTFY_TOKEN", "").strip()  # publish token if ACL enabled
    PUBLIC_URL = os.getenv("PUBLIC_URL", "https://water.buildfor.life/").strip()
    REQUEST_TIMEOUT = int(os.getenv("REQUEST_TIMEOUT", "30"))
    USER_AGENT = (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
@@ -139,12 +139,16 @@ class InfluxDBAdapter(DatabaseAdapter):
                    "time": measurement["timestamp"].isoformat(),
                    "fields": {
                        "water_level": float(measurement["water_level"]),
-                        "discharge": float(measurement["discharge"])
+                        "discharge": (
                            float(measurement["discharge"])
                            if measurement.get("discharge") is not None
-                        else None,
+                            else None
-                        "discharge_percent": float(measurement["discharge_percent"])
+                        ),
                        "discharge_percent": (
                            float(measurement["discharge_percent"])
                            if measurement.get("discharge_percent")
-                        else None,
+                            else None
                        ),
                    },
                }
                points.append(point)
@@ -551,13 +555,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_code": row[1],
                            "station_name_en": row[2],
                            "station_name_th": row[3],
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -611,13 +615,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_code": row[1],
                            "station_name_en": row[2],
                            "station_name_th": row[3],
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -666,13 +670,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_id": row[1],
                            "station_code": row[2] or f"Station_{row[1]}",
                            "station_name_th": row[3] or f"Station {row[1]}",
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -767,9 +771,7 @@ class SQLAdapter(DatabaseAdapter):
            return hours_by_day
        except Exception as e:
-            logging.error(
+            logging.error(f"Error querying {self.db_type.upper()} recorded hours: {e}")
                f"Error querying {self.db_type.upper()} recorded hours: {e}"
            )
            return None
    def get_database_stats(self) -> Optional[Dict]:
@@ -814,9 +816,7 @@ class SQLAdapter(DatabaseAdapter):
            # the DISTINCT day-hour slots and coverage cannot exceed 100%
            first_slot = first_ts.replace(minute=0, second=0, microsecond=0)
            last_slot = last_ts.replace(minute=0, second=0, microsecond=0)
-            expected_hours = (
+            expected_hours = int((last_slot - first_slot).total_seconds() // 3600) + 1
                int((last_slot - first_slot).total_seconds() // 3600) + 1
            )
            recorded_hours = int(row[4])
            coverage_percent = round(100.0 * recorded_hours / expected_hours, 1)
@@ -125,9 +125,9 @@ class DatabaseHealthCheck(HealthCheck):
                "message": "Database connection OK",
                "details": {
                    "latest_data_count": len(latest_data),
-                    "latest_timestamp": str(latest_data[0].get("timestamp"))
+                    "latest_timestamp": (
-                    if latest_data
+                        str(latest_data[0].get("timestamp")) if latest_data else None
-                    else None,
+                    ),
                },
            }
@@ -17,9 +17,9 @@ import time
 from typing import Dict, List, Optional
 from .hii_collector import (
    PING_BASIN_CODE,
    HiiClient,
    HiiStore,
    PING_BASIN_CODE,
    _parse_datetime,
    _to_float,
 )
@@ -145,9 +145,7 @@ def backfill(
                station_rows += store.save_waterlevel_history(sid, rows)
            except Exception as e:
                totals["errors"] += 1
-                logger.warning(
+                logger.warning(f"{label}: {chunk_start}..{chunk_end} failed: {e}")
                    f"{label}: {chunk_start}..{chunk_end} failed: {e}"
                )
            time.sleep(sleep_seconds)
        totals["rows"] += station_rows
        logger.info(f"{label} (id {sid}): {station_rows} rows saved")
@@ -66,9 +66,7 @@ def rid_code_from_oldcode(oldcode: Optional[str]) -> Optional[str]:
    return match.group(1) if match else None
-def parse_rain_records(
+def parse_rain_records(payload: Dict, basin_code: int = PING_BASIN_CODE) -> List[Dict]:
    payload: Dict, basin_code: int = PING_BASIN_CODE
 ) -> List[Dict]:
    """Extract per-station rainfall rows from a rain_24h payload."""
    records = []
    for row in payload.get("data") or []:
@@ -186,9 +184,7 @@ class HiiStore:
    def __init__(self, connection_string: str, db_type: str):
        self.db_type = db_type.lower()
        if self.db_type not in ("sqlite", "postgresql", "mysql"):
-            raise ValueError(
+            raise ValueError(f"HII collection requires a SQL database, got '{db_type}'")
                f"HII collection requires a SQL database, got '{db_type}'"
            )
        self.connection_string = connection_string
        self.engine = None
@@ -400,9 +396,7 @@ class HiiStore:
        from sqlalchemy import text
        now = datetime.datetime.now()
-        station_sql = self._upsert(
+        station_sql = self._upsert(station_table, ["id"], station_cols + ["updated_at"])
            station_table, ["id"], station_cols + ["updated_at"]
        )
        measurement_sql = self._upsert(
            measurement_table, ["station_id", "timestamp"], measurement_cols
        )
@@ -61,9 +61,7 @@ def load_daily(
                params["start"] = start
            engine = create_engine(resolved, pool_pre_ping=True)
            with engine.connect() as conn:
-                daily = pd.read_sql(
+                daily = pd.read_sql(text(query + " ORDER BY date"), conn, params=params)
                    text(query + " ORDER BY date"), conn, params=params
                )
            daily["date"] = pd.to_datetime(daily["date"])
            daily = daily.set_index("date")
            for col in DAM_COLUMNS:
@@ -23,7 +23,9 @@ from .features import UPSTREAM_LEADS
 logger = logging.getLogger(__name__)
-DEFAULT_API_URL = "http://100.81.167.42:8000"
+# Public dashboard. Override with --api-url for a local instance; the
 # Tailscale address of the server is deliberately not the default here.
 DEFAULT_API_URL = "https://water.buildfor.life"
 # Anchored to the repo root so training/prediction work from any CWD; a relative
 # path here silently produced 0 rows when the CLI ran outside the repo root.
 CACHE_DIR = Path(__file__).resolve().parents[2] / "models" / "cache"
@@ -205,15 +207,13 @@ def fill_from_hii(
                "timestamp": missing["timestamp"],
                "station_code": code,
                "water_level": missing["wl_msl"] - offset,
-                "discharge": missing["discharge"]
+                "discharge": (
-                if code in _HII_EXACT_MIRRORS
+                    missing["discharge"] if code in _HII_EXACT_MIRRORS else float("nan")
-                else float("nan"),
+                ),
            }
        )
        fills.append(fill)
-        logger.info(
+        logger.info(f"HII gap-fill {code}: +{len(fill)} hours (offset {offset:.3f} m)")
            f"HII gap-fill {code}: +{len(fill)} hours (offset {offset:.3f} m)"
        )
    if not fills:
        return df
    return _normalize_long(pd.concat([df] + fills, ignore_index=True))
@@ -282,6 +282,10 @@ def _read_cache(cache_dir: Path, stations: Optional[List[str]]) -> pd.DataFrame:
    frames = []
    for path in sorted(cache_dir.glob("*.csv.gz")):
        code = path.name[: -len(".csv.gz")]
        # The dir is shared with rain.py / dam.py caches (rain_openmeteo,
        # dam_<id>): only station files (P.<n>) are measurements.
        if not code.startswith("P."):
            continue
        if stations and code not in stations:
            continue
        with gzip.open(path, "rt", encoding="utf-8") as handle:
@@ -52,22 +52,40 @@ def _flood_weights(y_abs: pd.Series) -> np.ndarray:
    return 1.0 + 4.0 * np.clip((y_abs.to_numpy() - 2.5) / 1.2, 0.0, 1.0)
 # Experimental forward-48h rain sum, built in evaluate_station (not in
 # features.build_features) so the served feature set is untouched until the
 # harness says it helps. Serving could supply it: fetch_forecast() already
 # pulls forecast_days=2.
 EXTRA_RAIN_FEATURES = ("rain_fc48",)
 class Variant:
    """A trainable candidate producing (pred_abs, sigma_per_row) on test rows."""
-    def __init__(self, name: str, target: str, weighted: bool = False,
+    def __init__(
-                 quantile: bool = False, use_rain: bool = False,
+        self,
-                 use_dam: bool = False):
+        name: str,
        target: str,
        weighted: bool = False,
        quantile: bool = False,
        use_rain: bool = False,
        use_dam: bool = False,
        use_fc48: bool = False,
        qsigma: bool = False,
    ):
        self.name = name
        self.target = target  # 'abs' or 'rise'
        self.weighted = weighted
        self.quantile = quantile
        self.use_rain = use_rain
        self.use_dam = use_dam
        self.use_fc48 = use_fc48
        # Hybrid: L2 head for the point prediction (keeps the lead-time
        # behaviour of the deployed model exactly, since p>=0.5 alerts are
        # sigma-independent) and quantile heads ONLY for a per-row sigma.
        self.qsigma = qsigma
-    def fit_predict(
+    def fit_predict(self, X_tr, y_abs_tr, X_te) -> Tuple[np.ndarray, np.ndarray]:
        self, X_tr, y_abs_tr, X_te
    ) -> Tuple[np.ndarray, np.ndarray]:
        if not self.use_rain:
            drop = [c for c in features.RAIN_FEATURES if c in X_tr.columns]
            X_tr = X_tr.drop(columns=drop)
@@ -84,6 +102,12 @@ class Variant:
            raise ValueError(
                f"{self.name} requires the dam series (rid_reservoir_daily backfilled)"
            )
        if not self.use_fc48:
            drop = [c for c in EXTRA_RAIN_FEATURES if c in X_tr.columns]
            X_tr = X_tr.drop(columns=drop)
            X_te = X_te.drop(columns=drop)
        elif "rain_fc48" not in X_tr.columns:
            raise ValueError(f"{self.name} requires the rain series")
        level_tr = X_tr["level"]
        level_te = X_te["level"].to_numpy()
        y_tr = (y_abs_tr - level_tr) if self.target == "rise" else y_abs_tr
@@ -99,6 +123,12 @@ class Variant:
        else:
            reg = _make_regressor().fit(X_tr, y_tr, sample_weight=weights)
            pred = reg.predict(X_te)
            if self.qsigma:
                q50 = _quantile_regressor(0.5).fit(X_tr, y_tr, sample_weight=weights)
                q90 = _quantile_regressor(0.9).fit(X_tr, y_tr, sample_weight=weights)
                spread = np.maximum(q90.predict(X_te) - q50.predict(X_te), 0.0)
                sigma = np.maximum(spread / 1.2816, 0.05)
            else:
                sigma = np.full(len(X_te), FIXED_SIGMA)
        pred_abs = pred + level_te if self.target == "rise" else pred
@@ -110,18 +140,44 @@ VARIANTS: Dict[str, Variant] = {
    "baseline_abs": Variant("baseline_abs", target="abs"),
    "rise": Variant("rise", target="rise"),
    "rise_weighted": Variant("rise_weighted", target="rise", weighted=True),
-    "rise_quantile": Variant("rise_quantile", target="rise", weighted=True,
+    "rise_quantile": Variant(
-                             quantile=True),
+        "rise_quantile", target="rise", weighted=True, quantile=True
    ),
    "rise_rain": Variant("rise_rain", target="rise", use_rain=True),
-    "rise_rain_dam": Variant("rise_rain_dam", target="rise", use_rain=True,
+    "rise_rain_dam": Variant(
-                             use_dam=True),
+        "rise_rain_dam", target="rise", use_rain=True, use_dam=True
    ),
    "rise_dam": Variant("rise_dam", target="rise", use_dam=True),
    # 2026-09-12 experiments on top of the deployed rise_rain configuration:
    # per-row sigma from quantile heads (the served sigma sits on the 0.15
    # floor at every P.1 horizon, so stage probabilities are constant-
    # calibrated), and a longer forecast-rain window for the 24 h horizon.
    "rise_rain_quantile": Variant(
        "rise_rain_quantile", target="rise", weighted=True, quantile=True, use_rain=True
    ),
    "rise_rain_quantile_uw": Variant(
        "rise_rain_quantile_uw", target="rise", quantile=True, use_rain=True
    ),
    "rise_rain_fc48": Variant(
        "rise_rain_fc48", target="rise", use_rain=True, use_fc48=True
    ),
    "rise_rain_qsigma": Variant(
        "rise_rain_qsigma", target="rise", use_rain=True, qsigma=True
    ),
 }
 # Dam variants are opt-in by name: they require dam columns that only exist
 # for features.DAM_STATIONS and only when the reservoir series loaded, and
-# the 2026-08-13 ablation concluded them a negative result.
+# the 2026-08-13 ablation concluded them a negative result. The 2026-09-12
-DEFAULT_VARIANTS = [k for k, v in VARIANTS.items() if not v.use_dam]
+# experiments are opt-in too (see their results in docs/FLOOD_FORECASTING.md).
 DEFAULT_VARIANTS = [
    k
    for k, v in VARIANTS.items()
    if not v.use_dam
    and not v.use_fc48
    and not v.qsigma
    and not (v.quantile and v.use_rain)
 ]
 def _find_events(observed: pd.Series, thr: float) -> List[dict]:
@@ -170,19 +226,21 @@ def _first_alert_lead(
    window = p.loc[start : crossing + pd.Timedelta(hours=24)]
    if len(window) < 2:
        return None
-    alert = (window >= ALERT_P) & (window.shift(-1) >= ALERT_P) & (
+    alert = (
        (window >= ALERT_P)
        & (window.shift(-1) >= ALERT_P)
        & (
            (window.index.to_series().shift(-1) - window.index.to_series())
            <= pd.Timedelta(hours=2)
        )
    )
    hits = window.index[alert.fillna(False)]
    if len(hits) == 0:
        return None
    return float((crossing - hits[0]).total_seconds() / 3600.0)
-def _false_alarm_episodes(
+def _false_alarm_episodes(p: pd.Series, observed: pd.Series, thr: float) -> int:
    p: pd.Series, observed: pd.Series, thr: float
 ) -> int:
    """Alert episodes with no observed >=thr within +/- FALSE_ALARM_GRACE_H."""
    alert_hours = p[p >= ALERT_P].index
    if len(alert_hours) == 0:
@@ -222,6 +280,12 @@ def evaluate_station(
    warn_thr, _ = features.get_thresholds(station)
    grid = features.make_hourly_grid(df_long)
    X_all = features.build_features(grid, station, rain=rain, dam=dam)
    if rain is not None:
        # forward sum over (t, t+48]; same construction as rain_fc24
        r = rain.reindex(X_all.index)
        X_all["rain_fc48"] = (
            r.shift(-1).iloc[::-1].rolling(48, min_periods=1).sum().iloc[::-1]
        )
    observed = grid.observed[(station, "water_level")]
    keep = X_all["obs_age_h"].notna()
@@ -247,7 +311,9 @@ def evaluate_station(
        tr = (X_all.index <= train_end) & y_abs.notna()
        te = (X_all.index >= test_lo) & (X_all.index <= test_hi)
        if tr.sum() < 5000 or te.sum() < 500:
-            logger.info(f"{station} {year}: skipped (train {tr.sum()}, test {te.sum()})")
+            logger.info(
                f"{station} {year}: skipped (train {tr.sum()}, test {te.sum()})"
            )
            continue
        X_tr, X_te = X_all.loc[tr], X_all.loc[te]
@@ -323,9 +389,7 @@ def evaluate_station(
                )
            fold["variants"][name] = {
                "mae": float(errors.mean()) if len(errors) else None,
-                "mae_above_2p5": (
+                "mae_above_2p5": (float(errors[high].mean()) if high.any() else None),
                    float(errors[high].mean()) if high.any() else None
                ),
                "brier_warn": brier,
                "events": event_rows,
                "false_alarm_episodes": _false_alarm_episodes(
@@ -346,7 +410,8 @@ def summarize(results: Dict) -> str:
    lines.append(header)
    for fold in results["folds"]:
        for name, m in fold["variants"].items():
-            events = "  ".join(
+            events = (
                "  ".join(
                    f"[{e['crossing'][:10]}: "
                    f"{'—' if e['lead_h'] is None else format(e['lead_h'], '+.0f')}h"
                    + (
@@ -356,7 +421,9 @@ def summarize(results: Dict) -> str:
                    )
                    + "]"
                    for e in m["events"]
-            ) or "no events"
+                )
                or "no events"
            )
            lines.append(
                f"{name:16} {fold['year']:>5} "
                f"{m['mae'] if m['mae'] is not None else float('nan'):6.3f} "
@@ -373,18 +440,31 @@ def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--stations", default="P.1")
    parser.add_argument("--db-url", default=None)
-    parser.add_argument("--variants", default=None,
+    parser.add_argument("--variants", default=None, help="comma list; default all")
                        help="comma list; default all")
    parser.add_argument("--out", default="models/eval_variants.json")
-    parser.add_argument("--no-rain", action="store_true",
+    parser.add_argument(
-                        help="skip loading the Open-Meteo rain series")
+        "--no-rain", action="store_true", help="skip loading the Open-Meteo rain series"
-    parser.add_argument("--no-dam", action="store_true",
+    )
-                        help="skip loading the Mae Ngat reservoir series")
+    parser.add_argument(
        "--no-dam",
        action="store_true",
        help="skip loading the Mae Ngat reservoir series",
    )
    parser.add_argument(
        "--from-cache",
        action="store_true",
        help="offline: read models/cache/ only (no DB, no API, "
        "no Open-Meteo refresh) -- reproducible reruns",
    )
    args = parser.parse_args(argv)
    logging.basicConfig(
        level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
    )
    if args.from_cache:
        df = data._read_cache(data.CACHE_DIR, None)
        logger.info(f"measurements from cache: {len(df)} rows")
    else:
        df = data.load_measurements(db_url=args.db_url)
    if df.empty:
        logger.error("no measurement data")
@@ -394,7 +474,9 @@ def main(argv=None) -> int:
    if not args.no_rain:
        from . import rain as rain_mod
-        rain_series = rain_mod.catchment_mean(rain_mod.load_history())
+        rain_series = rain_mod.catchment_mean(
            rain_mod.load_history(refresh=not args.from_cache)
        )
        if rain_series is None:
            logger.warning("rain history unavailable; rain features will be NaN")
        else:
@@ -404,7 +486,7 @@ def main(argv=None) -> int:
            )
    dam_frame = None
-    if not args.no_dam:
+    if not args.no_dam and not args.from_cache:
        from . import dam as dam_mod
        dam_frame = dam_mod.load_history(db_url=args.db_url)
@@ -37,9 +37,17 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
    "P.4A": (3.40, 3.90),
    "P.5": (4.55, 4.95),
    "P.67": (2.45, 2.90),
-    "P.75": (2.75, 3.50),
+    # P.75: 2024 (the only year with a full flood record, 191% capacity peak)
    # puts 75-85% at 3.45 m and 95-105% at 3.72 m; 2018/2022 agree within
    # 0.15 m. The 2026-08 value (2.75) alerted on 15 quiet-season hours.
    "P.75": (3.20, 3.65),
    "P.76": (5.35, 5.45),
-    "P.77": (2.85, 3.35),
+    # P.77: recalibrated 2026-09-12. The 2026-08 value (2.85) sat below the
    # gauge's own dry-season baseline (2.6-2.7 m at 8-14% capacity), so the
    # first ntfy cycle fired a "warning" at 22% capacity. Across 2018-2024,
    # 75-85% capacity reads 3.35-4.57 m and 95-105% 4.27-5.08 m; 2024 (the
    # best-sampled flood year) gives 4.57 / 5.08. Slightly conservative:
    "P.77": (4.30, 4.90),
    "P.81": (5.15, 6.30),
    # P.82 never reached 100% capacity in the record (max level 3.78, max 96.4%);
    # danger sits just below the observed maximum so the head can actually train.
@@ -0,0 +1,122 @@
 """Catchment-mean hourly rain from the HII/ThaiWater gauge network.
 Independent of Open-Meteo (src/ml/rain.py): those are model-analysis values,
 these are what the gauges measured. The `hii_rainfall` table has been filled
 by the hourly collector since 2026-08-11 and there is NO archive behind it
 (the api-v3 rain_24h_graph endpoint ignores its date range, see
 docs/DATA_SOURCES.md 2.1), so this series cannot yet be a training feature:
 every training row before 2026-08 would be NaN and HistGradientBoosting
 would learn nothing from the column. It becomes a candidate once a full
 monsoon season of gauge rows exists in the rolling-origin harness's test
 span -- the 2027 fold (train through 2027-04-30, test Jun-Nov 2027) is the
 first that could show anything.
 Until then it serves two purposes:
  * a live cross-check of the Open-Meteo catchment mean (/api/hii/rainfall
    already exposes the raw gauges; this gives the comparable aggregate);
  * accumulating the comparison so the eventual feature evaluation has a
    documented bias/variance relationship between the two sources.
 """
 import logging
 from typing import Optional, Sequence, Tuple
 import pandas as pd
 from .data import resolve_db_url
 logger = logging.getLogger(__name__)
 # Same footprint as rain.CATCHMENT_POINTS: the upper Ping above P.1. Gauges
 # inside this box are averaged; there are ~130 with recent data (DWR, FOP,
 # HII, RID, TMD), far denser than the five Open-Meteo points.
 CATCHMENT_BOX: Tuple[float, float, float, float] = (18.75, 19.60, 98.60, 99.30)
 # A gauge that reports the same rain_24h for many hours is stuck; drop hours
 # where fewer than this many gauges reported at all.
 MIN_GAUGES_PER_HOUR = 5
 def load_gauge_mean(
    db_url: Optional[str] = None,
    start: Optional[pd.Timestamp] = None,
    end: Optional[pd.Timestamp] = None,
    box: Sequence[float] = CATCHMENT_BOX,
    engine=None,
 ) -> Optional[pd.Series]:
    """Hourly catchment-mean rain_1h (mm) across HII gauges in `box`.
    Pass `engine` (the API's HII store engine) to reuse a pool; otherwise a
    connection is resolved from db_url / config. Returns None if the DB is
    unavailable or the table is empty. Hours with fewer than
    MIN_GAUGES_PER_HOUR reporting gauges are NaN.
    """
    if engine is None:
        resolved = resolve_db_url(db_url)
        if not resolved:
            return None
    lat_lo, lat_hi, lon_lo, lon_hi = box
    try:
        from sqlalchemy import create_engine, text
        query = (
            "SELECT m.timestamp, COUNT(m.rain_1h) AS n, AVG(m.rain_1h) AS rain_1h "
            "FROM hii_rainfall m JOIN hii_rain_stations s ON s.id = m.station_id "
            "WHERE s.latitude BETWEEN :lat_lo AND :lat_hi "
            "AND s.longitude BETWEEN :lon_lo AND :lon_hi "
            "AND m.rain_1h IS NOT NULL"
        )
        params = {
            "lat_lo": lat_lo,
            "lat_hi": lat_hi,
            "lon_lo": lon_lo,
            "lon_hi": lon_hi,
        }
        if start is not None:
            query += " AND m.timestamp >= :start"
            params["start"] = pd.Timestamp(start).to_pydatetime()
        if end is not None:
            query += " AND m.timestamp <= :end"
            params["end"] = pd.Timestamp(end).to_pydatetime()
        query += " GROUP BY m.timestamp ORDER BY m.timestamp"
        if engine is None:
            engine = create_engine(resolved, pool_pre_ping=True)
        with engine.connect() as conn:
            frame = pd.read_sql(text(query), conn, params=params)
    except Exception as error:
        logger.warning(f"HII gauge rain load failed: {error}")
        return None
    if frame.empty:
        return None
    frame["timestamp"] = pd.to_datetime(frame["timestamp"]).dt.floor("h")
    frame = frame.groupby("timestamp").agg(n=("n", "sum"), rain_1h=("rain_1h", "mean"))
    series = pd.to_numeric(frame["rain_1h"], errors="coerce")
    series[frame["n"] < MIN_GAUGES_PER_HOUR] = float("nan")
    series.name = "hii_gauge_mean"
    return series
 def compare_with_openmeteo(
    gauge: pd.Series, openmeteo: pd.Series, window_h: int = 24
 ) -> dict:
    """Bias/correlation of Open-Meteo against the gauges over the overlap.
    Both are summed over trailing `window_h` so single-hour timing offsets
    (gauges report at :00, the model's hour is an interval) do not dominate.
    """
    joined = pd.concat({"gauge": gauge, "openmeteo": openmeteo}, axis=1).dropna()
    if joined.empty:
        return {"overlap_hours": 0}
    g = joined["gauge"].rolling(window_h, min_periods=window_h).sum()
    o = joined["openmeteo"].rolling(window_h, min_periods=window_h).sum()
    both = pd.concat({"g": g, "o": o}, axis=1).dropna()
    if both.empty:
        return {"overlap_hours": int(len(joined))}
    return {
        "overlap_hours": int(len(joined)),
        "window_h": window_h,
        "gauge_mean_mm": float(both["g"].mean()),
        "openmeteo_mean_mm": float(both["o"].mean()),
        "bias_mm": float((both["o"] - both["g"]).mean()),
        "mae_mm": float((both["o"] - both["g"]).abs().mean()),
        "corr": float(both["g"].corr(both["o"])),
    }
@@ -182,14 +182,18 @@ def _model_forecast(
        )
        p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
        if warn_head is not None:
-            p_warning = max(p_warning, float(warn_head.predict_proba(feature_row)[0][1]))
+            p_warning = max(
                p_warning, float(warn_head.predict_proba(feature_row)[0][1])
            )
        danger_head = (
            None if thresholds_stale else bundle["heads"].get(f"danger_{horizon_h}")
        )
        p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
        if danger_head is not None:
-            p_danger = max(p_danger, float(danger_head.predict_proba(feature_row)[0][1]))
+            p_danger = max(
                p_danger, float(danger_head.predict_proba(feature_row)[0][1])
            )
        p_warning = _clip_probability(p_warning)
        p_danger = min(_clip_probability(p_danger), p_warning)
@@ -400,6 +404,4 @@ def get_latest_forecasts(
        logger.warning("dam state unavailable; dam features will be NaN")
        dam = pd.DataFrame()
-    return get_forecasts(
+    return get_forecasts(readings_by_station, models_dir=models_dir, rain=rain, dam=dam)
        readings_by_station, models_dir=models_dir, rain=rain, dam=dam
    )
@@ -121,12 +121,8 @@ def load_history(
    cursor = fetch_from.date()
    try:
        while cursor <= end:
-            chunk_end = min(
+            chunk_end = min(datetime.date(cursor.year, 12, 31), end)
-                datetime.date(cursor.year, 12, 31), end
+            chunks.append(fetch_history(cursor.isoformat(), chunk_end.isoformat()))
            )
            chunks.append(
                fetch_history(cursor.isoformat(), chunk_end.isoformat())
            )
            cursor = datetime.date(cursor.year + 1, 1, 1)
    except Exception as error:
        logger.warning(f"Open-Meteo history fetch failed: {error}")
@@ -204,8 +200,7 @@ def save_to_db(df: pd.DataFrame, engine, db_type: str) -> int:
    cols = ["timestamp"] + point_cols + ["catchment_mean"]
    placeholders = ", ".join(f":{c}" for c in cols)
    updates = ", ".join(
-        f"{c} = "
+        f"{c} = " + (f"VALUES({c})" if db_type == "mysql" else f"EXCLUDED.{c}")
        + (f"VALUES({c})" if db_type == "mysql" else f"EXCLUDED.{c}")
        for c in cols[1:]
    )
    if db_type == "mysql":
@@ -0,0 +1,170 @@
 """Live forecast skill: what the deployed model said versus what the river did.
 Every hour the precompute stores the issued 24 h forecast (forecast_history);
 water_measurements holds what actually happened. Joining the two gives a
 verification that needs no retraining and answers the question the dashboard
 is asked most: "is the model getting better?" — per model version, on the
 hours that version was actually serving.
 Metrics per version and horizon:
  n           verified forecasts (issued, and the horizon has since elapsed)
  mae         |predicted_max - observed_max| over the horizon window, metres
  bias        mean(predicted - observed): >0 over-predicts the peak
  persistence MAE of the trivial "peak = current level" forecast on the
              same rows; a model is only useful if it beats this
  skill       1 - mae/persistence  (0 = no better than persistence, 1 = perfect)
  above_2m    same MAE restricted to rows where the observed peak >= 2 m,
              i.e. the flood-relevant regime
 Only the P.1 gauge is verified by default: it is the one the city threshold
 is keyed to, and one station keeps the query cheap enough to run on request.
 """
 import datetime
 import logging
 from typing import Dict, List, Optional
 logger = logging.getLogger(__name__)
 DEFAULT_STATION = "P.1"
 DEFAULT_HORIZON = 24
 MIN_VERIFIED = 24  # fewer than a day of verified hours is not a number
 def _sql_for(db_type: str) -> str:
    """Join each issued forecast to the observed max over (as_of, as_of + h]."""
    if db_type == "postgresql":
        window_end = "f.as_of + (f.horizon_hours || ' hours')::interval"
    elif db_type == "mysql":
        window_end = "DATE_ADD(f.as_of, INTERVAL f.horizon_hours HOUR)"
    else:  # sqlite
        window_end = "datetime(f.as_of, '+' || f.horizon_hours || ' hours')"
    return f"""
        SELECT f.as_of, f.model_version, f.predicted_max_level, f.current_level,
               (SELECT MAX(m.water_level) FROM water_measurements m
                 JOIN stations s ON s.id = m.station_id
                WHERE s.station_code = f.station_code
                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_max,
               (SELECT COUNT(m.water_level) FROM water_measurements m
                 JOIN stations s ON s.id = m.station_id
                WHERE s.station_code = f.station_code
                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_n
        FROM forecast_history f
        WHERE f.station_code = :code AND f.horizon_hours = :horizon
          AND f.source = 'model' AND f.predicted_max_level IS NOT NULL
          AND f.as_of <= :verifiable_before
        ORDER BY f.as_of
    """
 def compute_skill(
    engine,
    db_type: str,
    station_code: str = DEFAULT_STATION,
    horizon_hours: int = DEFAULT_HORIZON,
    now: Optional[datetime.datetime] = None,
 ) -> Dict:
    """Per-model-version verification of issued forecasts against observations."""
    from sqlalchemy import text
    now = now or datetime.datetime.now()
    verifiable_before = now - datetime.timedelta(hours=horizon_hours)
    with engine.connect() as conn:
        rows = [
            dict(r._mapping)
            for r in conn.execute(
                text(_sql_for(db_type)),
                {
                    "code": station_code,
                    "horizon": horizon_hours,
                    "verifiable_before": verifiable_before,
                },
            )
        ]
    def _ts(value):
        # sqlite hands back strings; postgres/mysql give datetimes
        if isinstance(value, datetime.datetime):
            return value
        return datetime.datetime.fromisoformat(str(value).replace(" ", "T"))
    by_version: Dict[str, List[dict]] = {}
    for r in rows:
        r["as_of"] = _ts(r["as_of"])
        # need most of the window observed, or the "max" is not the peak
        if r["observed_max"] is None or (r["observed_n"] or 0) < horizon_hours * 0.75:
            continue
        by_version.setdefault(r["model_version"] or "unknown", []).append(r)
    versions = []
    for version, vrows in by_version.items():
        pred = [float(r["predicted_max_level"]) for r in vrows]
        obs = [float(r["observed_max"]) for r in vrows]
        cur = [
            float(r["current_level"]) if r["current_level"] is not None else None
            for r in vrows
        ]
        err = [p - o for p, o in zip(pred, obs)]
        mae = sum(abs(e) for e in err) / len(err)
        bias = sum(err) / len(err)
        pers_rows = [(c, o) for c, o in zip(cur, obs) if c is not None]
        persistence = (
            sum(abs(c - o) for c, o in pers_rows) / len(pers_rows)
            if pers_rows
            else None
        )
        high = [(p, o) for p, o in zip(pred, obs) if o >= 2.0]
        versions.append(
            {
                "model_version": version,
                "first_issued": min(r["as_of"] for r in vrows).isoformat(),
                "last_issued": max(r["as_of"] for r in vrows).isoformat(),
                "n": len(vrows),
                "mae_m": round(mae, 3),
                "bias_m": round(bias, 3),
                "persistence_mae_m": (
                    None if persistence is None else round(persistence, 3)
                ),
                "skill": (
                    None if not persistence else round(1.0 - mae / persistence, 3)
                ),
                "above_2m_n": len(high),
                "above_2m_mae_m": (
                    round(sum(abs(p - o) for p, o in high) / len(high), 3)
                    if high
                    else None
                ),
                "enough_data": len(vrows) >= MIN_VERIFIED,
            }
        )
    versions.sort(key=lambda v: v["first_issued"])
    # Headline: current version vs the previous one that had enough data
    current = versions[-1] if versions else None
    previous = (
        next((v for v in reversed(versions[:-1]) if v["enough_data"]), None)
        if versions
        else None
    )
    trend = None
    if current and previous and current["enough_data"]:
        trend = {
            "previous_version": previous["model_version"],
            "mae_delta_m": round(current["mae_m"] - previous["mae_m"], 3),
            "skill_delta": (
                None
                if current["skill"] is None or previous["skill"] is None
                else round(current["skill"] - previous["skill"], 3)
            ),
            "better": current["mae_m"] < previous["mae_m"],
        }
    return {
        "station_code": station_code,
        "horizon_hours": horizon_hours,
        "verified_until": verifiable_before.isoformat(),
        "min_verified": MIN_VERIFIED,
        "versions": versions,
        "current": current,
        "trend": trend,
    }
@@ -45,6 +45,15 @@ MIN_SIGMA = 0.15
 MIN_ROWS_TO_TRAIN = 200
 MIN_ROWS_FOR_HEAD = 50
 class RainUnavailableError(RuntimeError):
    """Raised when a rain-enabled training run cannot obtain the rain series.
    Training would otherwise fall through to gauge-only (v2) bundles and
    overwrite the deployed v3 artifacts without anyone noticing.
    """
 HGB_PARAMS = {
    "max_iter": 300,
    "learning_rate": 0.06,
@@ -311,9 +320,9 @@ def train_station(
                    skipped_heads,
                )
            else:
-                skipped_heads[
+                skipped_heads[head_key] = (
-                    head_key
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
            heads[head_key] = clf
            if not skip_eval:
@@ -408,9 +417,9 @@ def train_station(
                if clf is not None:
                    skipped_heads.pop(head_key, None)
            else:
-                skipped_heads[
+                skipped_heads[head_key] = (
-                    head_key
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
                final_heads[head_key] = None
    # v4 = + Mae Ngat dam features; v3 = rise + rain; v2 = rise target only
@@ -456,8 +465,13 @@ def train_all(
    models_dir = Path(models_dir)
    models_dir.mkdir(parents=True, exist_ok=True)
-    # Catchment rain (Open-Meteo archive, 2021+). Optional: without it the
+    # Catchment rain (Open-Meteo archive, 2021+). A rain-less run produces v2
-    # models train as v2 (no rain columns) and still serve correctly.
+    # bundles that serve fine but have measurably less flood lead (the 2024
    # record flood: 13 h early with rain vs 18 h late without). The 2026-09-01
    # server retrain hit exactly that -- the archive fetch failed on a checkout
    # with no models/cache/ and the run quietly wrote v2 over v3. So the
    # downgrade is now an error unless the caller opts out with use_rain=False
    # (the --no-rain flag), which is the only way to get v2 deliberately.
    rain_series = None
    if use_rain:
        try:
@@ -465,7 +479,19 @@ def train_all(
            rain_series = rain_mod.catchment_mean(rain_mod.load_history())
        except Exception as error:
-            logger.warning(f"rain history unavailable, training without it: {error}")
+            raise RainUnavailableError(
                f"rain history unavailable ({error}); refusing to silently "
                "downgrade to v2 bundles -- fix Open-Meteo access or restore "
                "models/cache/rain_openmeteo.csv.gz, or pass --no-rain to "
                "train gauge-only bundles on purpose"
            ) from error
        if rain_series is None:
            raise RainUnavailableError(
                "rain history unavailable (Open-Meteo archive unreachable and "
                "no models/cache/rain_openmeteo.csv.gz); refusing to silently "
                "downgrade to v2 bundles -- fix access, restore the cache file, "
                "or pass --no-rain to train gauge-only bundles on purpose"
            )
    if rain_series is not None:
        logger.info(
            f"rain series: {rain_series.index.min()} .. {rain_series.index.max()}"
@@ -493,9 +519,7 @@ def train_all(
                "training v3-style bundles WITHOUT dam features"
            )
    if dam_frame is not None:
-        logger.info(
+        logger.info(f"dam series: {dam_frame.index.min()} .. {dam_frame.index.max()}")
            f"dam series: {dam_frame.index.min()} .. {dam_frame.index.max()}"
        )
    # Run-level version: v4 only if some requested station actually receives
    # dam columns (they are gated to DAM_STATIONS; per-bundle versions are
@@ -584,7 +608,9 @@ def main(argv: Optional[List[str]] = None) -> None:
    parser.add_argument(
        "--no-rain",
        action="store_true",
-        help="train without the Open-Meteo rain features (v2-style bundles)",
+        help="DELIBERATELY train without the Open-Meteo rain features "
        "(v2-style bundles). Without this flag a missing rain series aborts "
        "the run instead of quietly downgrading the deployed model",
    )
    parser.add_argument(
        "--dam",
@@ -627,9 +653,21 @@ def main(argv: Optional[List[str]] = None) -> None:
        1 for s in metrics_payload["stations"].values() if s["status"] == "trained"
    )
    logger.info(
-        f"Done: {trained}/{len(stations)} stations trained. metrics.json written to {args.models_dir}"
+        f"Done: {trained}/{len(stations)} stations trained "
        f"({metrics_payload['model_version']}). "
        f"metrics.json written to {args.models_dir}"
    )
-if __name__ == "__main__":
+def cli() -> int:
    """Console entry: RainUnavailableError becomes a one-line error, exit 2."""
    try:
        main()
    except RainUnavailableError as error:
        logger.error(str(error))
        return 2
    return 0
 if __name__ == "__main__":
    raise SystemExit(cli())
@@ -0,0 +1,454 @@
 """Public flood notifications over ntfy.
 Runs once per collection cycle inside the API process (leader only), right
 after the forecast precompute, so it sees the same readings and forecasts the
 dashboard shows. Publishes to a self-hosted ntfy server; anyone subscribes to
 a topic from the free app or a browser, no account needed.
 Topics (all under one configurable prefix, default "ping"):
  {prefix}-{station}-warning     observed level crossed the station's warning threshold
  {prefix}-{station}-danger      observed level crossed the danger threshold
  {prefix}-warning               any station crossed warning (basin-wide digest)
  {prefix}-danger                any station crossed danger
  {prefix}-p1-outlook            model early warning for Chiang Mai city: P.1's 24 h
                                 warning probability crossed the alert level (opt-in;
                                 the forecast is experimental and says so)
  {prefix}-status                feed/monitor health: data stale, recovered
 Each notification is a TRANSITION, not a state: crossing UP into a level sends
 one message; dropping back below (with hysteresis) sends an all-clear. While
 the river sits above a threshold nothing is repeated, so a subscriber in a
 flood gets a handful of messages, not one an hour. The per-topic state is
 persisted (notification_state table) so a restart never re-sends.
 Everything is fail-safe: ntfy unreachable, table missing, malformed
 reading -> a logged warning, never an exception into the collection loop.
 """
 import datetime
 import logging
 from dataclasses import dataclass
 from typing import Dict, Iterable, List, Optional
 import requests
 from .ml import features
 logger = logging.getLogger(__name__)
 # Hysteresis: an all-clear needs the level this far BELOW the threshold, so a
 # river bobbing around 3.70 m does not toggle warning/clear every hour.
 CLEAR_MARGIN_M = 0.10
 # Capacity guard. The level thresholds in features.THRESHOLDS were calibrated
 # from RID's discharge_percent (% of channel capacity); if RID re-rates a
 # gauge or moves its datum, the level crosses while capacity says the channel
 # is nearly empty (P.77, 2026-09: 3.0 m "warning" at 22 %). A crossing is
 # only announced when the reported capacity agrees that the river is high.
 # P.1 is exempt: its stages come from the municipal inundation map, not from
 # capacity. Readings without a capacity figure fall back to level only.
 CAPACITY_GUARD_MIN_PCT = 60.0
 CAPACITY_GUARD_EXEMPT = {"P.1"}
 # Outlook alert fires when p_warning(24h) rises through ON, clears below OFF.
 OUTLOOK_ON = 0.50
 OUTLOOK_OFF = 0.25
 # Below this the outlook is not announced at all (avoid "5 % chance" noise).
 OUTLOOK_HORIZON = 24
 STATION_NAMES: Dict[str, str] = {
    "P.1": "Nawarat Bridge, Chiang Mai city",
    "P.103": "Ring Road Bridge 3, Chiang Mai",
    "P.67": "Ban Tae (Mae Taeng)",
    "P.21": "Ban Rim Tai (Mae Rim)",
    "P.75": "Ban Chai Lat",
    "P.92": "Ban Muang Aut",
    "P.20": "Ban Chiang Dao",
    "P.4A": "Ban Mae Taeng",
    "P.5": "Tha Nang Bridge (downstream)",
    "P.81": "Ban Pong (downstream)",
    "P.82": "Ban Sob Win",
    "P.84": "Ban Panton",
    "P.87": "Ban Pa Sang",
    "P.77": "Ban Sop Mae Sapuat",
    "P.85": "Ban Lai Kaew",
    "P.76": "Ban Mae I Hai",
 }
 def _slug(code: str) -> str:
    return code.lower().replace(".", "")
@dataclass
 class Notification:
    topic: str
    title: str
    message: str
    priority: int = 3  # ntfy: 1 min .. 5 max
    tags: Optional[List[str]] = None
    click: Optional[str] = None
 class NtfyPublisher:
    def __init__(
        self,
        server: str,
        prefix: str = "ping",
        token: Optional[str] = None,
        dashboard_url: str = "https://water.buildfor.life/",
        timeout: int = 10,
    ):
        self.server = server.rstrip("/")
        self.prefix = prefix
        self.token = token
        self.dashboard_url = dashboard_url
        self.timeout = timeout
    def topic(self, *parts: str) -> str:
        return "-".join([self.prefix, *parts])
    def publish(self, n: Notification) -> bool:
        headers = {
            "Title": n.title,
            "Priority": str(n.priority),
            "Click": n.click or self.dashboard_url,
            "Actions": f"view, Open dashboard, {n.click or self.dashboard_url}",
        }
        if n.tags:
            headers["Tags"] = ",".join(n.tags)
        if self.token:
            headers["Authorization"] = f"Bearer {self.token}"
        try:
            r = requests.post(
                f"{self.server}/{n.topic}",
                data=n.message.encode("utf-8"),
                headers=headers,
                timeout=self.timeout,
            )
            if r.status_code >= 300:
                logger.warning(f"ntfy {n.topic}: HTTP {r.status_code} {r.text[:120]}")
                return False
            return True
        except Exception as error:
            logger.warning(f"ntfy {n.topic}: {error}")
            return False
 class NotificationState:
    """Per-key last-sent state, in the monitor's own SQL database."""
    def __init__(self, engine, db_type: str):
        self.engine = engine
        self.db_type = db_type
        self._ensure()
    def _ensure(self) -> None:
        from sqlalchemy import text
        ddl = (
            "CREATE TABLE IF NOT EXISTS notification_state ("
            "key VARCHAR(64) PRIMARY KEY, state VARCHAR(16) NOT NULL, "
            "value NUMERIC(8,3), updated_at TIMESTAMP NOT NULL)"
        )
        try:
            with self.engine.begin() as conn:
                conn.execute(text(ddl))
        except Exception as error:
            # Postgres: two sessions racing CREATE TABLE IF NOT EXISTS can
            # both pass the existence check; the loser fails with a unique
            # violation on pg_type. The table exists either way; verify.
            with self.engine.connect() as conn:
                conn.execute(text("SELECT 1 FROM notification_state WHERE 1=0"))
            logger.debug(f"notification_state DDL raced, table present: {error}")
    def get(self, key: str) -> Optional[str]:
        from sqlalchemy import text
        with self.engine.connect() as conn:
            row = conn.execute(
                text("SELECT state FROM notification_state WHERE key = :k"), {"k": key}
            ).fetchone()
        return row[0] if row else None
    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
        from sqlalchemy import text
        now = datetime.datetime.now()
        with self.engine.begin() as conn:
            if self.db_type == "mysql":
                sql = (
                    "INSERT INTO notification_state (key, state, value, updated_at) "
                    "VALUES (:k, :s, :v, :t) ON DUPLICATE KEY UPDATE "
                    "state = VALUES(state), value = VALUES(value), updated_at = VALUES(updated_at)"
                )
            else:
                sql = (
                    "INSERT INTO notification_state (key, state, value, updated_at) "
                    "VALUES (:k, :s, :v, :t) ON CONFLICT (key) DO UPDATE SET "
                    "state = EXCLUDED.state, value = EXCLUDED.value, updated_at = EXCLUDED.updated_at"
                )
            conn.execute(text(sql), {"k": key, "s": state, "v": value, "t": now})
 class InMemoryState(NotificationState):
    """For tests and when no SQL engine is available (loses state on restart)."""
    def __init__(self):  # noqa: D107 - intentionally skips the SQL parent
        self._d: Dict[str, str] = {}
    def get(self, key: str) -> Optional[str]:
        return self._d.get(key)
    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
        self._d[key] = state
 def _level_state(level: float, warn: float, danger: float, prev: Optional[str]) -> str:
    """'clear' | 'warning' | 'danger', with hysteresis on the way down."""
    if level >= danger:
        return "danger"
    if level >= warn:
        # from danger: stay 'danger' until below danger - margin
        if prev == "danger" and level >= danger - CLEAR_MARGIN_M:
            return "danger"
        return "warning"
    if prev in ("warning", "danger") and level >= warn - CLEAR_MARGIN_M:
        return "warning"
    return "clear"
 def evaluate(
    readings: Iterable[dict],
    forecasts: Iterable[dict],
    state: NotificationState,
    publisher: NtfyPublisher,
    stale_after_h: float = 3.0,
    now: Optional[datetime.datetime] = None,
 ) -> List[Notification]:
    """Compare current readings/forecasts with last-sent state; publish transitions.
    readings: rows with station_code, water_level, timestamp (latest per station)
    forecasts: /forecast rows (station_code, horizon_hours, p_warning, predicted_max_level)
    Returns the notifications that were published (for logs/tests).
    """
    now = now or datetime.datetime.now()
    sent: List[Notification] = []
    def emit(n: Notification) -> bool:
        ok = publisher.publish(n)
        if ok:
            sent.append(n)
        return ok
    # ---- observed levels, per station, plus basin-wide fan-out
    basin_changes: Dict[str, List[str]] = {"warning": [], "danger": [], "clear": []}
    latest_ts: Optional[datetime.datetime] = None
    for r in readings:
        code = r.get("station_code")
        level = r.get("water_level")
        if not code or level is None:
            continue
        try:
            level = float(level)
        except (TypeError, ValueError):
            continue
        ts = r.get("timestamp")
        if isinstance(ts, str):
            try:
                ts = datetime.datetime.fromisoformat(ts)
            except ValueError:
                ts = None
        if isinstance(ts, datetime.datetime) and (latest_ts is None or ts > latest_ts):
            latest_ts = ts
        warn, danger = features.get_thresholds(code)
        key = f"level:{code}"
        prev = state.get(key) or "clear"
        cur = _level_state(level, warn, danger, prev)
        pct = r.get("discharge_percent")
        if (
            cur != "clear"
            and prev == "clear"
            and code not in CAPACITY_GUARD_EXEMPT
            and pct is not None
        ):
            try:
                if float(pct) < CAPACITY_GUARD_MIN_PCT:
                    logger.info(
                        f"{code}: level {level:.2f} m >= {warn:.2f} but only "
                        f"{float(pct):.0f}% capacity; threshold looks stale, not alerting"
                    )
                    continue
            except (TypeError, ValueError):
                pass
        if cur == prev:
            continue
        name = STATION_NAMES.get(code, code)
        slug = _slug(code)
        when = (
            ts.strftime("%d %b %H:%M") if isinstance(ts, datetime.datetime) else "now"
        )
        if cur == "danger":
            ok = emit(
                Notification(
                    publisher.topic(slug, "danger"),
                    f"DANGER level at {code}",
                    f"{name}: {level:.2f} m at {when}, above the danger level of {danger:.2f} m.",
                    priority=5,
                    tags=["rotating_light", code],
                )
            )
            basin_changes["danger"].append(f"{code} {level:.2f} m")
        elif cur == "warning":
            if prev == "danger":
                ok = emit(
                    Notification(
                        publisher.topic(slug, "danger"),
                        f"{code} back below danger level",
                        f"{name}: {level:.2f} m at {when}; still above the warning level of {warn:.2f} m.",
                        priority=3,
                        tags=["arrow_down", code],
                    )
                )
                basin_changes["clear"].append(f"{code} below danger ({level:.2f} m)")
            else:
                ok = emit(
                    Notification(
                        publisher.topic(slug, "warning"),
                        f"Warning level at {code}",
                        f"{name}: {level:.2f} m at {when}, above the warning level of {warn:.2f} m.",
                        priority=4,
                        tags=["warning", code],
                    )
                )
                basin_changes["warning"].append(f"{code} {level:.2f} m")
        else:  # clear
            ok = emit(
                Notification(
                    publisher.topic(slug, "warning"),
                    f"{code} back to normal",
                    f"{name}: {level:.2f} m at {when}, below the warning level of {warn:.2f} m.",
                    priority=2,
                    tags=["white_check_mark", code],
                )
            )
            basin_changes["clear"].append(f"{code} normal ({level:.2f} m)")
        # Only remember the transition once it was actually delivered: if ntfy
        # was down, the next cycle retries instead of silently swallowing a
        # flood crossing.
        if ok:
            state.set(key, cur, level)
    if basin_changes["danger"]:
        emit(
            Notification(
                publisher.topic("danger"),
                "Ping River: danger level reached",
                "; ".join(basin_changes["danger"]),
                priority=5,
                tags=["rotating_light"],
            )
        )
    if basin_changes["warning"]:
        emit(
            Notification(
                publisher.topic("warning"),
                "Ping River: warning level reached",
                "; ".join(basin_changes["warning"]),
                priority=4,
                tags=["warning"],
            )
        )
    if basin_changes["clear"]:
        emit(
            Notification(
                publisher.topic("warning"),
                "Ping River: levels falling",
                "; ".join(basin_changes["clear"]),
                priority=2,
                tags=["white_check_mark"],
            )
        )
    # ---- model outlook for the city gauge (opt-in topic, experimental)
    p1 = next(
        (
            f
            for f in forecasts
            if f.get("station_code") == "P.1"
            and f.get("horizon_hours") == OUTLOOK_HORIZON
            and f.get("source") == "model"
        ),
        None,
    )
    if p1 and p1.get("p_warning") is not None:
        p = float(p1["p_warning"])
        key = "outlook:P.1"
        prev = state.get(key) or "off"
        cur = (
            "on" if (p >= OUTLOOK_ON or (prev == "on" and p >= OUTLOOK_OFF)) else "off"
        )
        if cur != prev:
            peak = p1.get("predicted_max_level")
            warn, _ = features.get_thresholds("P.1")
            if cur == "on":
                ok = emit(
                    Notification(
                        publisher.topic("p1-outlook"),
                        "Early warning: Chiang Mai flood risk rising",
                        f"The forecast model gives a {p * 100:.0f}% chance that Nawarat Bridge (P.1) "
                        f"reaches {warn:.2f} m within 24 h"
                        + (
                            f" (expected peak {float(peak):.2f} m)"
                            if peak is not None
                            else ""
                        )
                        + ". Experimental model output, not an official warning; "
                        "follow ThaiWater/TMD for official alerts.",
                        priority=4,
                        tags=["crystal_ball"],
                    )
                )
            else:
                ok = emit(
                    Notification(
                        publisher.topic("p1-outlook"),
                        "Chiang Mai flood risk easing",
                        f"The model's 24 h probability of reaching {warn:.2f} m at P.1 has dropped to {p * 100:.0f}%.",
                        priority=2,
                        tags=["crystal_ball"],
                    )
                )
            if ok:
                state.set(key, cur, p)
    # ---- feed health
    if latest_ts is not None:
        age_h = (now - latest_ts).total_seconds() / 3600.0
        key = "feed"
        prev = state.get(key) or "ok"
        cur = "stale" if age_h >= stale_after_h else "ok"
        if cur != prev:
            if cur == "stale":
                ok = emit(
                    Notification(
                        publisher.topic("status"),
                        "Ping River monitor: gauge feed stale",
                        f"No new readings for {age_h:.0f} h (last {latest_ts:%d %b %H:%M}). "
                        "Levels and forecasts on the dashboard are not current.",
                        priority=3,
                        tags=["hourglass"],
                    )
                )
            else:
                ok = emit(
                    Notification(
                        publisher.topic("status"),
                        "Ping River monitor: feed recovered",
                        f"Readings are current again (latest {latest_ts:%d %b %H:%M}).",
                        priority=2,
                        tags=["white_check_mark"],
                    )
                )
            if ok:
                state.set(key, cur, age_h)
    return sent
@@ -51,8 +51,7 @@ class PostgresHistory:
        if start >= end:
            raise ValueError("start must be before end")
-        query = text(
+        query = text("""
            """
            SELECT m.timestamp, s.station_code, m.water_level,
                   m.discharge, m.discharge_percent
            FROM water_measurements m
@@ -62,8 +61,7 @@ class PostgresHistory:
              AND m.timestamp <= :end_time
            ORDER BY m.timestamp ASC
            LIMIT :limit
-            """
+            """)
        )
        with self.engine.connect() as connection:
            rows = connection.execute(
                query,
@@ -91,9 +89,9 @@ class PostgresHistory:
                        "station_code": station_code,
                        "water_level": water_level,
                        "discharge": discharge,
-                        "discharge_percent": float(row[4])
+                        "discharge_percent": (
-                        if row[4] is not None
+                            float(row[4]) if row[4] is not None else None
-                        else None,
+                        ),
                    }
                )
            return result
@@ -173,8 +173,10 @@ class RequestTracker:
                "failed_requests": self.failed_requests,
                "success_rate": self.successful_requests / self.total_requests,
                "average_response_time": self.total_response_time / self.total_requests,
-                "last_request_time": self.last_request_time.isoformat()
+                "last_request_time": (
                    self.last_request_time.isoformat()
                    if self.last_request_time
-                else None,
+                    else None
                ),
                "error_breakdown": dict(self.error_count_by_type),
            }
@@ -323,9 +323,11 @@ class RidReservoirStore:
            if not preserve_cols:
                return f"INSERT OR REPLACE INTO {table} ({col_list}) VALUES ({params})"
            updates = ", ".join(
                (
                    f"{c} = COALESCE(excluded.{c}, {table}.{c})"
                    if c in preserve_cols
                    else f"{c} = excluded.{c}"
                )
                for c in value_cols
            )
            return (
@@ -334,9 +336,11 @@ class RidReservoirStore:
            )
        if self.db_type == "postgresql":
            updates = ", ".join(
                (
                    f"{c} = COALESCE(EXCLUDED.{c}, {table}.{c})"
                    if c in preserve_cols
                    else f"{c} = EXCLUDED.{c}"
                )
                for c in value_cols
            )
            return (
@@ -344,9 +348,11 @@ class RidReservoirStore:
                f"ON CONFLICT ({conflict}) DO UPDATE SET {updates}"
            )
        updates = ", ".join(
            (
                f"{c} = COALESCE(VALUES({c}), {c})"
                if c in preserve_cols
                else f"{c} = VALUES({c})"
            )
            for c in value_cols
        )
        return (
@@ -402,9 +408,7 @@ class RidReservoirStore:
            measure_row = {
                c: _bounded(record.get(c), _MEASURE_BOUNDS[c]) for c in measure_cols
            }
-            measure_row.update(
+            measure_row.update({"dam_id": record["dam_id"], "date": record["date"]})
                {"dam_id": record["dam_id"], "date": record["date"]}
            )
            measurements.append(measure_row)
        try:
            with self.engine.begin() as conn:
@@ -495,9 +499,7 @@ def backfill(
    if not store.engine and not store.connect():
        logger.error("backfill aborted: database connection failed")
        return 0
-    span = [
+    span = [start + datetime.timedelta(days=i) for i in range((end - start).days + 1)]
        start + datetime.timedelta(days=i) for i in range((end - start).days + 1)
    ]
    present = store.present_dates(start, end)
    targets = [d for d in span if d not in present]
    logger.info(
@@ -33,6 +33,57 @@
            --red: #cc4b37;
            --border: #dce7e3;
            --shadow: 0 16px 40px rgba(23, 57, 67, .10);
            --surface: #ffffff;          /* buttons, inputs, overlays */
            --surface-2: #f1f7f5;        /* hover rows, popup metrics */
            --surface-3: #f7fbfa;        /* outlook box */
            --overlay: rgba(255,255,255,.93);
            --overlay-border: rgba(207,224,218,.9);
            --map-bg: #dcebea;
            --glow: rgba(56, 180, 213, .12);
            --map-filter: none;
            --chart-grid: rgba(19,43,53,.08);
            --mint-ink: #146644;
            --mint-border: #b6dfc9;
            --ok-bg: #e2f3ea; --ok-ink: #0c5138; --ok-border: #bfe3d2;
            --watch-bg: #fdf1dc; --watch-ink: #6b4a05; --watch-border: #f0d9a8;
            --danger-bg: #fbe3de; --danger-ink: #7c1d10; --danger-border: #f2c0b6;
            --demo-bg: #fdeeda; --demo-ink: #9a6200;
            color-scheme: light;
        }
        /* Dark theme: same hues, inverted lightness. Flow/rain/marker colours
           on the map are data encodings and stay identical in both themes. */
        [data-theme="dark"] {
            --ink: #e6eef1;
            --muted: #97a9b0;
            --paper: #0f1a1f;
            --card: #16252c;
            --river: #3fb3d8;
            --river-light: #5ccbe8;
            --mint: #143b2c;
            --green: #3fb07f;
            --amber: #e6a83a;
            --red: #e46a55;
            --border: #27393f;
            --shadow: 0 16px 40px rgba(0, 0, 0, .45);
            --surface: #1c2c33;
            --surface-2: #213238;
            --surface-3: #1a2a30;
            --overlay: rgba(22,37,44,.92);
            --overlay-border: rgba(64,86,94,.9);
            --map-bg: #1a262b;
            --glow: rgba(63, 179, 216, .10);
            /* OSM tiles are light; invert + rotate keeps roads/labels legible
               while the water/river overlays (drawn in SVG, unfiltered) keep
               their real colours. */
            --map-filter: invert(1) hue-rotate(180deg) brightness(.92) contrast(.9) saturate(.75);
            --chart-grid: rgba(230,238,241,.10);
            --mint-ink: #7fd7ac;
            --mint-border: #2a5c45;
            --ok-bg: #143b2c; --ok-ink: #9fe3c3; --ok-border: #2a6b4d;
            --watch-bg: #3d2f0d; --watch-ink: #f3d489; --watch-border: #6b4f14;
            --danger-bg: #421c15; --danger-ink: #ffb3a6; --danger-border: #7a2f22;
            --demo-bg: #3d2f0d; --demo-ink: #f3d489;
            color-scheme: dark;
        }
        * { box-sizing: border-box; }
        html, body { margin: 0; min-height: 100%; }
@@ -43,7 +94,7 @@
            color: var(--ink);
            overflow-x: hidden;
            background:
-                radial-gradient(circle at 8% 0%, rgba(56, 180, 213, .12), transparent 25rem),
+                radial-gradient(circle at 8% 0%, var(--glow), transparent 25rem),
                var(--paper);
            /* Thai families first so Thai text uses a proper Thai face where one
               is installed (Android/iOS/Windows all ship one); Latin falls
@@ -60,7 +111,32 @@
        html[lang="th"] body { line-break: normal; overflow-wrap: anywhere; }
        html[lang="th"] .stat-value, html[lang="th"] .flow-value { overflow-wrap: normal; }
        .lang-toggle { padding: 9px 12px; font-size: .78rem; font-weight: 800; white-space: nowrap; }
-        .lang-toggle[data-active-lang="th"] { background: var(--mint); border-color: #b6dfc9; color: #146644; }
+        .lang-toggle[data-active-lang="th"] { background: var(--mint); border-color: var(--mint-border); color: var(--mint-ink); }
        .theme-toggle { padding: 9px 11px; font-size: .95rem; line-height: 1; }
        .alerts-panel { margin-bottom: 14px; padding: 18px 20px; border-radius: 16px; background: var(--card); border: 1px solid var(--border); box-shadow: var(--shadow); }
        .alerts-head { display: flex; justify-content: space-between; align-items: center; gap: 12px; }
        .alerts-head h2 { margin: 0; font-size: 1.15rem; }
        .alerts-close { background: transparent; border: 0; color: var(--muted); font-size: 1.1rem; cursor: pointer; padding: 4px 8px; }
        .alerts-intro { color: var(--muted); font-size: .92rem; line-height: 1.5; margin: 8px 0 12px; }
        .alerts-server { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; font-size: .9rem; margin-bottom: 12px; }
        .alerts-server code { background: var(--surface); border: 1px solid var(--border); padding: 4px 8px; border-radius: 8px; font-size: .9rem; }
        .alerts-server button { padding: 4px 10px; font-size: .8rem; }
        .alerts-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 10px; }
        .alerts-topic { border: 1px solid var(--border); border-radius: 12px; padding: 10px 12px; background: var(--surface); display: flex; flex-direction: column; gap: 4px; }
        .alerts-topic .name { font-weight: 600; font-size: .95rem; }
        .alerts-topic .desc { color: var(--muted); font-size: .82rem; line-height: 1.4; }
        .alerts-topic .row { display: flex; align-items: center; gap: 8px; margin-top: 4px; flex-wrap: wrap; }
        .alerts-topic code { font-size: .82rem; background: var(--card); border: 1px solid var(--border); padding: 2px 6px; border-radius: 6px; }
        .alerts-topic a { font-size: .82rem; }
        .alerts-topic.danger { border-color: rgba(220, 38, 38, .45); }
        .alerts-topic.outlook { border-style: dashed; }
        .alerts-foot { color: var(--muted); font-size: .82rem; margin: 12px 0 0; line-height: 1.6; }
        .alerts-disclaimer { display: block; margin-top: 4px; }
        .leaflet-popup-content-wrapper, .leaflet-popup-tip { background: var(--card); color: var(--ink); }
        .leaflet-container a.leaflet-popup-close-button { color: var(--muted); }
        .leaflet-bar a, .leaflet-control-attribution { background: var(--surface); color: var(--ink); border-color: var(--border); }
        .leaflet-control-attribution a { color: var(--river); }
        [data-theme="dark"] .leaflet-control-attribution { background: var(--overlay); }
        .shell { max-width: 1500px; margin: 0 auto; padding: 24px; }
        header {
            display: flex; align-items: center; justify-content: space-between; gap: 20px;
@@ -78,16 +154,16 @@
        .header-actions { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; }
        .live-pill {
            display: flex; gap: 8px; align-items: center; padding: 9px 13px; border-radius: 999px;
-            background: var(--mint); color: #146644; font-weight: 750; font-size: .8rem;
+            background: var(--mint); color: var(--mint-ink); font-weight: 750; font-size: .8rem;
        }
        .live-dot { width: 8px; height: 8px; border-radius: 50%; background: #22a66c; box-shadow: 0 0 0 5px rgba(34,166,108,.12); }
-        .live-pill.demo { background: #fdeeda; color: #9a6200; }
+        .live-pill.demo { background: var(--demo-bg); color: var(--demo-ink); }
        .live-pill.demo .live-dot { background: #e6a23c; box-shadow: 0 0 0 5px rgba(230,162,60,.15); }
        button {
-            border: 1px solid var(--border); border-radius: 11px; background: white; color: var(--ink);
+            border: 1px solid var(--border); border-radius: 11px; background: var(--surface); color: var(--ink);
            padding: 10px 14px; cursor: pointer; font-weight: 700; box-shadow: 0 3px 10px rgba(22,52,62,.05);
        }
-        button:hover { border-color: #a9c4bb; transform: translateY(-1px); }
+        button:hover { border-color: var(--river-light); transform: translateY(-1px); }
        button:disabled { opacity: .55; cursor: wait; transform: none; }
        .stats { display: grid; grid-template-columns: repeat(4, minmax(0,1fr)); gap: 14px; margin-bottom: 14px; }
        .stat {
@@ -97,16 +173,31 @@
        .stat-label { color: var(--muted); text-transform: uppercase; letter-spacing: .09em; font-size: .68rem; font-weight: 800; }
        .stat-value { margin-top: 9px; font-size: 1.65rem; font-weight: 800; letter-spacing: -.04em; white-space: nowrap; }
        .stat-note { color: var(--muted); margin-top: 3px; font-size: .77rem; }
        .stat.stale { border-color: var(--red); background: rgba(204,75,55,.08); }
        .skill-panel { border: 1px solid var(--border); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: var(--surface-3); }
        .skill-head { display: flex; justify-content: space-between; align-items: baseline; gap: 12px; flex-wrap: wrap; }
        .skill-headline { margin: 8px 0 10px; font-weight: 700; font-size: .9rem; }
        .skill-headline.better { color: var(--green); }
        .skill-headline.worse { color: var(--amber); }
        .skill-table-wrap { overflow-x: auto; }
        .skill-table { border-collapse: collapse; font-size: .76rem; width: 100%; min-width: 560px; }
        .skill-table th { text-align: left; color: var(--muted); font-weight: 700; font-size: .66rem; text-transform: uppercase; letter-spacing: .06em; padding: 4px 8px; border-bottom: 1px solid var(--border); }
        .skill-table td { padding: 5px 8px; border-bottom: 1px solid var(--border); white-space: nowrap; }
        .skill-table tr.current td { font-weight: 700; }
        .skill-table td.num { text-align: right; font-variant-numeric: tabular-nums; }
        .skill-table td.dim { color: var(--muted); }
        .stat.stale .stat-value, .stat.stale .stat-note { color: var(--red); }
        .workspace { display: grid; grid-template-columns: minmax(0, 1fr) 330px; gap: 14px; min-height: 640px; }
        .map-card, .side-card { background: var(--card); border: 1px solid var(--border); border-radius: 19px; box-shadow: var(--shadow); overflow: hidden; }
        .map-card { position: relative; }
-        #station-map { height: 640px; width: 100%; background: #dcebea; }
+        #station-map { height: 640px; width: 100%; background: var(--map-bg); }
        .leaflet-tile-pane { filter: var(--map-filter); }
        .map-overlay {
            position: absolute; z-index: 500; top: 16px; left: 52px; right: 16px;
            display: flex; justify-content: space-between; align-items: flex-start; pointer-events: none;
        }
        .map-heading, .legend {
-            background: rgba(255,255,255,.93); backdrop-filter: blur(9px); border: 1px solid rgba(207,224,218,.9);
+            background: var(--overlay); backdrop-filter: blur(9px); border: 1px solid var(--overlay-border);
            border-radius: 13px; padding: 11px 13px; box-shadow: 0 7px 20px rgba(22,58,68,.12);
            pointer-events: auto; /* .map-overlay disables events; re-enable for the legend's rain toggle */
        }
@@ -128,7 +219,7 @@
            width: 100%; display: grid; grid-template-columns: 40px minmax(0,1fr) auto; gap: 10px; align-items: center;
            padding: 11px; border: 0; border-radius: 12px; box-shadow: none; text-align: left; background: transparent;
        }
-        .station-row:hover { background: #f1f7f5; transform: none; }
+        .station-row:hover { background: var(--surface-2); transform: none; }
        .station-code { width: 40px; height: 40px; display: grid; place-items: center; border-radius: 11px; color: white; font-size: .68rem; font-weight: 850; }
        .station-code.long { font-size: .55rem; word-break: break-all; line-height: 1.15; padding: 2px; text-align: center; }
        .station-name { overflow: hidden; }
@@ -137,9 +228,9 @@
        .station-name span { color: var(--muted); font-size: .68rem; margin-top: 3px; }
        .flow-value { text-align: right; font-size: .82rem; font-weight: 800; }
        .flow-value span { display: block; color: var(--muted); font-size: .61rem; font-weight: 650; margin-top: 2px; }
-        .loading-panel, .error-panel { position: absolute; z-index: 600; inset: 0; display: grid; place-items: center; background: rgba(243,247,245,.88); }
+        .loading-panel, .error-panel { position: absolute; z-index: 600; inset: 0; display: grid; place-items: center; background: var(--overlay); }
-        .loading-card { background: white; padding: 18px 22px; border-radius: 14px; box-shadow: var(--shadow); font-weight: 750; }
+        .loading-card { background: var(--card); padding: 18px 22px; border-radius: 14px; box-shadow: var(--shadow); font-weight: 750; }
-        .error-panel { display: none; color: #8d2f22; text-align: center; padding: 25px; }
+        .error-panel { display: none; color: var(--red); text-align: center; padding: 25px; }
        .marker-wrap { background: none; border: 0; }
        .flow-marker {
            --marker-color: #087da5; --marker-size: 26px;
@@ -179,7 +270,7 @@
        .risk-chips { display: flex; gap: 6px; }
        .risk-chip { flex: 1; text-align: center; border-radius: 8px; padding: 5px 4px; font-size: .64rem; font-weight: 800; color: white; }
        .risk-chip span { display: block; font-weight: 650; font-size: .58rem; opacity: .85; }
-        .p1-outlook { border: 1px solid var(--border); border-left: 4px solid var(--river); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: #f7fbfa; }
+        .p1-outlook { border: 1px solid var(--border); border-left: 4px solid var(--river); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: var(--surface-3); }
        .p1-outlook-head { display: flex; justify-content: space-between; align-items: center; gap: 10px; flex-wrap: wrap; }
        .p1-outlook-head strong { font-size: .88rem; }
        .p1-peak { color: var(--muted); font-size: .76rem; }
@@ -193,16 +284,16 @@
        .popup h3 { margin: 4px 0 2px; font-size: 1rem; }
        .popup-th { color: var(--muted); font-size: .74rem; }
        .popup-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 8px; margin-top: 12px; }
-        .popup-metric { background: #f1f7f5; padding: 8px; border-radius: 9px; }
+        .popup-metric { background: var(--surface-2); padding: 8px; border-radius: 9px; }
        .popup-metric span { display: block; color: var(--muted); font-size: .62rem; }
        .popup-metric strong { display: block; margin-top: 2px; font-size: .86rem; }
        .popup-time { margin-top: 9px; color: var(--muted); font-size: .64rem; }
        /* Verdict banner: light tinted backgrounds with dark ink (contrast-safe) */
-        #flood-verdict.ok { background: #e2f3ea; color: #0c5138; border-color: #bfe3d2; }
+        #flood-verdict.ok { background: var(--ok-bg); color: var(--ok-ink); border-color: var(--ok-border); }
-        #flood-verdict.watch { background: #fdf1dc; color: #6b4a05; border-color: #f0d9a8; }
+        #flood-verdict.watch { background: var(--watch-bg); color: var(--watch-ink); border-color: var(--watch-border); }
-        #flood-verdict.danger { background: #fbe3de; color: #7c1d10; border-color: #f2c0b6; }
+        #flood-verdict.danger { background: var(--danger-bg); color: var(--danger-ink); border-color: var(--danger-border); }
        #legend-toggle {
-            display: none; pointer-events: auto; background: rgba(255,255,255,.93); border: 1px solid rgba(207,224,218,.9);
+            display: none; pointer-events: auto; background: var(--overlay); border: 1px solid var(--overlay-border);
            border-radius: 11px; padding: 8px 12px; font-size: .72rem; font-weight: 800; color: var(--ink);
            box-shadow: 0 7px 20px rgba(22,58,68,.12);
        }
@@ -275,10 +366,34 @@
            <div class="live-pill" id="live-pill"><span class="live-dot"></span> <span id="live-pill-text" data-i18n="pill.live">LIVE DATA</span></div>
            <button id="refresh-button" type="button" data-i18n="action.refresh">↻ Refresh</button>
            <button id="lang-toggle" class="lang-toggle" type="button" data-active-lang="en" aria-label="Switch to Thai">ไทย</button>
            <button id="theme-toggle" class="theme-toggle" type="button" data-i18n-aria="theme.toggle" aria-label="Switch to dark mode" title="Switch to dark mode">🌙</button>
            <button id="alerts-button" type="button" data-i18n="alerts.button" style="display:none">🔔 Get alerts</button>
            <button id="replay-2024" type="button" data-i18n="replay.start">▶ Replay Oct 2024 flood</button>
        </div>
    </header>
    <section id="alerts-panel" class="alerts-panel" style="display:none" aria-labelledby="alerts-title">
        <div class="alerts-head">
            <h2 id="alerts-title" data-i18n="alerts.title">Flood alerts on your phone</h2>
            <button type="button" class="alerts-close" id="alerts-close" data-i18n-aria="alerts.close" aria-label="Close">✕</button>
        </div>
        <p class="alerts-intro" data-i18n="alerts.intro">Free push notifications when a gauge crosses its warning or danger level, and an all-clear when it drops back. No account: install the ntfy app (iOS / Android / any browser), add the server, subscribe to the topics you want. You get a message only when something changes: a few per flood, none in a quiet season.</p>
        <div class="alerts-server">
            <span data-i18n="alerts.server">Server</span>
            <code id="alerts-server-url"></code>
            <button type="button" id="alerts-copy" data-i18n="alerts.copy">Copy</button>
        </div>
        <div class="alerts-grid" id="alerts-topics"></div>
        <p class="alerts-foot">
            <span data-i18n="alerts.apps">Apps:</span>
            <a href="https://apps.apple.com/us/app/ntfy/id1625396347" target="_blank" rel="noopener">iOS</a> ·
            <a href="https://play.google.com/store/apps/details?id=io.heckel.ntfy" target="_blank" rel="noopener">Android</a> ·
            <a href="https://f-droid.org/en/packages/io.heckel.ntfy/" target="_blank" rel="noopener">F-Droid</a> ·
            <a id="alerts-web-link" href="#" target="_blank" rel="noopener" data-i18n="alerts.web">Web (no install)</a>
            <span class="alerts-disclaimer" data-i18n="alerts.disclaimer">Unofficial community service, best effort. For official warnings follow ThaiWater / TMD / your district office.</span>
        </p>
    </section>
    <section id="flood-verdict" role="status" aria-live="polite" style="display:none;margin-bottom:14px;padding:15px 18px;border-radius:16px;border:1px solid;display:none">
        <div style="display:flex;gap:12px;align-items:baseline;flex-wrap:wrap">
            <strong id="verdict-icon" style="font-size:1.2rem"></strong>
@@ -321,7 +436,7 @@
                    <div class="legend-row"><i class="swatch" style="background:#2a1668;border-radius:50%"></i> <span data-i18n="legend.rain.extreme">Extreme &gt; 150</span></div>
                    <div class="legend-title" style="margin-top:10px" data-i18n="legend.other">Other markers</div>
                    <div class="legend-row"><i class="swatch" style="background:#7b8f94"></i> <span data-i18n="legend.other.nodata">Gauge · no recent data</span></div>
-                    <div class="legend-row"><span style="font-weight:800;color:#0f6844">+</span> <span data-i18n="legend.other.sensor">Water-level sensor · colour = % of bank height</span></div>
+                    <div class="legend-row"><span style="font-weight:800;color:var(--green)">+</span> <span data-i18n="legend.other.sensor">Water-level sensor · colour = % of bank height</span></div>
                </div>
                </div>
            </div>
@@ -332,7 +447,7 @@
        <aside class="side-card">
            <div class="side-head"><h2 data-i18n="side.title">Current station flow</h2><p data-i18n="side.subtitle">Select a station to locate it and load its history</p>
-                <input type="search" id="station-search" data-i18n-placeholder="search.placeholder" data-i18n-aria="search.aria" placeholder="Search stations · code, name, river…" aria-label="Search stations" style="margin-top:10px;width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:white;font-size:.8rem">
+                <input type="search" id="station-search" data-i18n-placeholder="search.placeholder" data-i18n-aria="search.aria" placeholder="Search stations · code, name, river…" aria-label="Search stations" style="margin-top:10px;width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink);font-size:.8rem">
            </div>
            <div class="station-list" id="river-flow" aria-live="polite"></div>
            <div class="side-head" id="sensors-head" style="cursor:pointer" role="button" tabindex="0" aria-expanded="false" aria-controls="thaiwater-sensors" data-i18n-title="sensors.tip" title="Show / hide the ThaiWater/HII station list"><h2><span data-i18n="sensors.title">Additional basin stations</span> <span id="sensors-arrow" style="color:var(--muted);font-size:.8rem">▸</span></h2><p id="thaiwater-count" data-i18n="sensors.loading">Loading ThaiWater/HII stations…</p></div>
@@ -356,6 +471,16 @@
            <div class="p1-peak" style="margin-top:7px" data-i18n="outlook.explainer">Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.</div>
        </div>
        <button type="button" class="zones-button" id="forecast-expand" style="display:none;margin-top:12px">Show all station forecasts ▾</button>
        <div class="skill-panel" id="skill-panel" style="display:none">
            <div class="skill-head">
                <strong data-i18n="skill.title">Is the model getting better?</strong>
                <span class="subtitle" id="skill-sub"></span>
            </div>
            <div class="skill-headline" id="skill-headline"></div>
            <p class="subtitle skill-caveat" id="skill-caveat" style="margin:-4px 0 10px"></p>
            <div class="skill-table-wrap"><table class="skill-table" id="skill-table"></table></div>
            <p class="subtitle" style="margin:8px 0 0" data-i18n="skill.explain">Every hour the deployed model's 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.</p>
        </div>
        <div class="forecast-grid" id="forecast-grid" style="display:none"></div>
    </section>
@@ -363,10 +488,10 @@
        <div style="display:flex;justify-content:space-between;align-items:center;gap:14px;flex-wrap:wrap">
            <div><h2 id="history-title" style="margin:0;font-size:1rem" data-i18n="history.title">Station history</h2><p id="history-status" class="subtitle" data-i18n="history.status">Select a station to load the last 7 days</p></div>
            <div class="history-controls" style="display:flex;gap:8px;align-items:center;flex-wrap:wrap">
-                <select id="history-range" style="padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:white"><option value="24" data-i18n="range.24h">Last 24 hours</option><option value="168" selected data-i18n="range.7d">Last 7 days</option><option value="720" data-i18n="range.30d">Last 30 days</option><option value="2160" data-i18n="range.90d">Last 90 days</option><option value="876000" data-i18n="range.all">All time</option><option value="custom" hidden data-i18n="range.custom">Custom range</option></select>
+                <select id="history-range" style="padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink)"><option value="24" data-i18n="range.24h">Last 24 hours</option><option value="168" selected data-i18n="range.7d">Last 7 days</option><option value="720" data-i18n="range.30d">Last 30 days</option><option value="2160" data-i18n="range.90d">Last 90 days</option><option value="876000" data-i18n="range.all">All time</option><option value="custom" hidden data-i18n="range.custom">Custom range</option></select>
-                <input type="date" id="history-start" data-i18n-title="range.from" title="From date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:white">
+                <input type="date" id="history-start" data-i18n-title="range.from" title="From date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink)">
                <span style="color:var(--muted)">–</span>
-                <input type="date" id="history-end" data-i18n-title="range.to" title="To date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:white">
+                <input type="date" id="history-end" data-i18n-title="range.to" title="To date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink)">
            </div>
        </div>
        <div style="height:260px;margin-top:14px;overflow:hidden;position:relative"><canvas id="history-chart" data-i18n-aria="aria.chart" aria-label="Historical water level and discharge chart" style="display:block"></canvas><div id="history-placeholder" data-i18n="history.placeholder" style="position:absolute;inset:0;display:grid;place-items:center;color:var(--muted);font-size:.85rem;text-align:center;padding:20px">Click any station on the map or in the list to see its history</div></div>
@@ -407,6 +532,8 @@
            'app.title': 'Ping River Live Monitor',
            'app.subtitle': 'Current water level and discharge across Northern Thailand',
            'pill.live': 'LIVE DATA',
            'theme.toggle': 'Toggle dark mode',
            'pill.stale': '⚠ STALE FEED',
            'pill.replay': '⏪ 2024 REPLAY',
            'pill.sim': '⚠ SIMULATION',
            'action.refresh': '↻ Refresh',
@@ -439,7 +566,9 @@
            'stat.stress.tip': 'How full the river channel is at the busiest gauge — 100% means water reaches the top of the bank',
            'stat.updated': 'Last updated',
            'stat.updated.note': 'Loading latest readings',
-            'stat.updated.ago': (date, mins) => `${date} · ${mins} min ago`,
+            'stat.updated.ago': (date, mins) => `${date} · ${mins} min ago (ICT)`,
            'stat.updated.agoh': (date, hours) => `${date} · ${hours} h ago (ICT)`,
            'stat.updated.stale': (date, hours) => `⚠ Feed stale · last reading ${date}, ${hours} h ago`,
            'stat.updated.none': 'No timestamp available',
            'map.title': 'Station flow map',
            'map.subtitle': 'River width, colour & dash speed follow live discharge',
@@ -491,6 +620,51 @@
            'forecast.chip.peak': (lvl) => ` · peak ~${lvl} m`,
            'forecast.chip.heuristic': ' · heuristic fallback',
            'forecast.expand': (n) => `Show all ${n} station forecasts ▾`,
            'skill.title': 'Is the model getting better?',
            'skill.sub': (n, since) => `${n} verified 24 h forecasts for P.1 since ${since}`,
            'skill.explain': 'Every hour the deployed model\'s 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.',
            'skill.better': (v, prev, d) => `Current model ${v} is more accurate than ${prev}: peak error ${d} cm lower on the hours it has served.`,
            'skill.worse': (v, prev, d) => `Current model ${v} has a higher peak error than ${prev} so far (+${d} cm).`,
            'skill.caveat.quiet': 'All verified hours so far were below 2 m: this measures quiet-river accuracy only. The model is built and judged for flood onset (lead time before 3.70 m), which no quiet week can test — see the backtests in the documentation.',
            'skill.caveat.regime': 'Versions served different weeks; the ≥ 2 m column compares them on the hours that matter.',
            'alerts.button': '🔔 Get alerts',
            'alerts.title': 'Flood alerts on your phone',
            'alerts.intro': 'Free push notifications when a gauge crosses its warning or danger level, and an all-clear when it drops back. No account: install the ntfy app (iOS / Android / any browser), add the server, subscribe to the topics you want. You get a message only when something changes: a few per flood, none in a quiet season.',
            'alerts.server': 'Server',
            'alerts.copy': 'Copy',
            'alerts.copied': 'Copied',
            'alerts.close': 'Close',
            'alerts.apps': 'Apps:',
            'alerts.web': 'Web (no install)',
            'alerts.disclaimer': 'Unofficial community service, best effort. For official warnings follow ThaiWater / TMD / your district office.',
            'alerts.subscribe': 'Subscribe in app',
            'alerts.t.warning': 'Any gauge: warning level',
            'alerts.t.warning.d': 'One message when any Ping River gauge crosses its warning level, and when levels fall back. The one to pick if unsure.',
            'alerts.t.danger': 'Any gauge: danger level',
            'alerts.t.danger.d': 'Only the serious crossings, basin-wide. Highest priority: rings through Do Not Disturb on most phones.',
            'alerts.t.p1.warning': 'Chiang Mai city (P.1) warning',
            'alerts.t.p1.warning.d': 'Nawarat Bridge crosses 3.70 m (stage 1: low-lying riverside areas), and the all-clear.',
            'alerts.t.p1.danger': 'Chiang Mai city (P.1) danger',
            'alerts.t.p1.danger.d': 'Nawarat Bridge crosses 4.20 m (stage 5: inner city districts).',
            'alerts.t.p103.warning': 'Ring Road 3 (P.103) warning',
            'alerts.t.p103.warning.d': 'Downstream city gauge crosses 5.95 m.',
            'alerts.t.outlook': 'Early warning (model forecast)',
            'alerts.t.outlook.d': 'Experimental: the forecast model gives a ≥ 50 % chance that P.1 reaches its warning level within 24 h. Up to ~13 h earlier than the gauge, but it can be wrong.',
            'alerts.t.status': 'Monitor status',
            'alerts.t.status.d': 'Gauge feed stale / recovered. For people who rely on the dashboard.',
            'skill.single': (v) => `Only ${v} has enough verified hours yet; the next retrain adds a row to compare.`,
            'skill.young': (v, n, min) => `${v} has ${n} verified hours; a comparison needs ${min}.`,
            'skill.none': 'No verified forecasts yet — the first appear 24 h after a model starts serving.',
            'skill.col.version': 'Model',
            'skill.col.period': 'Served',
            'skill.col.n': 'Hours',
            'skill.col.mae': 'Peak error',
            'skill.col.bias': 'Bias',
            'skill.col.pers': 'Persistence',
            'skill.col.skill': 'Skill',
            'skill.col.high': '≥ 2 m error',
            'skill.cm': (v) => `${v} cm`,
            'skill.na': '—',
            'forecast.collapse': 'Hide station forecasts ▴',
            'outlook.title': 'Chiang Mai city flood outlook · P.1 Nawarat Bridge',
            'outlook.explainer': 'Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.',
@@ -579,6 +753,8 @@
            'app.title': 'ติดตามระดับน้ำปิงแบบเรียลไทม์',
            'app.subtitle': 'ระดับน้ำและอัตราการไหลปัจจุบันทั่วภาคเหนือของประเทศไทย',
            'pill.live': 'ข้อมูลสด',
            'theme.toggle': 'สลับโหมดมืด/สว่าง',
            'pill.stale': '⚠ ข้อมูลไม่อัปเดต',
            'pill.replay': '⏪ ย้อนเหตุการณ์ 2567',
            'pill.sim': '⚠ การจำลอง',
            'action.refresh': '↻ รีเฟรช',
@@ -611,7 +787,9 @@
            'stat.stress.tip': 'ระดับความเต็มของลำน้ำที่สถานีที่มีน้ำมากที่สุด — 100% หมายถึงน้ำถึงระดับตลิ่ง',
            'stat.updated': 'อัปเดตล่าสุด',
            'stat.updated.note': 'กำลังโหลดข้อมูลล่าสุด',
-            'stat.updated.ago': (date, mins) => `${date} · ${mins} นาทีที่แล้ว`,
+            'stat.updated.ago': (date, mins) => `${date} · ${mins} นาทีที่แล้ว (เวลาไทย)`,
            'stat.updated.agoh': (date, hours) => `${date} · ${hours} ชั่วโมงที่แล้ว (เวลาไทย)`,
            'stat.updated.stale': (date, hours) => `⚠ ข้อมูลไม่อัปเดต · ค่าล่าสุด ${date}, ${hours} ชั่วโมงที่แล้ว`,
            'stat.updated.none': 'ไม่มีข้อมูลเวลา',
            'map.title': 'แผนที่การไหลของน้ำ',
            'map.subtitle': 'ความกว้าง สี และความเร็วเส้นประของแม่น้ำแสดงอัตราการไหลจริง',
@@ -663,6 +841,51 @@
            'forecast.chip.peak': (lvl) => ` · ระดับสูงสุดประมาณ ${lvl} ม.`,
            'forecast.chip.heuristic': ' · ใช้การประมาณอย่างง่าย',
            'forecast.expand': (n) => `แสดงพยากรณ์ทั้ง ${n} สถานี ▾`,
            'skill.title': 'โมเดลแม่นยำขึ้นหรือไม่?',
            'skill.sub': (n, since) => `พยากรณ์ 24 ชม. ของ P.1 ที่ตรวจสอบแล้ว ${n} ครั้ง ตั้งแต่ ${since}`,
            'skill.explain': 'ทุกชั่วโมงระบบบันทึกค่าพยากรณ์ระดับน้ำสูงสุดใน 24 ชม. ของ P.1 ไว้ เมื่อครบ 24 ชม. จึงนำมาเทียบกับระดับน้ำจริง "ทักษะ" คือโมเดลดีกว่าการสมมติว่าระดับน้ำคงที่มากเพียงใด (0 = ไม่ดีกว่า, 1 = สมบูรณ์แบบ) โมเดลที่ฝึกใหม่ด้วยข้อมูลมากขึ้นจะปรากฏเป็นแถวใหม่ จึงเห็นได้ว่าดีขึ้นจริงหรือไม่',
            'skill.better': (v, prev, d) => `โมเดลปัจจุบัน ${v} แม่นยำกว่า ${prev}: ค่าคลาดเคลื่อนต่ำกว่า ${d} ซม. ในช่วงที่ให้บริการ`,
            'skill.worse': (v, prev, d) => `โมเดลปัจจุบัน ${v} มีค่าคลาดเคลื่อนสูงกว่า ${prev} (+${d} ซม.)`,
            'skill.caveat.quiet': 'ชั่วโมงที่ตรวจสอบทั้งหมดอยู่ต่ำกว่า 2 ม.: วัดได้เพียงความแม่นยำช่วงน้ำปกติ โมเดลถูกสร้างและประเมินสำหรับช่วงน้ำเริ่มท่วม (เวลาเตือนล่วงหน้าก่อน 3.70 ม.) ซึ่งสัปดาห์ปกติทดสอบไม่ได้ — ดูผลทดสอบย้อนหลังในเอกสาร',
            'skill.caveat.regime': 'แต่ละเวอร์ชันให้บริการคนละช่วงเวลา คอลัมน์ ≥ 2 ม. เปรียบเทียบเฉพาะชั่วโมงที่สำคัญ',
            'alerts.button': '🔔 รับการแจ้งเตือน',
            'alerts.title': 'แจ้งเตือนน้ำท่วมบนมือถือของคุณ',
            'alerts.intro': 'การแจ้งเตือนฟรีเมื่อระดับน้ำที่สถานีใดข้ามระดับเฝ้าระวังหรือระดับอันตราย และแจ้งเมื่อกลับสู่ปกติ ไม่ต้องสมัครสมาชิก: ติดตั้งแอป ntfy (iOS / Android / เบราว์เซอร์) เพิ่มเซิร์ฟเวอร์ แล้วเลือกหัวข้อที่ต้องการ คุณจะได้รับข้อความเฉพาะเมื่อมีการเปลี่ยนแปลง: ไม่กี่ข้อความต่อเหตุการณ์น้ำท่วม และไม่มีเลยในช่วงปกติ',
            'alerts.server': 'เซิร์ฟเวอร์',
            'alerts.copy': 'คัดลอก',
            'alerts.copied': 'คัดลอกแล้ว',
            'alerts.close': 'ปิด',
            'alerts.apps': 'แอป:',
            'alerts.web': 'เว็บ (ไม่ต้องติดตั้ง)',
            'alerts.disclaimer': 'บริการชุมชนอย่างไม่เป็นทางการ พยายามอย่างดีที่สุด สำหรับคำเตือนอย่างเป็นทางการโปรดติดตาม ThaiWater / กรมอุตุนิยมวิทยา / สำนักงานอำเภอของคุณ',
            'alerts.subscribe': 'สมัครในแอป',
            'alerts.t.warning': 'สถานีใดก็ได้: ระดับเฝ้าระวัง',
            'alerts.t.warning.d': 'หนึ่งข้อความเมื่อสถานีใดในแม่น้ำปิงข้ามระดับเฝ้าระวัง และเมื่อระดับน้ำลดลง หากไม่แน่ใจให้เลือกอันนี้',
            'alerts.t.danger': 'สถานีใดก็ได้: ระดับอันตราย',
            'alerts.t.danger.d': 'เฉพาะการข้ามระดับที่ร้ายแรง ทั้งลุ่มน้ำ ความสำคัญสูงสุด: ดังผ่านโหมดห้ามรบกวนในโทรศัพท์ส่วนใหญ่',
            'alerts.t.p1.warning': 'เมืองเชียงใหม่ (P.1) ระดับเฝ้าระวัง',
            'alerts.t.p1.warning.d': 'สะพานนวรัฐข้าม 3.70 ม. (ระยะที่ 1: พื้นที่ริมน้ำที่ต่ำ) และแจ้งเมื่อกลับสู่ปกติ',
            'alerts.t.p1.danger': 'เมืองเชียงใหม่ (P.1) ระดับอันตราย',
            'alerts.t.p1.danger.d': 'สะพานนวรัฐข้าม 4.20 ม. (ระยะที่ 5: ย่านใจกลางเมือง)',
            'alerts.t.p103.warning': 'ถนนวงแหวน 3 (P.103) ระดับเฝ้าระวัง',
            'alerts.t.p103.warning.d': 'สถานีท้ายเมืองข้าม 5.95 ม.',
            'alerts.t.outlook': 'เตือนล่วงหน้า (แบบจำลองพยากรณ์)',
            'alerts.t.outlook.d': 'ทดลอง: แบบจำลองพยากรณ์ให้โอกาส ≥ 50% ที่ P.1 จะถึงระดับเฝ้าระวังภายใน 24 ชม. เร็วกว่าสถานีวัดได้ถึง ~13 ชม. แต่อาจผิดพลาดได้',
            'alerts.t.status': 'สถานะระบบ',
            'alerts.t.status.d': 'ข้อมูลสถานีล่าช้า / กลับมาปกติ สำหรับผู้ที่พึ่งพาแดชบอร์ด',
            'skill.single': (v) => `มีเพียง ${v} ที่มีข้อมูลตรวจสอบเพียงพอ การฝึกครั้งถัดไปจะเพิ่มแถวให้เปรียบเทียบ`,
            'skill.young': (v, n, min) => `${v} มีข้อมูลตรวจสอบ ${n} ชั่วโมง ต้องการอย่างน้อย ${min} เพื่อเปรียบเทียบ`,
            'skill.none': 'ยังไม่มีพยากรณ์ที่ตรวจสอบได้ — จะเริ่มมี 24 ชม. หลังโมเดลเริ่มทำงาน',
            'skill.col.version': 'โมเดล',
            'skill.col.period': 'ช่วงเวลา',
            'skill.col.n': 'ชั่วโมง',
            'skill.col.mae': 'คลาดเคลื่อน',
            'skill.col.bias': 'อคติ',
            'skill.col.pers': 'ระดับคงที่',
            'skill.col.skill': 'ทักษะ',
            'skill.col.high': 'คลาดเคลื่อน ≥ 2 ม.',
            'skill.cm': (v) => `${v} ซม.`,
            'skill.na': '—',
            'forecast.collapse': 'ซ่อนพยากรณ์รายสถานี ▴',
            'outlook.title': 'แนวโน้มน้ำท่วมเมืองเชียงใหม่ · P.1 สะพานนวรัฐ',
            'outlook.explainer': 'โอกาสที่ระดับน้ำจะถึงแต่ละระดับการท่วมตามประกาศทางการภายใน 24 ชม. — น้ำเริ่มท่วมเมืองที่ระดับ 1 (3.70 ม.) และแต่ละระดับจะท่วมพื้นที่เพิ่มขึ้น',
@@ -774,6 +997,60 @@
        return typeof value === 'function' ? value(...args) : value;
    }
    // ---- public push notifications (ntfy) -------------------------------------
    let ALERTS_CFG = null;
    const ALERT_TOPICS = [
        { key: 'warning',     topic: 'warning',     cls: '' },
        { key: 'danger',      topic: 'danger',      cls: 'danger' },
        { key: 'p1.warning',  topic: 'p1-warning',  cls: '' },
        { key: 'p1.danger',   topic: 'p1-danger',   cls: 'danger' },
        { key: 'p103.warning', topic: 'p103-warning', cls: '' },
        { key: 'outlook',     topic: 'p1-outlook',  cls: 'outlook' },
        { key: 'status',      topic: 'status',      cls: '' },
    ];
    async function loadAlertsConfig() {
        try {
            const r = await fetch('/api/notifications');
            if (!r.ok) return;
            const cfg = await r.json();
            if (!cfg.enabled || !cfg.server) return;
            ALERTS_CFG = cfg;
            $('alerts-button').style.display = '';
            renderAlertsPanel();
        } catch (e) { /* no notifications configured */ }
    }
    function renderAlertsPanel() {
        if (!ALERTS_CFG) return;
        const server = ALERTS_CFG.server.replace(/\/$/, '');
        const host = server.replace(/^https?:\/\//, '');
        $('alerts-server-url').textContent = host;
        $('alerts-web-link').href = server + '/' + ALERTS_CFG.prefix + '-warning';
        $('alerts-topics').innerHTML = ALERT_TOPICS.map(tp => {
            const full = ALERTS_CFG.prefix + '-' + tp.topic;
            const url = server + '/' + full;
            return `<div class="alerts-topic ${tp.cls}">
                <div class="name">${esc(t('alerts.t.' + tp.key))}</div>
                <div class="desc">${esc(t('alerts.t.' + tp.key + '.d'))}</div>
                <div class="row"><code>${esc(full)}</code> <a href="ntfy://${esc(host)}/${esc(full)}">${esc(t('alerts.subscribe'))}</a> · <a href="${esc(url)}" target="_blank" rel="noopener">web</a></div>
            </div>`;
        }).join('');
    }
    function esc(x) { return String(x).replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c])); }
    $('alerts-button').addEventListener('click', () => {
        const p = $('alerts-panel');
        const open = p.style.display === 'none';
        p.style.display = open ? '' : 'none';
        if (open) { renderAlertsPanel(); p.scrollIntoView({ behavior: 'smooth', block: 'start' }); }
    });
    $('alerts-close').addEventListener('click', () => { $('alerts-panel').style.display = 'none'; });
    $('alerts-copy').addEventListener('click', async () => {
        try {
            await navigator.clipboard.writeText(ALERTS_CFG ? ALERTS_CFG.server : '');
            $('alerts-copy').textContent = t('alerts.copied');
            setTimeout(() => { $('alerts-copy').textContent = t('alerts.copy'); }, 1500);
        } catch (e) { /* clipboard blocked */ }
    });
    function applyTranslations() {
        document.documentElement.lang = state.lang;
        document.querySelectorAll('[data-i18n]').forEach((el) => {
@@ -801,7 +1078,34 @@
        }
    }
    // Theme: explicit choice persists; otherwise follow the OS and track its changes.
    const THEME_KEY = 'ping-river-theme';
    const osDark = window.matchMedia('(prefers-color-scheme: dark)');
    function currentTheme() { return document.documentElement.dataset.theme === 'dark' ? 'dark' : 'light'; }
    function applyTheme(theme, persist) {
        document.documentElement.dataset.theme = theme;
        const button = $('theme-toggle');
        if (button) {
            button.textContent = theme === 'dark' ? '☀️' : '🌙';
        }
        if (persist) { try { localStorage.setItem(THEME_KEY, theme); } catch (e) { /* private mode */ } }
        // Chart.js reads colours at construction: rebuild the open chart
        if (state.historyChart && state.selectedStation) loadHistory(state.selectedStation);
    }
    (() => {
        let saved = null;
        try { saved = localStorage.getItem(THEME_KEY); } catch (e) { /* private mode */ }
        applyTheme(saved === 'dark' || saved === 'light' ? saved : (osDark.matches ? 'dark' : 'light'), false);
        osDark.addEventListener('change', (e) => {
            let pinned = null;
            try { pinned = localStorage.getItem(THEME_KEY); } catch (err) { /* ignore */ }
            if (!pinned) applyTheme(e.matches ? 'dark' : 'light', false);
        });
    })();
    function cssVar(name) { return getComputedStyle(document.documentElement).getPropertyValue(name).trim(); }
    function setLang(lang) {
        setTimeout(renderAlertsPanel, 0);
        state.lang = lang;
        try { localStorage.setItem(LANG_KEY, lang); } catch (e) { /* private mode */ }
        applyTranslations();
@@ -841,6 +1145,25 @@
    // what the replay label ("ต.ค. 2567") already says — pinning Gregorian here
    // put two different year systems on the same screen.
    function loc() { return state.lang === 'th' ? 'th-TH' : 'en-GB'; }
    // Every timestamp the API emits is Asia/Bangkok wall-clock WITHOUT an
    // offset ("2026-09-12T02:00:00"). new Date() on such a string uses the
    // browser's own zone, so a viewer in Europe read a 02:00 ICT reading as
    // five hours in the future and saw "0 min ago" forever. Pin the offset
    // here and always format with timeZone: TZ so the site shows river time
    // no matter where it is opened.
    const TZ = 'Asia/Bangkok';
    const STALE_AFTER_MIN = 180;
    function parseTs(value) {
        if (value == null || value === '') return null;
        if (value instanceof Date) return value;
        const text = String(value).trim();
        const naive = /^\d{4}-\d\d-\d\d[T ]\d\d:\d\d(:\d\d(\.\d+)?)?$/.test(text);
        const date = new Date(naive ? text.replace(' ', 'T') + '+07:00' : text);
        return Number.isNaN(date.getTime()) ? null : date;
    }
    // Bucket keys in river-local time (a Bangkok calendar day, not a UTC one)
    const dayKey = (value) => parseTs(value).toLocaleDateString('en-CA', { timeZone: TZ });
    const hourKey = (value) => `${dayKey(value)}-${parseTs(value).toLocaleTimeString('en-GB', { timeZone: TZ, hour: '2-digit' })}`;
    // Metre abbreviation: "ม." reads as a unit in Thai, "m" mid-sentence does not.
    function metres(value, digits = 2) {
        return `${Number(value).toFixed(digits)} ${t('unit.m')}`;
@@ -885,7 +1208,7 @@
            const mm = r.rain_24h == null ? null : Number(r.rain_24h);
            const bin = rainBin(mm);
            const name = (state.lang === 'th' ? r.name_th || r.name_en : r.name_en || r.name_th) || r.oldcode || `Station ${r.station_id}`;
-            const time = r.timestamp ? new Date(r.timestamp).toLocaleString(loc(), { dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
+            const time = r.timestamp ? parseTs(r.timestamp).toLocaleString(loc(), { timeZone: TZ, dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
            L.circleMarker([r.latitude, r.longitude], {
                radius: bin.radius, color: '#ffffff', weight: 1.5,
                fillColor: bin.color, fillOpacity: bin.opacity ?? .85
@@ -912,7 +1235,7 @@
        const latest = new Map();
        measurements.forEach((item) => {
            const prior = latest.get(item.station_code);
-            if (!prior || new Date(item.timestamp) > new Date(prior.timestamp)) latest.set(item.station_code, item);
+            if (!prior || parseTs(item.timestamp) > parseTs(prior.timestamp)) latest.set(item.station_code, item);
        });
        return latest;
    }
@@ -945,7 +1268,7 @@
    function buildPopup(station, measurement) {
        const flow = measurement ? measurement.discharge : null;
        const level = measurement ? measurement.water_level : null;
-        const time = measurement ? new Date(measurement.timestamp).toLocaleString(loc(), { dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
+        const time = measurement ? parseTs(measurement.timestamp).toLocaleString(loc(), { timeZone: TZ, dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
        // Station names are bilingual in the data: lead with the reader's language
        const primary = state.lang === 'th' ? station.thai_name : station.english_name;
        const secondary = state.lang === 'th' ? station.english_name : station.thai_name;
@@ -1125,8 +1448,7 @@
            const downsample = (data) => {
                const buckets = {};
                data.forEach((row) => {
-                    const date = new Date(row.timestamp);
+                    const key = dayKey(row.timestamp);
                    const key = `${date.getUTCFullYear()}-${date.getUTCMonth()}-${date.getUTCDate()}`;
                    if (!buckets[key]) buckets[key] = { ts: row.timestamp, discharge: [], level: [] };
                    const b = buckets[key];
                    if (row.discharge != null) b.discharge.push(row.discharge);
@@ -1147,12 +1469,7 @@
                const fr = await fetch(`/api/forecast/history/${encodeURIComponent(stationCode)}?${query}&horizon=24`);
                const forecastRows = fr.ok ? await fr.json() : [];
                if (forecastRows.length) {
-                    const keyOf = (value) => {
+                    const keyOf = (value) => rows.length > 2000 ? dayKey(value) : hourKey(value);
                        const d = new Date(value);
                        return rows.length > 2000
                            ? `${d.getUTCFullYear()}-${d.getUTCMonth()}-${d.getUTCDate()}`
                            : `${d.getUTCFullYear()}-${d.getUTCMonth()}-${d.getUTCDate()}-${d.getUTCHours()}`;
                    };
                    const byKey = new Map();
                    forecastRows.forEach((r) => {
                        if (r.predicted_max_level == null) return;
@@ -1177,7 +1494,7 @@
            state.historyChart = new Chart($('history-chart'), {
                type: 'line',
                data: {
-                    labels: sampled.map((row) => new Date(row.timestamp).toLocaleString(loc(), { timeZone: 'Asia/Bangkok', month: 'short', day: 'numeric', year: '2-digit', hour: '2-digit' })),
+                    labels: sampled.map((row) => parseTs(row.timestamp).toLocaleString(loc(), { timeZone: TZ, month: 'short', day: 'numeric', year: '2-digit', hour: '2-digit' })),
                    datasets: [
                        { label: t('chart.discharge'), data: sampled.map((row) => row.discharge), borderColor: '#087da5', backgroundColor: 'rgba(8,125,165,.12)', yAxisID: 'flow', pointRadius: 0, tension: .25 },
                        { label: t('chart.level'), data: sampled.map((row) => row.water_level), borderColor: '#d99018', yAxisID: 'level', pointRadius: 0, tension: .25 },
@@ -1187,13 +1504,14 @@
                options: {
                    responsive: true, maintainAspectRatio: false, animation: { duration: 0 },
                    interaction: { mode: 'index', intersect: false },
                    color: cssVar('--ink'),
                    scales: {
-                        flow: { type: 'linear', position: 'left' },
+                        flow: { type: 'linear', position: 'left', grid: { color: cssVar('--chart-grid') }, ticks: { color: cssVar('--muted') } },
-                        level: { type: 'linear', position: 'right', grid: { drawOnChartArea: false } },
+                        level: { type: 'linear', position: 'right', grid: { drawOnChartArea: false }, ticks: { color: cssVar('--muted') } },
-                        x: { ticks: { maxTicksLimit: 12 } }
+                        x: { ticks: { maxTicksLimit: 12, color: cssVar('--muted') }, grid: { color: cssVar('--chart-grid') } }
                    },
                    plugins: {
-                        legend: { display: true },
+                        legend: { display: true, labels: { color: cssVar('--ink') } },
                        floodBands: { enabled: true }
                    }
                },
@@ -1343,7 +1661,7 @@
    function setP1Level(value, at, opts) {
        const options = opts || {};
        if (value == null || Number.isNaN(Number(value))) return;
-        const stamp = at ? new Date(at).getTime() : null;
+        const stamp = at ? (parseTs(at)?.getTime() ?? NaN) : null;
        const known = Number.isFinite(stamp) ? stamp : null;
        if (!options.force && known != null && state.p1NowAt != null && known < state.p1NowAt) {
            return; // an older snapshot must not overwrite a newer one
@@ -1383,7 +1701,7 @@
    function renderSummary(stations, readings) {
        const current = stations.map((s) => readings.get(s.station_code)).filter(Boolean);
-        const timestamps = current.map((m) => new Date(m.timestamp)).filter((date) => !Number.isNaN(date.getTime()));
+        const timestamps = current.map((m) => parseTs(m.timestamp)).filter(Boolean);
        const latest = timestamps.length ? new Date(Math.max(...timestamps.map((date) => date.getTime()))) : null;
        $('station-count').textContent = `${current.length} / ${stations.length}`;
        if (state.hiiReportingCount != null) {
@@ -1403,12 +1721,24 @@
        const worst = stressed.length ? stressed.reduce((max, item) => item.percent > max.percent ? item : max) : null;
        $('peak-flow').textContent = worst ? `${worst.percent.toFixed(0)}%` : '—';
        $('peak-station').textContent = worst ? t('stat.stress.capacity', worst.code) : t('stat.stress.nodata');
-        $('last-updated').textContent = latest ? latest.toLocaleTimeString(loc(), { hour: '2-digit', minute: '2-digit' }) : '—';
+        $('last-updated').textContent = latest ? latest.toLocaleTimeString(loc(), { timeZone: TZ, hour: '2-digit', minute: '2-digit' }) : '—';
        const tile = $('last-updated').closest('.stat');
        if (latest) {
            const minutes = Math.max(0, Math.round((Date.now() - latest.getTime()) / 60000));
-            $('data-age').textContent = t('stat.updated.ago',
+            const day = latest.toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' });
-                latest.toLocaleDateString(loc(), { day: 'numeric', month: 'short' }), minutes);
+            // RID publishes hourly and the scrape runs hourly, so anything past
-        } else $('data-age').textContent = t('stat.updated.none');
+            // ~3 h means the feed or the collector has stopped: say so loudly
            // instead of letting "6000 min ago" pass as a number.
            const stale = minutes >= STALE_AFTER_MIN;
            $('data-age').textContent = stale
                ? t('stat.updated.stale', day, Math.round(minutes / 60))
                : minutes >= 120 ? t('stat.updated.agoh', day, Math.round(minutes / 60))
                : t('stat.updated.ago', day, minutes);
            tile.classList.toggle('stale', stale);
            if (!state.replayTimer && (state.liveMode === 'live' || state.liveMode === 'stale')) {
                setLiveIndicator(stale ? 'stale' : 'live', stale ? 'pill.stale' : 'pill.live');
            }
        } else { $('data-age').textContent = t('stat.updated.none'); tile.classList.remove('stale'); }
    }
    async function loadDashboard() {
@@ -1655,13 +1985,13 @@
                // force: replay frames are 2024 timestamps, older than anything live
                if (p1Level != null) setP1Level(p1Level, null, { force: true });
                restyleFloodZones();
-                const ts = new Date(data.timestamps[frame]);
+                const ts = parseTs(data.timestamps[frame]);
                // minute included: a lone "17" reads as a year in Thai output
-                const when = ts.toLocaleString(loc(), { day: 'numeric', month: 'short', hour: '2-digit', minute: '2-digit' });
+                const when = ts.toLocaleString(loc(), { timeZone: TZ, day: 'numeric', month: 'short', hour: '2-digit', minute: '2-digit' });
                const p1Text = state.p1Now == null ? '—' : metres(state.p1Now);
                const basinFlow = Math.round(totalFlow).toLocaleString(loc());
                $('p1-peak').textContent = t('replay.peak', when, p1Text, basinFlow);
-                $('replay-clock-time').textContent = ts.toLocaleString(loc(), { day: 'numeric', month: 'short', year: 'numeric', hour: '2-digit', minute: '2-digit' });
+                $('replay-clock-time').textContent = ts.toLocaleString(loc(), { timeZone: TZ, day: 'numeric', month: 'short', year: 'numeric', hour: '2-digit', minute: '2-digit' });
                $('replay-clock-sub').textContent = t('replay.clock.sub', p1Text, basinFlow);
                // model track: what the forecast system (trained pre-flood) said at this moment
                const model = data.model || {};
@@ -1850,9 +2180,9 @@
                    `<div class="risk-chips">${chips}</div>`;
                grid.appendChild(cardEl);
            });
-            const asOf = rows[0].as_of ? new Date(rows[0].as_of).toLocaleString(loc(), { dateStyle: 'medium', timeStyle: 'short' }) : null;
+            const asOf = rows[0].as_of ? parseTs(rows[0].as_of).toLocaleString(loc(), { timeZone: TZ, dateStyle: 'medium', timeStyle: 'short' }) : null;
            const modelRow = rows.find((r) => r.source === 'model');
-            const trainedAt = modelRow?.trained_at ? new Date(modelRow.trained_at).toLocaleDateString(loc(), { day: 'numeric', month: 'short' }) : null;
+            const trainedAt = modelRow?.trained_at ? parseTs(modelRow.trained_at).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' }) : null;
            const modelInfo = modelRow?.model_version
                ? t('forecast.status.model', modelRow.model_version, trainedAt ? t('forecast.status.trained', trainedAt) : '')
                : '';
@@ -1868,11 +2198,60 @@
                ? t('forecast.collapse')
                : t('forecast.expand', stations.length);
            card.style.display = 'block';
            loadSkill(); // non-blocking; panel stays hidden until there is verified data
            loadAlertsConfig(); // shows the "Get alerts" button only when ntfy is configured
        } catch (error) {
            card.style.display = 'none';
        }
    }
    async function loadSkill() {
        const panel = $('skill-panel');
        try {
            const response = await fetch('/api/forecast/skill?station_code=P.1&horizon=24');
            if (!response.ok) throw new Error(`HTTP ${response.status}`);
            const data = await response.json();
            const versions = (data.versions || []).filter((v) => v.n > 0);
            if (!versions.length) { panel.style.display = 'none'; return; }
            const cm = (m) => m == null ? t('skill.na') : t('skill.cm', (m * 100).toFixed(1));
            const fmtDay = (v) => parseTs(v).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' });
            const total = versions.reduce((a, v) => a + v.n, 0);
            $('skill-sub').textContent = t('skill.sub', total.toLocaleString(loc()), fmtDay(versions[0].first_issued));
            const head = $('skill-headline');
            head.className = 'skill-headline';
            const cur = data.current;
            if (data.trend && cur) {
                const delta = Math.abs(data.trend.mae_delta_m * 100).toFixed(1);
                head.textContent = data.trend.better
                    ? t('skill.better', cur.model_version, data.trend.previous_version, delta)
                    : t('skill.worse', cur.model_version, data.trend.previous_version, delta);
                head.classList.add(data.trend.better ? 'better' : 'worse');
            } else if (cur && cur.enough_data) {
                head.textContent = t('skill.single', cur.model_version);
            } else if (cur) {
                head.textContent = t('skill.young', cur.model_version, cur.n, data.min_verified);
            } else head.textContent = t('skill.none');
            const anyHigh = versions.some((v) => v.above_2m_n > 0);
            const compared = versions.filter((v) => v.enough_data).length > 1;
            $('skill-caveat').textContent = !anyHigh ? t('skill.caveat.quiet') : compared ? t('skill.caveat.regime') : '';
            const cols = ['version', 'period', 'n', 'mae', 'bias', 'pers', 'skill', 'high'];
            const rows = versions.map((v) => `<tr class="${v === cur ? 'current' : ''}${v.enough_data ? '' : ' young'}">`
                + `<td>${escapeHtml(v.model_version)}</td>`
                + `<td class="dim">${fmtDay(v.first_issued)} – ${fmtDay(v.last_issued)}</td>`
                + `<td class="num">${v.n.toLocaleString(loc())}</td>`
                + `<td class="num">${cm(v.mae_m)}</td>`
                + `<td class="num">${v.bias_m == null ? t('skill.na') : (v.bias_m >= 0 ? '+' : '') + (v.bias_m * 100).toFixed(1)}</td>`
                + `<td class="num dim">${cm(v.persistence_mae_m)}</td>`
                + `<td class="num">${v.skill == null ? t('skill.na') : v.skill.toFixed(2)}</td>`
                + `<td class="num">${v.above_2m_n ? `${cm(v.above_2m_mae_m)} <span class="dim">(${v.above_2m_n})</span>` : t('skill.na')}</td>`
                + '</tr>').join('');
            $('skill-table').innerHTML = `<thead><tr>${cols.map((c) => `<th>${escapeHtml(t('skill.col.' + c))}</th>`).join('')}</tr></thead><tbody>${rows}</tbody>`;
            panel.style.display = 'block';
        } catch (error) {
            panel.style.display = 'none';
        }
    }
    async function loadDbStats() {
        const strip = $('db-stats');
        try {
@@ -1888,7 +2267,7 @@
        const strip = $('db-stats');
        try {
            state.lastStats = stats;
-            const fmtDate = (value) => new Date(value).toLocaleDateString(loc(), { day: 'numeric', month: 'short', year: 'numeric' });
+            const fmtDate = (value) => parseTs(value).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short', year: 'numeric' });
            state.dbFirstDate = String(stats.first_timestamp).slice(0, 10); // feeds the All-time date range
            $('db-total').textContent = Number(stats.total_measurements).toLocaleString(loc());
            if (stats.rid_measurements != null) {
@@ -1924,6 +2303,7 @@
            : t('forecast.expand', count);
    });
    $('lang-toggle').addEventListener('click', () => setLang(state.lang === 'th' ? 'en' : 'th'));
    $('theme-toggle').addEventListener('click', () => applyTheme(currentTheme() === 'dark' ? 'light' : 'dark', true));
    $('station-search').addEventListener('input', applyStationSearch);
    $('sensors-head').addEventListener('click', () => setSensorsOpen(!state.sensorsOpen));
    $('sensors-head').addEventListener('keydown', (event) => {
@@ -632,9 +632,7 @@ class EnhancedWaterMonitorScraper:
                if data:
                    if self.save_to_database(data):
                        filled_count += len(data)
-                        logger.info(
+                        logger.info(f"Filled {len(data)} measurements for {fetch_date}")
                            f"Filled {len(data)} measurements for {fetch_date}"
                        )
                    else:
                        logger.warning(f"Failed to save data for {fetch_date}")
                else:
@@ -210,7 +210,9 @@ async def lifespan(app: FastAPI):
    app_state["leader_lock"] = _acquire_collection_leadership(
        Config.COLLECTION_LEADER_PORT
    )
    app_state["notify"] = None
    if app_state["leader_lock"]:
        app_state["notify"] = _init_notifications()
        app_state["scraping_task"] = asyncio.create_task(background_scraping_task())
        logger.info("This worker is the background-collection leader")
    else:
@@ -296,6 +298,73 @@ async def _persist_rain():
        logger.warning(f"rain persistence failed: {e}")
 def _init_notifications():
    """Publisher + persisted state for ntfy, or None if off/unavailable.
    Called only by the collection leader: it is the one process that
    publishes, so the notification_state DDL runs exactly once per host.
    """
    if not Config.NTFY_SERVER:
        return None
    try:
        from . import notify as notify_mod
        store = app_state.get("forecast_store")
        if store and not store.engine:
            store.connect()
        state = (
            notify_mod.NotificationState(store.engine, store.db_type)
            if store and store.engine
            else notify_mod.InMemoryState()
        )
        if isinstance(state, notify_mod.InMemoryState):
            logger.warning(
                "ntfy: no SQL store; notification state is in-memory "
                "(a restart may re-send the current level)"
            )
        publisher = notify_mod.NtfyPublisher(
            Config.NTFY_PUBLISH_URL,
            prefix=Config.NTFY_TOPIC_PREFIX,
            token=Config.NTFY_TOKEN or None,
            dashboard_url=Config.PUBLIC_URL,
        )
        logger.info(
            f"ntfy notifications: publish to {Config.NTFY_PUBLISH_URL}, "
            f"subscribers use {Config.NTFY_SERVER}, topics {Config.NTFY_TOPIC_PREFIX}-*"
        )
        return publisher, state
    except Exception as e:
        logger.error(f"ntfy init failed (notifications off): {e}")
        return None
 async def _notify_transitions():
    """Publish flood/outlook/feed transitions to ntfy (leader only, fail-safe)."""
    cfg = app_state.get("notify")
    if not cfg:
        return
    publisher, state = cfg
    try:
        from . import notify as notify_mod
        scraper = app_state["scraper"]
        readings = await asyncio.to_thread(
            scraper.db_adapter.get_latest_measurements, 200
        )
        with FORECAST_CACHE_LOCK:
            cached = FORECAST_CACHE.get("all")
        forecasts = cached[1] if cached else []
        sent = await asyncio.to_thread(
            notify_mod.evaluate, readings, forecasts, state, publisher
        )
        if sent:
            logger.info(
                "ntfy: published " + ", ".join(f"{n.topic}: {n.title}" for n in sent)
            )
    except Exception as e:
        logger.warning(f"ntfy notify cycle failed: {e}")
 async def _precompute_forecasts():
    """Refresh the forecast cache and persist the issued forecasts (leader only)."""
    try:
@@ -374,9 +443,7 @@ async def background_scraping_task():
                            hii_counts = await asyncio.get_event_loop().run_in_executor(
                                None, hii_collector.run_cycle
                            )
-                            set_gauge(
+                            set_gauge("hii_rainfall_rows_saved", hii_counts["rainfall"])
                                "hii_rainfall_rows_saved", hii_counts["rainfall"]
                            )
                            set_gauge(
                                "hii_waterlevel_rows_saved", hii_counts["waterlevel"]
                            )
@@ -404,6 +471,10 @@ async def background_scraping_task():
                    # evaluation.
                    await _precompute_forecasts()
                    # Push notifications for threshold crossings (uses the
                    # forecasts just computed; no-op unless NTFY_SERVER set).
                    await _notify_transitions()
                app_state["is_scraping"] = False
            # Calculate next run time
@@ -521,7 +592,9 @@ _STATIC_DIR = os.path.dirname(_DASHBOARD_HTML_PATH)
@app.get("/robots.txt", include_in_schema=False)
 async def robots_txt():
-    return FileResponse(os.path.join(_STATIC_DIR, "robots.txt"), media_type="text/plain")
+    return FileResponse(
        os.path.join(_STATIC_DIR, "robots.txt"), media_type="text/plain"
    )
@app.get("/llms.txt", include_in_schema=False)
@@ -864,7 +937,7 @@ def _hii_rows(sql: str, params: Dict[str, Any]) -> List[Dict[str, Any]]:
 # flood, slightly stale readings with a visible timestamp beat an error page.
 HII_CACHE: Dict[str, Any] = {}
 HII_CACHE_LOCK = Lock()
-_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock()}
+_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock(), "skill": Lock()}
 LATEST_CACHE: Dict[str, Any] = {}
 LATEST_CACHE_LOCK = Lock()
 _LATEST_COMPUTE_LOCK = Lock()
@@ -1001,6 +1074,91 @@ async def get_hii_rainfall_latest(
    return rows
@app.get("/api/hii/rainfall/catchment")
 async def get_hii_rainfall_catchment(
    response: Response, days: int = Query(14, ge=1, le=60)
 ):
    """Upper-Ping catchment-mean hourly rain: HII gauges vs the Open-Meteo
    series the flood model actually uses, plus their agreement over the window.
    Evidence-gathering endpoint (docs/FLOOD_FORECASTING.md, HII gauge rain):
    the gauge table only exists since 2026-08 so it cannot be a training
    feature yet; this makes the two sources' relationship observable meanwhile.
    """
    increment_counter("api_requests", labels={"endpoint": "hii_rain_catchment"})
    start = datetime.now() - timedelta(days=days)
    def compute():
        import pandas as pd
        from .ml import hii_rain
        engine = _hii_engine()
        if engine is None:
            return {
                "box": hii_rain.CATCHMENT_BOX,
                "gauge": [],
                "openmeteo": [],
                "comparison_24h_sums": {"overlap_hours": 0},
            }
        gauge = hii_rain.load_gauge_mean(start=pd.Timestamp(start), engine=engine)
        openmeteo = None
        try:
            from sqlalchemy import text
            with engine.connect() as conn:
                frame = pd.read_sql(
                    text(
                        "SELECT timestamp, catchment_mean FROM openmeteo_rain "
                        "WHERE timestamp >= :start ORDER BY timestamp"
                    ),
                    conn,
                    params={"start": start},
                )
            if not frame.empty:
                frame["timestamp"] = pd.to_datetime(frame["timestamp"])
                openmeteo = pd.to_numeric(
                    frame.set_index("timestamp")["catchment_mean"], errors="coerce"
                )
        except Exception as error:  # openmeteo_rain may not exist yet
            logger.warning(f"openmeteo_rain read failed: {error}")
        def series_rows(s):
            if s is None:
                return []
            return [
                {
                    "timestamp": ts.isoformat(),
                    "rain_mm": None if pd.isna(v) else round(float(v), 2),
                }
                for ts, v in s.items()
            ]
        comparison = (
            hii_rain.compare_with_openmeteo(gauge, openmeteo)
            if gauge is not None and openmeteo is not None
            else {"overlap_hours": 0}
        )
        return {
            "box": hii_rain.CATCHMENT_BOX,
            "gauge": series_rows(gauge),
            "openmeteo": series_rows(openmeteo),
            "comparison_24h_sums": comparison,
        }
    payload, stale = await _cached_swr(
        HII_CACHE,
        HII_CACHE_LOCK,
        _HII_COMPUTE_LOCKS["rain"],
        f"rain_catchment:{days}",
        Config.HII_CACHE_TTL_SECONDS,
        compute,
    )
    if stale:
        response.headers["X-Data-Stale"] = "true"
    return payload
@app.get("/api/hii/waterlevel/latest")
 async def get_hii_waterlevel_latest(
    response: Response, hours: int = Query(26, ge=1, le=168)
@@ -1043,9 +1201,7 @@ async def get_postgres_history(
            return cached[1]
    try:
        db_config = Config.get_database_config()
-        end_time = (
+        end_time = datetime.combine(end, datetime.max.time()) if end else datetime.now()
            datetime.combine(end, datetime.max.time()) if end else datetime.now()
        )
        start_time = (
            datetime.combine(start, datetime.min.time())
            if start
@@ -1138,17 +1294,85 @@ async def get_forecast_history(
    store = app_state.get("forecast_store")
    if not store:
        return []
-    end_dt = (
+    end_dt = datetime.combine(end, datetime.max.time()) if end else datetime.now()
        datetime.combine(end, datetime.max.time()) if end else datetime.now()
    )
    start_dt = (
        datetime.combine(start, datetime.min.time())
        if start
        else end_dt - timedelta(hours=hours)
    )
-    return await asyncio.to_thread(
+    return await asyncio.to_thread(store.fetch, station_code, start_dt, end_dt, horizon)
-        store.fetch, station_code, start_dt, end_dt, horizon
+
@app.get("/api/notifications")
 async def get_notifications_config():
    """Public ntfy settings so the dashboard can offer subscribe links."""
    server = Config.NTFY_SERVER
    if not server:
        return {"enabled": False}
    prefix = Config.NTFY_TOPIC_PREFIX
    return {
        "enabled": True,
        "server": server,
        "prefix": prefix,
        "topics": {
            "warning": f"{prefix}-warning",
            "danger": f"{prefix}-danger",
            "p1_outlook": f"{prefix}-p1-outlook",
            "status": f"{prefix}-status",
            "station_pattern": f"{prefix}-<station>-warning | {prefix}-<station>-danger (station code lowercase, no dot: p1, p103)",
        },
        "semantics": "transitions only: one message on crossing up, one all-clear on the way down (0.10 m hysteresis)",
    }
@app.get("/api/forecast/skill")
 async def get_forecast_skill(
    response: Response,
    station_code: str = Query("P.1"),
    horizon: int = Query(24, ge=1, le=48),
 ):
    """Is the model getting better? Issued forecasts verified against what the
    river then did, per model version, with a persistence baseline.
    Read from forecast_history (what each deployed version predicted, hourly)
    joined to water_measurements; no retraining involved. Cached like the HII
    feeds because the join is a few hundred correlated subqueries.
    """
    increment_counter("api_requests", labels={"endpoint": "forecast_skill"})
    store = app_state.get("forecast_store")
    if not store:
        return {
            "station_code": station_code,
            "horizon_hours": horizon,
            "versions": [],
            "current": None,
            "trend": None,
        }
    def compute():
        from .ml import skill
        if not store.engine and not store.connect():
            return {
                "station_code": station_code,
                "horizon_hours": horizon,
                "versions": [],
                "current": None,
                "trend": None,
            }
        return skill.compute_skill(store.engine, store.db_type, station_code, horizon)
    payload, stale = await _cached_swr(
        HII_CACHE,
        HII_CACHE_LOCK,
        _HII_COMPUTE_LOCKS["skill"],
        f"skill:{station_code}:{horizon}",
        max(Config.HII_CACHE_TTL_SECONDS, 900),
        compute,
    )
    if stale:
        response.headers["X-Data-Stale"] = "true"
    return payload
@app.get("/measurements/latest", response_model=List[MeasurementResponse])
@@ -1234,9 +1458,7 @@ async def get_database_stats():
        from sqlalchemy import text
        with engine.connect() as conn:
-            return conn.execute(
+            return conn.execute(text("""
                text(
                    """
                    SELECT (SELECT COUNT(*) FROM hii_rainfall) AS rain_n,
                           (SELECT COUNT(*) FROM hii_waterlevel) AS wl_n,
                           (SELECT COUNT(*) FROM hii_rain_stations) AS rain_s,
@@ -1245,9 +1467,7 @@ async def get_database_stats():
                           (SELECT MAX(timestamp) FROM hii_rainfall) AS rain_hi,
                           (SELECT MIN(timestamp) FROM hii_waterlevel) AS wl_lo,
                           (SELECT MAX(timestamp) FROM hii_waterlevel) AS wl_hi
-                    """
+                    """)).one()
                )
            ).one()
    def compute():
        # Heavy: full-table counts and coverage over ~1.7M rows. Runs at most
@@ -233,6 +233,75 @@ def test_heuristic_fallback(tmp_path):
        assert row["trained_at"] is None
 def _p1_synth(n: int = 300, seed: int = 11) -> pd.DataFrame:
    upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
    return make_synth(n, ["P.1"] + upstream, seed=seed, pulses={"P.1": [(100, 20, 2.0)]})
 def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
    """use_rain=True with no rain series must abort, not write v2 bundles.
    Regression for the 2026-09-01 server retrain that overwrote v3 with v2
    because the Open-Meteo archive fetch failed on a cache-less checkout.
    """
    from src.ml import rain as rain_mod
    df = _p1_synth()
    overrides = {"max_iter": 10}
    # Case 1: the loader returns None (archive unreachable, no cache file)
    monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: None)
    with pytest.raises(train.RainUnavailableError, match="--no-rain"):
        train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
    assert not (tmp_path / "flood_P.1.joblib").exists()
    assert not (tmp_path / "metrics.json").exists()
    # Case 2: the loader raises (network / parse error)
    def boom(*a, **k):
        raise ConnectionError("simulated Open-Meteo outage")
    monkeypatch.setattr(rain_mod, "load_history", boom)
    with pytest.raises(train.RainUnavailableError, match="simulated Open-Meteo outage"):
        train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
    assert not (tmp_path / "flood_P.1.joblib").exists()
    # Explicit opt-out still produces v2 bundles as before
    metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=False, use_dam=False)
    assert metrics["model_version"].startswith("hgb-v2+")
    assert (tmp_path / "flood_P.1.joblib").exists()
 def test_train_with_rain_series_yields_v3(tmp_path, monkeypatch):
    from src.ml import rain as rain_mod
    df = _p1_synth()
    idx = pd.date_range(df["timestamp"].min(), df["timestamp"].max(), freq="h")
    fake_rain = pd.DataFrame({"a": np.linspace(0, 1, len(idx)), "b": 0.5}, index=idx)
    monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: fake_rain)
    metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 10}, use_rain=True, use_dam=False)
    assert metrics["model_version"].startswith("hgb-v3+")
    bundle = joblib.load(tmp_path / "flood_P.1.joblib")
    assert set(features.RAIN_FEATURES) <= set(bundle["feature_names"])
 def test_cli_exit_code_on_rain_failure(tmp_path, monkeypatch, caplog):
    """The console entry turns the guard into a one-line error and exit 2."""
    from src.ml import rain as rain_mod
    df = _p1_synth()
    monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: None)
    monkeypatch.setattr(train, "load_measurements", lambda *a, **k: df)
    monkeypatch.setattr(train, "resolve_db_url", lambda *a, **k: None)
    monkeypatch.setattr(
        "sys.argv",
        ["train", "--stations", "P.1", "--models-dir", str(tmp_path), "--skip-eval"],
    )
    assert train.cli() == 2
    assert "refusing to silently downgrade" in caplog.text
    assert not (tmp_path / "metrics.json").exists()
 def test_feature_name_stability(tmp_path):
    upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
    data_stations = ["P.1"] + upstream
@@ -0,0 +1,93 @@
 """Forecast skill verification: issued forecasts vs observed peaks (sqlite)."""
 import datetime
 import pytest
 from sqlalchemy import create_engine, text
 from src.ml import skill
@pytest.fixture
 def engine(tmp_path):
    eng = create_engine(f"sqlite:///{tmp_path / 'skill.db'}")
    with eng.begin() as c:
        c.execute(text("CREATE TABLE stations (id INTEGER PRIMARY KEY, station_code TEXT)"))
        c.execute(text("INSERT INTO stations VALUES (1, 'P.1')"))
        c.execute(
            text(
                "CREATE TABLE water_measurements (timestamp DATETIME, station_id INTEGER, water_level REAL)"
            )
        )
        c.execute(
            text(
                "CREATE TABLE forecast_history (as_of TIMESTAMP, station_code TEXT, horizon_hours INTEGER, "
                "predicted_max_level REAL, p_warning REAL, p_danger REAL, current_level REAL, "
                "model_version TEXT, source TEXT)"
            )
        )
    return eng
 def _fill(engine, start, hours, level_fn, forecasts):
    """hours of hourly observations from `start`, plus (as_of_offset_h, version, pred) rows."""
    with engine.begin() as c:
        for h in range(hours):
            ts = start + datetime.timedelta(hours=h)
            c.execute(
                text("INSERT INTO water_measurements VALUES (:t, 1, :l)"),
                {"t": ts, "l": level_fn(h)},
            )
        for off, version, pred in forecasts:
            ts = start + datetime.timedelta(hours=off)
            c.execute(
                text(
                    "INSERT INTO forecast_history VALUES (:t, 'P.1', 24, :p, 0, 0, :cur, :v, 'model')"
                ),
                {"t": ts, "p": pred, "cur": level_fn(off), "v": version},
            )
 def test_skill_per_version_and_trend(engine):
    start = datetime.datetime(2026, 8, 1)
    # river: flat 1.5 m, with a bump to 2.4 m around hour 100
    level = lambda h: 2.4 if 96 <= h <= 104 else 1.5
    forecasts = []
    # old version: always predicts 1.5 (persistence-like, misses the bump)
    for off in range(0, 60):
        forecasts.append((off, "hgb-v2+aaaaaaa", 1.5))
    # new version: predicts 1.5 normally and 2.3 ahead of the bump
    for off in range(60, 200):
        pred = 2.3 if 72 <= off <= 104 else 1.5
        forecasts.append((off, "hgb-v3+bbbbbbb", pred))
    _fill(engine, start, 260, level, forecasts)
    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=300))
    assert [v["model_version"] for v in out["versions"]] == ["hgb-v2+aaaaaaa", "hgb-v3+bbbbbbb"]
    old, new = out["versions"]
    assert old["n"] == 60 and old["enough_data"]
    assert new["n"] == 140 and new["enough_data"]
    # the old version issued only on flat hours: perfect there, no bump rows
    assert old["mae_m"] == 0.0 and old["above_2m_n"] == 0
    # the new version saw the bump: nonzero MAE but positive skill vs persistence
    assert new["above_2m_n"] > 0
    assert new["skill"] is not None and new["skill"] > 0
    assert out["current"]["model_version"] == "hgb-v3+bbbbbbb"
    assert out["trend"]["previous_version"] == "hgb-v2+aaaaaaa"
    assert out["trend"]["better"] is False  # honest: old had an easier period
 def test_skill_requires_full_window(engine):
    start = datetime.datetime(2026, 8, 1)
    # forecasts issued at the very end have no observed window yet
    _fill(engine, start, 30, lambda h: 1.5, [(o, "hgb-v3+ccccccc", 1.5) for o in range(0, 30)])
    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=30))
    # only as_of <= now-24h AND with >= 18 observed hours in the window count
    assert out["versions"] and out["versions"][0]["n"] == 7  # as_of 0..6 h: <= now-24h with >= 18 observed hours
    assert out["versions"][0]["enough_data"] is False
    assert out["trend"] is None
 def test_skill_empty(engine):
    out = skill.compute_skill(engine, "sqlite", "P.1", 24)
    assert out["versions"] == [] and out["current"] is None and out["trend"] is None
@@ -353,6 +353,29 @@ class TestHiiApiEndpoints:
        self._get(web_api, "get_hii_rainfall_latest", hours=48)
        assert calls["n"] == 2
    def test_rainfall_catchment(self, web_api):
        """One gauge in the box (CHM005, 19.12N 98.94E) is below the
        MIN_GAUGES_PER_HOUR floor, so the catchment mean is NaN -> null, the
        openmeteo_rain table does not exist in this store, and the comparison
        reports no overlap. Shape is what matters: the endpoint must not 500
        on a fresh database."""
        payload, response = self._get(web_api, "get_hii_rainfall_catchment", days=7)
        assert "x-data-stale" not in response.headers
        assert list(payload) == ["box", "gauge", "openmeteo", "comparison_24h_sums"]
        assert payload["openmeteo"] == []
        assert payload["comparison_24h_sums"] == {"overlap_hours": 0}
        assert len(payload["gauge"]) == 1
        assert payload["gauge"][0]["rain_mm"] is None  # < MIN_GAUGES_PER_HOUR
    def test_rainfall_catchment_disabled(self, monkeypatch):
        from src import web_api
        monkeypatch.setitem(web_api.app_state, "hii_collector", None)
        web_api.HII_CACHE.clear()
        web_api._REFRESH_IN_FLIGHT.clear()
        payload, _ = self._get(web_api, "get_hii_rainfall_catchment", days=7)
        assert payload["gauge"] == [] and payload["openmeteo"] == []
    def test_stale_served_on_recompute_failure(self, web_api, monkeypatch):
        # Prime the cache, expire it, break the DB: the stale copy is served
        # and flagged via the X-Data-Stale header.
@@ -0,0 +1,47 @@
 """HII gauge-rain aggregate: pure-function tests (no DB)."""
 import numpy as np
 import pandas as pd
 from src.ml import hii_rain
 def _hourly(start, n):
    return pd.date_range(start, periods=n, freq="h")
 def test_compare_identical_series_has_zero_bias():
    idx = _hourly("2026-08-12", 200)
    rng = np.random.default_rng(1)
    rain = pd.Series(rng.exponential(0.5, len(idx)), index=idx)
    out = hii_rain.compare_with_openmeteo(rain, rain.copy(), window_h=24)
    assert out["overlap_hours"] == 200
    assert out["bias_mm"] == 0.0
    assert out["mae_mm"] == 0.0
    assert out["corr"] > 0.999
 def test_compare_reports_constant_bias():
    idx = _hourly("2026-08-12", 100)
    gauge = pd.Series(1.0, index=idx)
    model = pd.Series(1.5, index=idx)  # model wetter by 0.5 mm/h
    out = hii_rain.compare_with_openmeteo(gauge, model, window_h=24)
    assert abs(out["bias_mm"] - 12.0) < 1e-9  # 0.5 mm/h x 24 h
 def test_compare_uses_overlap_only():
    gauge = pd.Series(1.0, index=_hourly("2026-08-12", 100))
    model = pd.Series(1.0, index=_hourly("2026-08-14", 100))  # 52 h overlap
    out = hii_rain.compare_with_openmeteo(gauge, model, window_h=24)
    assert out["overlap_hours"] == 52
 def test_compare_no_overlap():
    gauge = pd.Series(1.0, index=_hourly("2026-01-01", 10))
    model = pd.Series(1.0, index=_hourly("2026-06-01", 10))
    assert hii_rain.compare_with_openmeteo(gauge, model) == {"overlap_hours": 0}
 def test_load_gauge_mean_without_db_returns_none(monkeypatch):
    monkeypatch.setattr(hii_rain, "resolve_db_url", lambda *a, **k: None)
    assert hii_rain.load_gauge_mean() is None
@@ -0,0 +1,285 @@
 """ntfy notification state machine: transitions only, hysteresis, restart-safe."""
 import datetime
 import pytest
 from src import notify
 class FakePublisher(notify.NtfyPublisher):
    def __init__(self):
        super().__init__("http://ntfy.test", prefix="ping")
        self.sent = []
    def publish(self, n):
        self.sent.append(n)
        return True
@pytest.fixture
 def pub():
    return FakePublisher()
 def _reading(code, level, ts="2026-09-24T12:00:00"):
    return {"station_code": code, "water_level": level, "timestamp": ts}
 def _fc(p, peak=None):
    return [
        {
            "station_code": "P.1",
            "horizon_hours": 24,
            "p_warning": p,
            "predicted_max_level": peak,
            "source": "model",
        }
    ]
 NOW = datetime.datetime(2026, 9, 24, 12, 30)
 def topics(pub):
    return [n.topic for n in pub.sent]
 def test_quiet_river_sends_nothing(pub):
    state = notify.InMemoryState()
    for h in range(48):
        notify.evaluate(
            [_reading("P.1", 1.6), _reading("P.103", 3.2)],
            _fc(0.01),
            state,
            pub,
            now=NOW,
        )
    assert pub.sent == []
 def test_warning_crossing_once_then_silence_then_clear(pub):
    state = notify.InMemoryState()
    # rising through 3.70 (P.1 warning)
    notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
    assert pub.sent == []
    notify.evaluate([_reading("P.1", 3.72)], [], state, pub, now=NOW)
    assert topics(pub) == ["ping-p1-warning", "ping-warning"]
    assert pub.sent[0].priority == 4 and "3.72 m" in pub.sent[0].message
    # stays above: no repeats for many hours
    for level in (3.80, 3.95, 4.05, 3.90, 3.75):
        notify.evaluate([_reading("P.1", level)], [], state, pub, now=NOW)
    assert len(pub.sent) == 2
    # dips to 3.65: within hysteresis, still no message
    notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
    assert len(pub.sent) == 2
    # 3.55: clear
    notify.evaluate([_reading("P.1", 3.55)], [], state, pub, now=NOW)
    assert topics(pub)[2:] == ["ping-p1-warning", "ping-warning"]
    assert "back to normal" in pub.sent[2].title
 def test_danger_escalation_and_deescalation(pub):
    state = notify.InMemoryState()
    notify.evaluate([_reading("P.1", 3.9)], [], state, pub, now=NOW)  # warning
    notify.evaluate(
        [_reading("P.1", 4.25)], [], state, pub, now=NOW
    )  # danger (>= 4.20)
    assert topics(pub) == [
        "ping-p1-warning",
        "ping-warning",
        "ping-p1-danger",
        "ping-danger",
    ]
    assert pub.sent[2].priority == 5
    notify.evaluate(
        [_reading("P.1", 4.15)], [], state, pub, now=NOW
    )  # hysteresis: still danger
    assert len(pub.sent) == 4
    notify.evaluate([_reading("P.1", 4.05)], [], state, pub, now=NOW)  # back to warning
    assert topics(pub)[4:] == ["ping-p1-danger", "ping-warning"]
    assert "below danger" in pub.sent[4].title
 def test_jump_straight_to_danger(pub):
    state = notify.InMemoryState()
    notify.evaluate(
        [_reading("P.103", 7.0)], [], state, pub, now=NOW
    )  # P.103 danger 6.75
    assert topics(pub) == ["ping-p103-danger", "ping-danger"]
 def test_basin_digest_groups_stations(pub):
    state = notify.InMemoryState()
    notify.evaluate(
        [_reading("P.1", 3.8), _reading("P.103", 6.0), _reading("P.67", 1.0)],
        [],
        state,
        pub,
        now=NOW,
    )
    basin = [n for n in pub.sent if n.topic == "ping-warning"]
    assert len(basin) == 1 and "P.1" in basin[0].message and "P.103" in basin[0].message
 def test_outlook_on_off_with_hysteresis(pub):
    state = notify.InMemoryState()
    r = [_reading("P.1", 2.9)]
    notify.evaluate(r, _fc(0.30), state, pub, now=NOW)
    assert pub.sent == []
    notify.evaluate(r, _fc(0.55, 3.9), state, pub, now=NOW)
    assert topics(pub) == ["ping-p1-outlook"]
    assert "55%" in pub.sent[0].message and "3.90 m" in pub.sent[0].message
    assert "not an official warning" in pub.sent[0].message
    notify.evaluate(
        r, _fc(0.40), state, pub, now=NOW
    )  # between OFF and ON: stays on, silent
    assert len(pub.sent) == 1
    notify.evaluate(r, _fc(0.20), state, pub, now=NOW)
    assert len(pub.sent) == 2 and "easing" in pub.sent[1].title
 def test_heuristic_forecast_ignored(pub):
    state = notify.InMemoryState()
    fc = [
        {
            "station_code": "P.1",
            "horizon_hours": 24,
            "p_warning": 0.9,
            "source": "heuristic",
        }
    ]
    notify.evaluate([_reading("P.1", 2.0)], fc, state, pub, now=NOW)
    assert pub.sent == []
 def test_stale_feed_and_recovery(pub):
    state = notify.InMemoryState()
    notify.evaluate(
        [_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=NOW
    )
    assert pub.sent == []
    later = NOW + datetime.timedelta(hours=4)
    notify.evaluate(
        [_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=later
    )
    assert topics(pub) == ["ping-status"] and "stale" in pub.sent[0].title
    notify.evaluate(
        [_reading("P.1", 1.6, "2026-09-24T12:00:00")],
        [],
        state,
        pub,
        now=later + datetime.timedelta(hours=1),
    )
    assert len(pub.sent) == 1  # still stale, no repeat
    notify.evaluate(
        [_reading("P.1", 1.6, "2026-09-24T17:00:00")],
        [],
        state,
        pub,
        now=later + datetime.timedelta(hours=1),
    )
    assert len(pub.sent) == 2 and "recovered" in pub.sent[1].title
 def test_capacity_guard_blocks_stale_threshold(pub):
    """P.77 2026-09: 3.02 m >= 2.85 m 'warning' at 22 % capacity -> not a flood."""
    state = notify.InMemoryState()
    r = {
        "station_code": "P.77",
        "water_level": 4.40,
        "timestamp": "2026-09-24T12:00:00",
        "discharge_percent": 10.3,
    }
    notify.evaluate([r], [], state, pub, now=NOW)
    assert pub.sent == [] and state.get("level:P.77") is None
    # same level with capacity agreeing -> alert
    r["discharge_percent"] = 82.0
    notify.evaluate([r], [], state, pub, now=NOW)
    assert topics(pub) == ["ping-p77-warning", "ping-warning"]
 def test_capacity_guard_exempts_p1_and_missing_pct(pub):
    state = notify.InMemoryState()
    notify.evaluate(
        [
            {
                "station_code": "P.1",
                "water_level": 3.75,
                "timestamp": "2026-09-24T12:00:00",
                "discharge_percent": 40.0,
            }
        ],
        [],
        state,
        pub,
        now=NOW,
    )
    assert topics(pub) == ["ping-p1-warning", "ping-warning"]
    pub.sent.clear()
    notify.evaluate(
        [
            {
                "station_code": "P.103",
                "water_level": 6.0,
                "timestamp": "2026-09-24T12:00:00",
            }
        ],
        [],
        state,
        pub,
        now=NOW,
    )
    assert topics(pub) == ["ping-p103-warning", "ping-warning"]
 def test_capacity_guard_does_not_block_clearing(pub):
    """Guard applies only to the clear->alert edge; the all-clear always goes out."""
    state = notify.InMemoryState()
    r = {
        "station_code": "P.67",
        "water_level": 2.6,
        "timestamp": "2026-09-24T12:00:00",
        "discharge_percent": 90.0,
    }
    notify.evaluate([r], [], state, pub, now=NOW)
    assert len(pub.sent) == 2
    r.update(water_level=2.2, discharge_percent=30.0)
    notify.evaluate([r], [], state, pub, now=NOW)
    assert "back to normal" in pub.sent[2].title
 def test_state_survives_restart_via_sql(tmp_path, pub):
    from sqlalchemy import create_engine
    eng = create_engine(f"sqlite:///{tmp_path / 'n.db'}")
    state = notify.NotificationState(eng, "sqlite")
    notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
    assert len(pub.sent) == 2
    # "restart": new state object on the same DB, same reading -> nothing re-sent
    state2 = notify.NotificationState(eng, "sqlite")
    notify.evaluate([_reading("P.1", 3.8)], [], state2, pub, now=NOW)
    assert len(pub.sent) == 2
 def test_publish_failure_does_not_advance_state():
    """If ntfy is down the transition must be retried next cycle, not lost."""
    class Down(notify.NtfyPublisher):
        def __init__(self):
            super().__init__("http://ntfy.test")
            self.calls = 0
        def publish(self, n):
            self.calls += 1
            return False
    pub = Down()
    state = notify.InMemoryState()
    notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
    assert pub.calls == 2 and state.get("level:P.1") is None
    # next cycle, ntfy back: the crossing is delivered
    good = FakePublisher()
    notify.evaluate([_reading("P.1", 3.8)], [], state, good, now=NOW)
    assert topics(good) == ["ping-p1-warning", "ping-warning"]
    assert state.get("level:P.1") == "warning"
Author	SHA1	Message	Date
grabowski	32399f1899	feat: recalibrate P.77/P.75 thresholds; capacity guard on alerts CI / Format & lint (push) Successful in 10s Details CI / Test suite (push) Successful in 17s Details Security / Static analysis (push) Successful in 12s Details Security / License report (push) Successful in 45s Details Security / Dependency vulnerabilities (push) Successful in 49s Details The first ntfy cycle announced "Warning level at P.77" at 3.02 m. That gauge's 2.85 m threshold sat below its own dry-season baseline (2.6-2.7 m at 8-14 % channel capacity): P.77 had been "above warning" for 761 of the last 2 146 hours, at 22 % capacity. Across 2018-2024, 75-85 % capacity reads 3.35-4.57 m and 95-105 % reads 4.27-5.08 m; set 4.30 / 4.90. P.75 moved 2.75/3.50 -> 3.20/3.65 on the same evidence (2024: 3.45 / 3.72). The predictor already handles changed thresholds (regression-derived probabilities until the Oct 1 retrain). Second line of defence in notify.py: a clear->alert transition is only announced when RID's discharge_percent for the reading is >= 60 %, so a re-rated or datum-shifted gauge cannot page subscribers again. P.1 is exempt (its stages come from the inundation map, not capacity); readings without a capacity figure fall back to level only; the all-clear edge is never blocked. 3 tests.	2026-09-12 00:34:59 +02:00
grabowski	f4d42c90f4	fix: ntfy listens on the Tailscale address; monitor publishes to it directly Security / Dependency vulnerabilities (push) Successful in 44s Details Security / Static analysis (push) Successful in 9s Details CI / Format & lint (push) Successful in 10s Details CI / Test suite (push) Successful in 26s Details Security / License report (push) Successful in 50s Details Docs / Validate documentation (push) Successful in 16s Details The reverse proxy is a separate VPS on the tailnet, so a loopback-only ntfy was unreachable from it. install_ntfy.sh now binds the host's Tailscale IP (NTFY_LISTEN overrides). New NTFY_PUBLISH_URL: where the monitor POSTs, separate from the public NTFY_SERVER subscribers see, so an alert never waits on DNS or the proxy (first cycle logged 502s from Cloudflare while the domain was not yet proxied).	2026-09-12 00:28:29 +02:00
grabowski	039d24a5c3	fix: init ntfy only in the collection leader CI / Test suite (push) Successful in 29s Details Docs / Validate documentation (push) Successful in 13s Details CI / Format & lint (push) Successful in 15s Details Security / Dependency vulnerabilities (push) Successful in 42s Details Security / License report (push) Successful in 48s Details Security / Static analysis (push) Successful in 9s Details Every uvicorn worker ran the notification_state DDL at startup; on Postgres the losers of that race get UniqueViolation on pg_type and the whole init was skipped (notifications off). Only the leader publishes, so only the leader initialises, after election. The DDL also tolerates a concurrent creator now: on failure it verifies the table exists instead of giving up.	2026-09-12 00:22:03 +02:00
grabowski	777b230baf	feat: public flood notifications over self-hosted ntfy CI / Format & lint (push) Successful in 11s Details Security / Static analysis (push) Successful in 12s Details CI / Test suite (push) Successful in 18s Details Docs / Validate documentation (push) Successful in 11s Details Security / Dependency vulnerabilities (push) Successful in 1m30s Details Security / License report (push) Successful in 47s Details Anyone can now get push alerts on their phone without an account: the monitor publishes to an ntfy server (one Go binary, ~30 MB RSS) and subscribers pick topics in the free iOS/Android/web app. Semantics are transitions, never state. One message when a gauge crosses its warning or danger threshold, one all-clear when it drops back (0.10 m hysteresis), nothing while it sits above. A three-day flood is two messages; a quiet season is zero. Topics: ping-warning / ping-danger (basin digest), ping-<station>-warning / -danger, ping-p1-outlook (opt-in: model P(warning within 24 h) at P.1 rises through 50 %, clears below 25 %, message says it is experimental), ping-status (feed stale >= 3 h / recovered). Priority 5 on danger so it rings through Do Not Disturb. src/notify.py runs once per collection cycle in the API process (leader only, after the forecast precompute, same data the dashboard shows). Last-sent state lives in a notification_state table so a restart never re-sends; a failed publish leaves state untouched so the crossing is retried next cycle instead of lost. Off unless NTFY_SERVER is set. Dashboard: a "Get alerts" button (only when configured) opens a panel with the server, per-topic cards, ntfy:// deep links and web links, app store links and a disclaimer. EN + TH. GET /api/notifications feeds it. scripts/install_ntfy.sh: .deb install, server.yml (loopback listen, anonymous read, token-only write scoped to ping-*, 72 h cache, signup/ login/metrics off, tight visitor limits), systemd, user + token, .env. Verified against ntfy 2.28.0: anon publish 403, token publish 200, token on foreign topic 403, anon read 200, and a seeded crossing through the real _notify_transitions path arrived in the topic with priority, tags, click and action button. docs/NOTIFICATIONS.md has the deployment and reverse-proxy notes. Tests: 10 for the state machine (159 total).	2026-09-12 00:18:38 +02:00
grabowski	0ec675e9c5	ci: license report from a clean venv, not the runner's site-packages CI / Format & lint (push) Successful in 10s Details Security / Dependency vulnerabilities (push) Successful in 53s Details Security / Static analysis (push) Successful in 10s Details CI / Test suite (push) Successful in 20s Details Security / License report (push) Successful in 50s Details	2026-09-11 23:54:44 +02:00
grabowski	7b31d4d0dd	feat: "Is the model getting better?" - live verification per model version CI / Test suite (push) Successful in 22s Details CI / Format & lint (push) Successful in 16s Details Docs / Validate documentation (push) Successful in 10s Details Security / Dependency vulnerabilities (push) Successful in 1m34s Details Security / Static analysis (push) Successful in 10s Details Security / License report (push) Successful in 13s Details src/ml/skill.py joins forecast_history (what each deployed version predicted for the 24 h peak, hourly) to water_measurements (what the river did) and reports per version: verified hours, peak MAE, bias, the persistence baseline (peak = current level), skill = 1 - MAE/persistence, and the same MAE restricted to observed peaks >= 2 m. Only forecasts whose window has elapsed with >= 75 % of hours observed count; a version needs 24 verified hours before it is compared. GET /api/forecast/skill?station_code=P.1&horizon=24 returns it (SWR cached, 15 min). The dashboard's forecast card gains a panel with a one-line verdict (current vs previous version), the per-version table, and a caveat that quiet weeks measure quiet-river accuracy only: the model is judged on flood-onset lead, which the backtests cover. EN + TH. On today's production data: hgb-v3+28b62e5 (369 h, Aug 13 - Sep 1) MAE 15.2 cm, skill -0.05; hgb-v2+f6570ac (224 h, Sep 1 - 11) MAE 12.3 cm, skill 0.36 - the "worse" v2 model scores better on a quieter fortnight, which is exactly why the panel shows the >= 2 m column and the caveat. Tests: 3, sqlite, synthetic. scripts/dev_proxy.py: DEV_PROXY_LOCAL lets a not-yet-deployed endpoint be answered from a local JSON file while everything else goes to prod.	2026-09-11 23:44:44 +02:00
grabowski	2e19974fad	docs: README describes the project that exists; CLAUDE.md for agents The README was the original template: P.1 "in Nakhon Sawan", P.103 "in Bangkok", VictoriaMetrics as the recommended database, Docker/Grafana sections, github.com/your-username support links. Rewritten around what runs: sources, gap fill, the forecast and its 13 h result, the live API table, systemd deployment with the retrain timer, repository layout, real docs links, data-source credits. Other DB adapters are mentioned as supported-but-not-production. CLAUDE.md replaces the untracked Ruflo boilerplate with project rules: Python 3.11, format/test gates, Bangkok timestamps, harness-first model changes judged on lead, the RainUnavailableError guard, no git add -A.	2026-09-11 23:44:43 +02:00
grabowski	b03318210c	security: pip-audit + bandit gates that can fail; patch 29 known CVEs security.yml previously ran safety/bandit/semgrep with `\|\| true` and could not go red. Now: pip-audit on requirements.txt is a hard gate (dev deps reported only), bandit HIGH fails (B104 bind-all skipped: intended behind Cloudflare/Caddy), pip-licenses uploaded as a report. Weekly + on dependency/source changes. Running it locally found 29 advisories, all in pinned-and-forgotten runtime deps: starlette 0.27 (7, incl. Host-header path confusion and form DoS), fastapi 0.104, requests 2.31 (3), pymysql 1.1. Bumped to current: fastapi 0.141.1 / starlette 1.6.0, pydantic 2.13.5, uvicorn 0.52.4, requests 2.34.2, pymysql 1.2.0; dev: pytest 9.1.1, black 26.5.1. pip-audit is now clean. requires-python narrowed to 3.11 (the truth: psycopg2-binary 2.9.9 fails on 3.13; pandas 2.0.3 has no 3.12 wheels). Full suite passes; API smoke-tested (health, stations, forecast, history, stats, docs, openapi) on the new stack. black 26 reformatted 8 files.	2026-09-11 23:44:43 +02:00
grabowski	97a6694ab2	feat(dashboard): light/dark theme CI / Format & lint (push) Successful in 36s Details CI / Test suite (push) Successful in 27s Details Docs / Validate documentation (push) Successful in 12s Details Sun/moon button next to the language switch. Follows the OS preference until the user picks one (persisted in localStorage, tracks OS changes only while unpinned). Every colour that was a literal white / grey is now a token with a dark counterpart; flood-verdict, live-pill and lang-toggle states got their own bg/ink/border tokens. OSM tiles are inverted with a hue rotate so roads and labels stay legible while the river network, markers and rain dots (SVG, unfiltered) keep their data colours. Chart.js reads grid/tick/legend colours from the tokens and the open chart is rebuilt on toggle. Leaflet popups and controls follow the theme.	2026-09-11 23:16:30 +02:00
grabowski	5ad8e4eac3	ci: green pipelines that check what exists; one formatting contract The Test Suite job failed on every push since the black check was added because the tree had never been formatted, and pre-commit said 120 columns while CI ran black's default 88. pyproject.toml now carries [tool.black] / [tool.isort] (88, black profile) as the single source; pre-commit reads it; `make format` applied it (13 files, whitespace only, 146 insertions / 128 deletions, tests unchanged at 146 passed). ci.yml: lint (black, isort, flake8 hard errors) + pytest. The Docker registry push, VictoriaMetrics integration test, staging/production deploy and Apache-Bench jobs were template scaffolding for hosts and registries that do not exist; production is a systemd unit updated by git pull. Removed rather than left permanently skipped. docs.yml: the "Check markdown links" step curl'd every URL in every .md and failed on localhost examples and the Tailscale IP, and the Sphinx jobs built artifacts nobody read. Replaced by two checks that mean something: relative links/images in README, CONTRIBUTING and docs/ resolve inside the repo, and the FastAPI OpenAPI schema exports with the documented endpoints present (uploaded as an artifact).	2026-09-11 23:05:37 +02:00
grabowski	6f4a86edbb	fix(dashboard): timestamps are Asia/Bangkok everywhere; stale feed says so The API emits naive ICT timestamps ("2026-09-12T02:00:00"). The page fed them to new Date(), which applies the BROWSER's zone: a viewer in Europe parsed a 02:00 ICT reading as 02:00 CEST, five hours in the future, so "Last updated" clamped to "0 min ago" forever; a viewer in the Americas saw thousands of minutes. parseTs() now pins +07:00 on naive strings and every display formats with timeZone: Asia/Bangkok, so the site shows river time regardless of where it is opened. Daily/hourly chart buckets key on the Bangkok calendar day instead of UTC getters. The tile also gets a real stale state: past 3 h (RID is hourly) it turns red, reads "Feed stale · last reading <day>, N h ago", and the header pill switches from LIVE DATA to STALE FEED. Age shows hours past 2 h. scripts/dev_proxy.py serves the working-copy dashboard with API calls proxied to water.buildfor.life so browser-side changes can be checked against live data (and any browser timezone) before deploying.	2026-09-11 23:00:39 +02:00
grabowski	ce08312c0f	docs: public dashboard URL (water.buildfor.life) replaces the Tailscale IP everywhere CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 32s Details Documentation / Documentation Summary (push) Successful in 3s Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 15s Details Documentation / Validate Documentation (push) Failing after 16s Details Documentation / Generate API Documentation (push) Successful in 11s Details Documentation / Build Sphinx Documentation (push) Successful in 18s Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s Details	2026-09-11 22:38:13 +02:00
grabowski	d621aa9ce7	eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 41s Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s Details Documentation / Validate Documentation (push) Failing after 16s Details Documentation / Generate API Documentation (push) Successful in 11s Details Documentation / Build Sphinx Documentation (push) Successful in 18s Details CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s Details Documentation / Documentation Summary (push) Successful in 3s Details Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw, rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all opt-in, plus --from-cache for reproducible offline reruns. Results in models/eval_2026-09-12.json, write-up in docs/FLOOD_FORECASTING.md: - quantile point prediction: better MAE, worse first-alert lead at 5 of 11 events (P.103 2022-08-14 +6h -> +1h) -> rejected - quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads - rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1 -> deferred to after the 2026 season src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a training feature (table exists only since 2026-08-11, no archive); exposed at GET /api/hii/rainfall/catchment so the two sources' agreement is on record by the time a fold can test it. data._read_cache now skips non-station files in models/cache/ (the shared dir also holds rain_openmeteo / dam_ caches, which crashed the reader). scripts/summarize_eval.py prints per-variant lead/peak-error tables.	2026-09-11 21:55:37 +02:00
grabowski	764764e07e	feat: refuse silent v3->v2 downgrade; monthly retrain timer with staged promote train_all() now raises RainUnavailableError when use_rain=True and the Open-Meteo history cannot be loaded, instead of logging a warning and writing gauge-only (v2) bundles over the deployed v3 set -- which is what the 2026-09-01 server retrain did unnoticed. --no-rain remains the explicit way to get v2. CLI exits 2 with a one-line error. Three tests cover the guard, the opt-out, and the v3 happy path. scripts/retrain.sh trains into models/.staging, refuses to promote unless metrics.json shows hgb-v3+ and >=14 trained stations, then renames bundles into place (previous generation kept in models/.previous). No API restart: predict.py reloads by mtime on the hourly precompute. water-monitor-retrain.{service,timer}: 1st of each month 03:30, Persistent, OMP_NUM_THREADS=4, Nice=15, same sandbox as the API unit. install.sh now does `uv sync` into .venv (one env rule; removes a stale venv/) and enables the timer. water-monitor.service in the repo matched neither the deployed unit nor the uv env; it now does (run.py --web-api, .venv, EnvironmentFile).	2026-09-11 21:37:11 +02:00