feat: recalibrate P.77/P.75 thresholds; capacity guard on alerts

The first ntfy cycle announced "Warning level at P.77" at 3.02 m. That
gauge's 2.85 m threshold sat below its own dry-season baseline (2.6-2.7 m
at 8-14 % channel capacity): P.77 had been "above warning" for 761 of the
last 2 146 hours, at 22 % capacity. Across 2018-2024, 75-85 % capacity
reads 3.35-4.57 m and 95-105 % reads 4.27-5.08 m; set 4.30 / 4.90. P.75
moved 2.75/3.50 -> 3.20/3.65 on the same evidence (2024: 3.45 / 3.72).
The predictor already handles changed thresholds (regression-derived
probabilities until the Oct 1 retrain).

Second line of defence in notify.py: a clear->alert transition is only
announced when RID's discharge_percent for the reading is >= 60 %, so a
re-rated or datum-shifted gauge cannot page subscribers again. P.1 is
exempt (its stages come from the inundation map, not capacity); readings
without a capacity figure fall back to level only; the all-clear edge is
never blocked. 3 tests.

.env.back

@@ -1,87 +0,0 @@
-# Northern Thailand Ping River Monitor Configuration
-# Copy this file to .env and customize for your environment
-
-# Database Configuration
-DB_TYPE=postgresql
-# Options: sqlite, mysql, postgresql, influxdb, victoriametrics
-
-# SQLite Configuration (default)
-WATER_DB_PATH=water_levels.db
-
-# VictoriaMetrics Configuration
-VM_HOST=localhost
-VM_PORT=8428
-VM_URL=
-
-# InfluxDB Configuration
-INFLUX_HOST=localhost
-INFLUX_PORT=8086
-INFLUX_DATABASE=ping_river_monitoring
-INFLUX_USERNAME=
-INFLUX_PASSWORD=
-
-# PostgreSQL Configuration (Remote Server)
-# Option 1: Full connection string (URL encode special characters in password)
-#POSTGRES_CONNECTION_STRING=postgresql://username:url_encoded_password@your-postgres-host:5432/water_monitoring
-
-# Option 2: Individual components (password will be automatically URL encoded)
-POSTGRES_HOST=10.0.10.201
-POSTGRES_PORT=5432
-POSTGRES_DB=ping_river
-POSTGRES_USER=ping_river
-POSTGRES_PASSWORD=3_%m]k:+16"rx?M#`swIA
-
-# Examples for connection string:
-# - Local: postgresql://postgres:password@localhost:5432/water_monitoring
-# - Remote: postgresql://user:pass@192.168.1.100:5432/water_monitoring
-# - With special chars: postgresql://user:my%3Apass%40word@host:5432/db
-# - With SSL: postgresql://user:pass@host:port/db?sslmode=require
-# - Connection pooling: postgresql://user:pass@host:port/db?pool_size=20&max_overflow=0
-
-# Special character URL encoding:
-# : → %3A    @ → %40    # → %23    ? → %3F    & → %26    / → %2F    % → %25
-
-# MySQL Configuration
-MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/ping_river_monitoring
-
-# API Configuration
-API_HOST=0.0.0.0
-API_PORT=8000
-API_WORKERS=1
-
-# Data Collection Settings
-SCRAPING_INTERVAL_HOURS=1
-REQUEST_TIMEOUT=30
-MAX_RETRIES=3
-RETRY_DELAY_SECONDS=60
-
-# Data Retention
-DATA_RETENTION_DAYS=365
-
-# Logging Configuration
-LOG_LEVEL=INFO
-LOG_FILE=water_monitor.log
-
-# Security (for production)
-SECRET_KEY=your-secret-key-here
-API_KEY=your-api-key-here
-
-# Monitoring
-ENABLE_METRICS=true
-ENABLE_HEALTH_CHECKS=true
-
-# Geographic Settings
-TIMEZONE=Asia/Bangkok
-DEFAULT_LATITUDE=18.7875
-DEFAULT_LONGITUDE=99.0045
-
-# External Services
-NOTIFICATION_EMAIL=
-SMTP_SERVER=
-SMTP_PORT=587
-SMTP_USERNAME=
-SMTP_PASSWORD=
-
-# Development Settings
-DEBUG=false
-DEVELOPMENT_MODE=false

.env.example

@@ -84,6 +84,19 @@ SMTP_PORT=587
 SMTP_USERNAME=
 SMTP_PASSWORD=
 
+# Public push notifications via self-hosted ntfy (https://ntfy.sh, single binary).
+# Leave NTFY_SERVER empty to disable. Topics published: <prefix>-<station>-warning,
+# <prefix>-<station>-danger, <prefix>-warning, <prefix>-danger, <prefix>-p1-outlook,
+# <prefix>-status. See docs/NOTIFICATIONS.md.
+NTFY_SERVER=
+# Where the monitor POSTs (defaults to NTFY_SERVER). Use the local ntfy
+# address (loopback or Tailscale IP) so publishing does not depend on
+# DNS / the reverse proxy being up.
+NTFY_PUBLISH_URL=
+NTFY_TOPIC_PREFIX=ping
+NTFY_TOKEN=
+PUBLIC_URL=https://water.buildfor.life/
+
 # Matrix Alerting Configuration
 MATRIX_HOMESERVER=https://matrix.org
 MATRIX_ACCESS_TOKEN=

.env.postgres

@@ -1,2 +0,0 @@
-DB_TYPE=postgresql
-POSTGRES_CONNECTION_STRING=postgresql://postgres:password@localhost:5432/water_monitoring

.gitea/workflows/ci.yml

@@ -1,342 +1,75 @@
-name: CI/CD Pipeline - Northern Thailand Ping River Monitor
+name: CI
+
+# What this checks, on every push and PR to master:
+#   1. formatting contract (black + isort, config in pyproject.toml)
+#   2. flake8 hard-error gate (syntax, undefined names)
+#   3. the pytest suite (synthetic data, no DB/network; ~1 min)
+# Docker build / staging / production / perf jobs from the original template
+# were removed: there is no registry, no staging host, and production is a
+# systemd unit deployed by `git pull` on the server (docs/FLOOD_FORECASTING.md
+# section 6, scripts/install.sh). Re-add a job when the thing it deploys exists.
 
 on:
   push:
-    branches: [ master, develop ]
+    branches: [master, develop]
   pull_request:
-    branches: [ master ]
+    branches: [master]
   schedule:
-    # Run tests daily at 2 AM UTC
-    - cron: '0 2 * * *'
+    # daily, catches dependency drift / upstream API changes in the tests
+    - cron: "0 2 * * *"
+  workflow_dispatch:
 
 env:
-  PYTHON_VERSION: '3.11'
-  REGISTRY: git.b4l.co.th
-  IMAGE_NAME: b4l/northern-thailand-ping-river-monitor
-  # GitHub token for better rate limits and authentication
-  GH_TOKEN: ${{ secrets.GH_TOKEN }}
+  PYTHON_VERSION: "3.11" # pandas 2.0.3 ships no 3.12 wheels; psycopg2-binary 2.9.9 breaks on 3.13
 
 jobs:
-  # Test job
+  lint:
+    name: Format & lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: pip
+          cache-dependency-path: requirements-dev.txt
+
+      - name: Install tools
+        run: |
+          python -m pip install --upgrade pip --root-user-action=ignore
+          pip install --root-user-action=ignore black==26.5.1 isort==5.12.0 flake8==6.1.0
+
+      - name: black
+        run: black --check --diff src/ *.py
+
+      - name: isort
+        run: isort --check-only --diff src/ *.py
+
+      - name: flake8 (errors only)
+        run: flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
+
   test:
-    name: Test Suite
+    name: Test suite
     runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version: ['3.11']  # pandas 2.0.3 ships no 3.12 wheels; widen after upgrading pandas
-    
     steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-        
-    - name: Cache pip dependencies
-      uses: actions/cache@v3
-      with:
-        path: ~/.cache/pip
-        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
-        restore-keys: |
-          ${{ runner.os }}-pip-
-          
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip --root-user-action=ignore
-        pip install --root-user-action=ignore -r requirements.txt
-        pip install --root-user-action=ignore -r requirements-dev.txt
-        
-    - name: Lint with flake8
-      run: |
-        flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
-        flake8 src/ --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
-        
-    - name: Type check with mypy (advisory)
-      run: |
-        # 86 pre-existing errors; blocking typing gate deferred until the debt is paid down
-        mypy src/ --ignore-missing-imports || true
-        
-    - name: Format check with black
-      run: |
-        black --check src/ *.py
-        
-    - name: Import sort check
-      run: |
-        isort --check-only src/ *.py
-        
-    - name: Run integration tests
-      run: |
-        python tests/test_integration.py
-        
-    - name: Run station management tests
-      run: |
-        python tests/test_station_management.py
-        
-    - name: Test application startup
-      run: |
-        timeout 10s python run.py --test || true
-        
-    - name: Security scan with bandit
-      run: |
-        bandit -r src/ -f json -o bandit-report.json || true
-        
-    - name: Upload test artifacts
-      uses: actions/upload-artifact@v3
-      if: always()
-      with:
-        name: test-results-${{ matrix.python-version }}
-        path: |
-          bandit-report.json
-          *.log
+      - uses: actions/checkout@v4
 
-  # Code quality job
-  code-quality:
-    name: Code Quality
-    runs-on: ubuntu-latest
-    
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Set up Python
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ env.PYTHON_VERSION }}
-        
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip --root-user-action=ignore
-        pip install --root-user-action=ignore -r requirements-dev.txt
-        
-    - name: Run safety check
-      run: |
-        safety check -r requirements.txt --json --output safety-report.json || true
-        
-    - name: Run bandit security scan
-      run: |
-        bandit -r src/ -f json -o bandit-report.json || true
-        
-    - name: Upload security reports
-      uses: actions/upload-artifact@v3
-      with:
-        name: security-reports
-        path: |
-          safety-report.json
-          bandit-report.json
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: pip
+          cache-dependency-path: |
+            requirements.txt
+            requirements-dev.txt
 
-  # Build Docker image
-  build:
-    name: Build Docker Image
-    runs-on: ubuntu-latest
-    needs: test
-    
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@v3
-      
-    - name: Log in to Container Registry
-      uses: docker/login-action@v3
-      with:
-        registry: ${{ env.REGISTRY }}
-        username: ${{ github.actor }}
-        password: ${{ secrets.GITEA_TOKEN }}
-        
-    - name: Extract metadata
-      id: meta
-      uses: docker/metadata-action@v5
-      with:
-        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
-        tags: |
-          type=ref,event=branch
-          type=ref,event=pr
-          type=sha,prefix={{branch}}-
-          type=raw,value=latest,enable={{is_default_branch}}
-          
-    - name: Build and push Docker image
-      uses: docker/build-push-action@v5
-      with:
-        context: .
-        platforms: linux/amd64,linux/arm64
-        push: true
-        tags: ${{ steps.meta.outputs.tags }}
-        labels: ${{ steps.meta.outputs.labels }}
-        cache-from: type=gha
-        cache-to: type=gha,mode=max
-      env:
-        GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
-        
-    - name: Test Docker image
-      run: |
-        docker run --rm ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} python run.py --test
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip --root-user-action=ignore
+          pip install --root-user-action=ignore -r requirements.txt
+          pip install --root-user-action=ignore pytest==9.1.1 pytest-asyncio==0.21.1
 
-  # Integration test with services
-  integration-test:
-    name: Integration Test with Services
-    runs-on: ubuntu-latest
-    needs: build
-    
-    services:
-      victoriametrics:
-        image: victoriametrics/victoria-metrics:latest
-        ports:
-          - 8428:8428
-        options: >-
-          --health-cmd "wget --quiet --tries=1 --spider http://localhost:8428/health"
-          --health-interval 30s
-          --health-timeout 10s
-          --health-retries 3
-          
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Wait for VictoriaMetrics
-      run: |
-        timeout 60s bash -c 'until curl -f http://localhost:8428/health; do sleep 2; done'
-        
-    - name: Set up Python
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ env.PYTHON_VERSION }}
-        
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip --root-user-action=ignore
-        pip install --root-user-action=ignore -r requirements.txt
-        
-    - name: Test with VictoriaMetrics
-      env:
-        DB_TYPE: victoriametrics
-        VM_HOST: localhost
-        VM_PORT: 8428
-      run: |
-        python run.py --test
-        
-    - name: Start API server
-      env:
-        DB_TYPE: victoriametrics
-        VM_HOST: localhost
-        VM_PORT: 8428
-      run: |
-        python run.py --web-api &
-        sleep 10
-        
-    - name: Test API endpoints
-      run: |
-        curl -f http://localhost:8000/health
-        curl -f http://localhost:8000/stations
-        curl -f http://localhost:8000/metrics
-
-  # Deploy to staging (only on develop branch)
-  deploy-staging:
-    name: Deploy to Staging
-    runs-on: ubuntu-latest
-    needs: [test, build, integration-test]
-    if: github.ref == 'refs/heads/develop'
-    environment:
-      name: staging
-      url: https://staging.ping-river-monitor.b4l.co.th
-      
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Deploy to staging
-      run: |
-        echo "Deploying to staging environment..."
-        # Add your staging deployment commands here
-        # Example: kubectl, docker-compose, or webhook call
-        
-    - name: Health check staging
-      run: |
-        sleep 30
-        curl -f https://staging.ping-river-monitor.b4l.co.th/health
-
-  # Deploy to production (only on main branch, manual approval)
-  deploy-production:
-    name: Deploy to Production
-    runs-on: ubuntu-latest
-    needs: [test, build, integration-test]
-    if: github.ref == 'refs/heads/master'
-    environment:
-      name: production
-      url: https://ping-river-monitor.b4l.co.th
-      
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Deploy to production
-      run: |
-        echo "Deploying to production environment..."
-        # Add your production deployment commands here
-        
-    - name: Health check production
-      run: |
-        sleep 30
-        curl -f https://ping-river-monitor.b4l.co.th/health
-        
-    - name: Notify deployment
-      run: |
-        echo "✅ Production deployment successful!"
-        echo "🌐 URL: https://ping-river-monitor.b4l.co.th"
-        echo "📊 Grafana: https://grafana.ping-river-monitor.b4l.co.th"
-
-  # Performance test (only on main branch)
-  performance-test:
-    name: Performance Test
-    runs-on: ubuntu-latest
-    needs: deploy-production
-    if: github.ref == 'refs/heads/master'
-    
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Install Apache Bench
-      run: |
-        sudo apt-get update
-        sudo apt-get install -y apache2-utils
-        
-    - name: Performance test API endpoints
-      run: |
-        # Test health endpoint
-        ab -n 100 -c 10 https://ping-river-monitor.b4l.co.th/health
-        
-        # Test stations endpoint
-        ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/stations
-        
-        # Test metrics endpoint
-        ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/metrics
-
-  # Cleanup old artifacts
-  cleanup:
-    name: Cleanup
-    runs-on: ubuntu-latest
-    if: always()
-    needs: [test, build, integration-test]
-    
-    steps:
-    - name: Clean up old Docker images
-      run: |
-        echo "Cleaning up old Docker images..."
-        # Add cleanup commands for old images/artifacts
+      - name: pytest
+        env:
+          DB_TYPE: sqlite
+        run: pytest -q -p no:cacheprovider

.gitea/workflows/docs.yml

@@ -1,368 +1,99 @@
-name: Documentation
+name: Docs
+
+# Checks that the documentation the project actually ships stays consistent:
+#   - every relative link / image path in docs/*.md and README.md resolves
+#     inside the repo (external URLs are NOT fetched: localhost examples,
+#     rate-limited hosts and the Tailscale-era links made that gate permanently
+#     red, and a 200 on a curl --head proves nothing about a doc anyway)
+#   - the FastAPI app imports and its OpenAPI schema is exportable (that is
+#     the reference at https://water.buildfor.life/docs)
+# The previous Sphinx/apidoc jobs produced artifacts nobody read and were
+# removed. Reference docs live in docs/*.md; the public overview is at
+# https://buildfor.life/docs/tooling/ping-river-monitor/.
 
 on:
   push:
-    branches: [ master, develop ]
+    branches: [master, develop]
     paths:
-      - 'docs/**'
-      - 'README.md'
-      - 'CONTRIBUTING.md'
-      - 'src/**/*.py'
+      - "docs/**"
+      - "README.md"
+      - "CONTRIBUTING.md"
+      - "src/web_api.py"
+      - "src/schemas.py"
+      - ".gitea/workflows/docs.yml"
   pull_request:
     paths:
-      - 'docs/**'
-      - 'README.md'
-      - 'CONTRIBUTING.md'
+      - "docs/**"
+      - "README.md"
+      - "CONTRIBUTING.md"
   workflow_dispatch:
 
 env:
-  PYTHON_VERSION: '3.11'
+  PYTHON_VERSION: "3.11"
 
 jobs:
-  # Validate documentation
-  validate-docs:
-    name: Validate Documentation
+  docs:
+    name: Validate documentation
     runs-on: ubuntu-latest
-    
     steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Set up Python
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ env.PYTHON_VERSION }}
-        
-    - name: Install documentation tools
-      run: |
-        python -m pip install --upgrade pip
-        pip install -r requirements.txt
-        pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
-        pip install markdown-link-check || true
-        
-    - name: Check markdown links
-      run: |
-        echo "🔗 Checking markdown links..."
-        find . -name "*.md" -not -path "./.git/*" -not -path "./node_modules/*" | while read file; do
-          echo "Checking $file"
-          # Basic link validation (you can enhance this)
-          grep -o 'http[s]*://[^)]*' "$file" | while read url; do
-            if curl -s --head "$url" | head -n 1 | grep -q "200 OK"; then
-              echo "✅ $url"
-            else
-              echo "❌ $url (in $file)"
-            fi
-          done
-        done
-        
-    - name: Validate README structure
-      run: |
-        echo "📋 Validating README structure..."
-        
-        required_sections=(
-          "# Northern Thailand Ping River Monitor"
-          "## Features"
-          "## Quick Start"
-          "## Installation"
-          "## Usage"
-          "## API Endpoints"
-          "## Docker"
-          "## Contributing"
-          "## License"
-        )
-        
-        for section in "${required_sections[@]}"; do
-          if grep -q "$section" README.md; then
-            echo "✅ Found: $section"
-          else
-            echo "❌ Missing: $section"
-          fi
-        done
-        
-    - name: Check documentation completeness
-      run: |
-        echo "📚 Checking documentation completeness..."
-        
-        # Check if all Python modules have docstrings
-        python -c "
-        import ast
-        import os
-        
-        def check_docstrings(filepath):
-            with open(filepath, 'r', encoding='utf-8') as f:
-                tree = ast.parse(f.read())
-            
-            missing_docstrings = []
-            
-            for node in ast.walk(tree):
-                if isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef)):
-                    if not ast.get_docstring(node):
-                        missing_docstrings.append(f'{node.name} in {filepath}')
-            
-            return missing_docstrings
-        
-        all_missing = []
-        for root, dirs, files in os.walk('src'):
-            for file in files:
-                if file.endswith('.py') and not file.startswith('__'):
-                    filepath = os.path.join(root, file)
-                    missing = check_docstrings(filepath)
-                    all_missing.extend(missing)
-        
-        if all_missing:
-            print('⚠️ Missing docstrings:')
-            for item in all_missing[:10]:  # Show first 10
-                print(f'  - {item}')
-            if len(all_missing) > 10:
-                print(f'  ... and {len(all_missing) - 10} more')
-        else:
-            print('✅ All functions and classes have docstrings')
-        "
+      - uses: actions/checkout@v4
 
-  # Generate API documentation
-  generate-api-docs:
-    name: Generate API Documentation
-    runs-on: ubuntu-latest
-    
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Set up Python
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ env.PYTHON_VERSION }}
-        
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install -r requirements.txt
-        
-    - name: Generate OpenAPI spec
-      run: |
-        echo "📝 Generating OpenAPI specification..."
-        python -c "
-        import json
-        import sys
-        sys.path.insert(0, 'src')
-        
-        try:
-            from web_api import app
-            openapi_spec = app.openapi()
-            
-            with open('openapi.json', 'w') as f:
-                json.dump(openapi_spec, f, indent=2)
-            
-            print('✅ OpenAPI spec generated: openapi.json')
-        except Exception as e:
-            print(f'❌ Failed to generate OpenAPI spec: {e}')
-        "
-        
-    - name: Generate API documentation
-      run: |
-        echo "📖 Generating API documentation..."
-        
-        # Create API documentation from OpenAPI spec
-        if [ -f openapi.json ]; then
-          cat > api-docs.md << 'EOF'
-        # API Documentation
-        
-        This document describes the REST API endpoints for the Northern Thailand Ping River Monitor.
-        
-        ## Base URL
-        
-        - Production: `https://ping-river-monitor.b4l.co.th`
-        - Staging: `https://staging.ping-river-monitor.b4l.co.th`
-        - Development: `http://localhost:8000`
-        
-        ## Authentication
-        
-        Currently, the API does not require authentication. This may change in future versions.
-        
-        ## Endpoints
-        
-        EOF
-          
-          # Extract endpoints from OpenAPI spec
-          python -c "
-        import json
-        
-        with open('openapi.json', 'r') as f:
-            spec = json.load(f)
-        
-        for path, methods in spec.get('paths', {}).items():
-            for method, details in methods.items():
-                print(f'### {method.upper()} {path}')
-                print()
-                print(details.get('summary', 'No description available'))
-                print()
-                if 'parameters' in details:
-                    print('**Parameters:**')
-                    for param in details['parameters']:
-                        print(f'- `{param[\"name\"]}` ({param.get(\"in\", \"query\")}): {param.get(\"description\", \"No description\")}')
-                    print()
-                print('---')
-                print()
-          " >> api-docs.md
-          
-          echo "✅ API documentation generated: api-docs.md"
-        fi
-        
-    - name: Upload documentation artifacts
-      uses: actions/upload-artifact@v3
-      with:
-        name: documentation-${{ github.run_number }}
-        path: |
-          openapi.json
-          api-docs.md
+      - name: Relative links and images resolve
+        run: |
+          python3 - <<'PY'
+          import re, sys, pathlib
+          root = pathlib.Path(".")
+          files = [root / "README.md", root / "CONTRIBUTING.md", *root.glob("docs/**/*.md")]
+          link = re.compile(r"!?\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
+          bad = []
+          for md in files:
+              if not md.exists():
+                  continue
+              for m in link.finditer(md.read_text(encoding="utf-8")):
+                  target = m.group(1)
+                  if target.startswith(("http://", "https://", "mailto:", "#")):
+                      continue
+                  path = target.split("#", 1)[0]
+                  if not path:
+                      continue
+                  resolved = (md.parent / path).resolve()
+                  if not resolved.exists():
+                      bad.append(f"{md}: {target}")
+          if bad:
+              print("Broken relative links:")
+              print("\n".join("  " + b for b in bad))
+              sys.exit(1)
+          print(f"checked {len(files)} files, all relative links resolve")
+          PY
 
-  # Build Sphinx documentation
-  build-sphinx-docs:
-    name: Build Sphinx Documentation
-    runs-on: ubuntu-latest
-    
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      with:
-        token: ${{ secrets.GITEA_TOKEN }}
-      
-    - name: Set up Python
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ env.PYTHON_VERSION }}
-        
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install -r requirements.txt
-        pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
-        
-    - name: Create Sphinx configuration
-      run: |
-        mkdir -p docs/sphinx
-        
-        cat > docs/sphinx/conf.py << 'EOF'
-        import os
-        import sys
-        sys.path.insert(0, os.path.abspath('../../src'))
-        
-        project = 'Northern Thailand Ping River Monitor'
-        copyright = '2025, Ping River Monitor Team'
-        author = 'Ping River Monitor Team'
-        version = '3.1.3'
-        release = '3.1.3'
-        
-        extensions = [
-            'sphinx.ext.autodoc',
-            'sphinx.ext.viewcode',
-            'sphinx.ext.napoleon',
-            'sphinx_autodoc_typehints',
-        ]
-        
-        templates_path = ['_templates']
-        exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
-        
-        html_theme = 'sphinx_rtd_theme'
-        html_static_path = ['_static']
-        
-        autodoc_default_options = {
-            'members': True,
-            'member-order': 'bysource',
-            'special-members': '__init__',
-            'undoc-members': True,
-            'exclude-members': '__weakref__'
-        }
-        EOF
-        
-        cat > docs/sphinx/index.rst << 'EOF'
-        Northern Thailand Ping River Monitor Documentation
-        ================================================
-        
-        .. toctree::
-           :maxdepth: 2
-           :caption: Contents:
-        
-           modules
-        
-        Indices and tables
-        ==================
-        
-        * :ref:`genindex`
-        * :ref:`modindex`
-        * :ref:`search`
-        EOF
-        
-    - name: Generate module documentation
-      run: |
-        cd docs/sphinx
-        sphinx-apidoc -o . ../../src
-        
-    - name: Build documentation
-      run: |
-        cd docs/sphinx
-        sphinx-build -b html . _build/html
-        
-    - name: Upload Sphinx documentation
-      uses: actions/upload-artifact@v3
-      with:
-        name: sphinx-docs-${{ github.run_number }}
-        path: docs/sphinx/_build/html/
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: pip
+          cache-dependency-path: requirements.txt
 
-  # Documentation summary
-  docs-summary:
-    name: Documentation Summary
-    runs-on: ubuntu-latest
-    needs: [validate-docs, generate-api-docs, build-sphinx-docs]
-    if: always()
-    
-    steps:
-    - name: Generate documentation summary
-      run: |
-        echo "# 📚 Documentation Build Summary" > docs-summary.md
-        echo "" >> docs-summary.md
-        echo "**Build Date:** $(date -u)" >> docs-summary.md
-        echo "**Repository:** ${{ github.repository }}" >> docs-summary.md
-        echo "**Commit:** ${{ github.sha }}" >> docs-summary.md
-        echo "" >> docs-summary.md
-        
-        echo "## 📊 Results" >> docs-summary.md
-        echo "" >> docs-summary.md
-        
-        if [ "${{ needs.validate-docs.result }}" = "success" ]; then
-          echo "- ✅ **Documentation Validation**: Passed" >> docs-summary.md
-        else
-          echo "- ❌ **Documentation Validation**: Failed" >> docs-summary.md
-        fi
-        
-        if [ "${{ needs.generate-api-docs.result }}" = "success" ]; then
-          echo "- ✅ **API Documentation**: Generated" >> docs-summary.md
-        else
-          echo "- ❌ **API Documentation**: Failed" >> docs-summary.md
-        fi
-        
-        if [ "${{ needs.build-sphinx-docs.result }}" = "success" ]; then
-          echo "- ✅ **Sphinx Documentation**: Built" >> docs-summary.md
-        else
-          echo "- ❌ **Sphinx Documentation**: Failed" >> docs-summary.md
-        fi
-        
-        echo "" >> docs-summary.md
-        echo "## 🔗 Available Documentation" >> docs-summary.md
-        echo "" >> docs-summary.md
-        echo "- [README.md](../README.md)" >> docs-summary.md
-        echo "- [API Documentation](../docs/)" >> docs-summary.md
-        echo "- [Contributing Guide](../CONTRIBUTING.md)" >> docs-summary.md
-        echo "- [Deployment Checklist](../DEPLOYMENT_CHECKLIST.md)" >> docs-summary.md
-        
-        cat docs-summary.md
-        
-    - name: Upload documentation summary
-      uses: actions/upload-artifact@v3
-      with:
-        name: docs-summary-${{ github.run_number }}
-        path: docs-summary.md
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip --root-user-action=ignore
+          pip install --root-user-action=ignore -r requirements.txt
+
+      - name: OpenAPI schema exports
+        env:
+          DB_TYPE: sqlite
+        run: |
+          python - <<'PY'
+          import json
+          from src.web_api import app
+          spec = app.openapi()
+          paths = sorted(spec["paths"])
+          required = {"/forecast", "/measurements/latest", "/measurements/history/{station_code}", "/stations", "/api/stats", "/health"}
+          missing = required - set(paths)
+          assert not missing, f"documented endpoints missing from the app: {missing}"
+          json.dump(spec, open("openapi.json", "w"), indent=1)
+          print(f"{len(paths)} paths; schema written to openapi.json")
+          PY
+
+      - uses: actions/upload-artifact@v3
+        with:
+          name: openapi-${{ github.run_number }}
+          path: openapi.json

.gitea/workflows/security.yml

@@ -1,293 +1,109 @@
-name: Security & Dependency Updates
+name: Security
+
+# Two gates that can actually fail, plus one report:
+#   - pip-audit against requirements.txt: any known vulnerability in a runtime
+#     dependency fails the job (dev-only tools are reported, not gated)
+#   - bandit on src/: HIGH severity findings fail; medium/low are listed.
+#     B104 (bind 0.0.0.0) is skipped: the service is meant to listen on all
+#     interfaces behind Cloudflare/Caddy.
+#   - pip-licenses report as an artifact (informational; the project is MIT
+#     and its runtime deps are MIT/BSD/Apache/PSF)
+# The old file ran safety/bandit/semgrep with `|| true` and could not go red.
 
 on:
   schedule:
-    # Run security scans daily at 3 AM UTC
-    - cron: "0 3 * * *"
+    - cron: "0 3 * * 1" # weekly, Monday 03:00 UTC
   workflow_dispatch:
   push:
     paths:
       - "requirements*.txt"
-      - "Dockerfile"
+      - "pyproject.toml"
+      - "uv.lock"
+      - "src/**/*.py"
       - ".gitea/workflows/security.yml"
+  pull_request:
+    paths:
+      - "requirements*.txt"
+      - "pyproject.toml"
+      - "src/**/*.py"
 
 env:
   PYTHON_VERSION: "3.11"
-  # GitHub token for better rate limits and authentication
-  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 
 jobs:
-  # Dependency vulnerability scan
-  dependency-scan:
-    name: Dependency Security Scan
+  dependencies:
+    name: Dependency vulnerabilities
     runs-on: ubuntu-latest
-
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
+      - uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 
-      - name: Install dependencies
+      - name: Install pip-audit
         run: |
           python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore safety bandit semgrep
+          pip install --root-user-action=ignore pip-audit
 
-      - name: Run Safety check
-        run: |
-          safety check -r requirements.txt --json --output safety-report.json || true
-          safety check -r requirements-dev.txt --json --output safety-dev-report.json || true
+      - name: Runtime dependencies (gate)
+        run: pip-audit -r requirements.txt --strict --desc on
 
-      - name: Run Bandit security scan
-        run: |
-          bandit -r src/ -f json -o bandit-report.json || true
+      - name: Dev dependencies (report only)
+        run: pip-audit -r requirements-dev.txt --desc on || echo "::warning::dev-only dependency advisories above"
 
-      - name: Run Semgrep security scan
-        run: |
-          semgrep --config=auto src/ --json --output=semgrep-report.json || true
-
-      - name: Upload security reports
-        uses: actions/upload-artifact@v3
-        with:
-          name: security-reports-${{ github.run_number }}
-          path: |
-            safety-report.json
-            safety-dev-report.json
-            bandit-report.json
-            semgrep-report.json
-
-      - name: Check for critical vulnerabilities
-        run: |
-          echo "Checking for critical vulnerabilities..."
-
-          # Check Safety results
-          if [ -f safety-report.json ]; then
-            critical_count=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null || echo "0")
-            if [ "$critical_count" -gt 0 ]; then
-              echo "Found $critical_count dependency vulnerabilities"
-              jq '.vulnerabilities[] | "- \(.package_name) \(.installed_version): \(.vulnerability_id)"' safety-report.json
-            else
-              echo "No dependency vulnerabilities found"
-            fi
-          fi
-
-          # Check Bandit results
-          if [ -f bandit-report.json ]; then
-            high_severity=$(jq '.results[] | select(.issue_severity == "HIGH") | length' bandit-report.json 2>/dev/null | wc -l)
-            if [ "$high_severity" -gt 0 ]; then
-              echo "Found $high_severity high-severity security issues"
-            else
-              echo "No high-severity security issues found"
-            fi
-          fi
-
-  # License compliance check
-  license-check:
-    name: License Compliance
+  code:
+    name: Static analysis
     runs-on: ubuntu-latest
-
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
+      - uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 
-      - name: Install pip-licenses
+      - name: Install bandit
         run: |
           python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-licenses
-          pip install --root-user-action=ignore -r requirements.txt
+          pip install --root-user-action=ignore bandit
 
-      - name: Check licenses
+      - name: bandit (HIGH fails; medium/low listed)
         run: |
-          echo "Checking dependency licenses..."
+          bandit -r src/ -q --skip B104 -ll -ii || true
+          bandit -r src/ -q --skip B104 --severity-level high --confidence-level medium
+
+  licenses:
+    name: License report
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: pip
+          cache-dependency-path: requirements.txt
+
+      # A fresh venv, not the runner's site-packages: the report must list the
+      # project's runtime deps, not whatever the runner image or a previous
+      # workflow happened to leave installed (semgrep once showed up here).
+      - name: Install into a clean venv
+        run: |
+          python -m venv .lic && . .lic/bin/activate
+          pip install --upgrade pip --root-user-action=ignore
+          pip install --root-user-action=ignore -r requirements.txt pip-licenses
+
+      - name: Report
+        run: |
+          . .lic/bin/activate
+          pip-licenses --format=markdown --with-urls --output-file=licenses.md
           pip-licenses --format=json --output-file=licenses.json
-          pip-licenses --format=markdown --output-file=licenses.md
+          echo "Copyleft licenses among runtime deps (informational; LGPL is fine to link from MIT):"
+          pip-licenses --format=plain --ignore-packages pip-licenses | grep -iE 'GPL|AGPL|LGPL' || echo "  none"
 
-          # Check for problematic licenses
-          problematic_licenses=("GPL" "AGPL" "LGPL")
-
-          for license in "${problematic_licenses[@]}"; do
-            if grep -i "$license" licenses.json; then
-              echo "Found potentially problematic license: $license"
-            fi
-          done
-
-          echo "License check completed"
-
-      - name: Upload license report
-        uses: actions/upload-artifact@v3
+      - uses: actions/upload-artifact@v3
         with:
-          name: license-report-${{ github.run_number }}
+          name: licenses-${{ github.run_number }}
           path: |
-            licenses.json
             licenses.md
-
-  # Dependency update check
-  dependency-update:
-    name: Check for Dependency Updates
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
-
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-
-      - name: Install pip-check-updates equivalent
-        run: |
-          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-review
-
-      - name: Check for outdated packages
-        run: |
-          echo "Checking for outdated packages..."
-          pip install --root-user-action=ignore -r requirements.txt
-          pip list --outdated --format=json > outdated-packages.json || true
-
-          if [ -s outdated-packages.json ]; then
-            echo "Outdated packages found:"
-            cat outdated-packages.json | jq -r '.[] | "- \(.name): \(.version) -> \(.latest_version)"'
-          else
-            echo "All packages are up to date"
-          fi
-
-      - name: Upload dependency reports
-        uses: actions/upload-artifact@v3
-        with:
-          name: dependency-reports-${{ github.run_number }}
-          path: |
-            outdated-packages.json
-
-  # Code quality metrics
-  code-quality:
-    name: Code Quality Metrics
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          token: ${{ secrets.GITEA_TOKEN }}
-
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-
-      - name: Install quality tools
-        run: |
-          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore radon xenon vulture
-          pip install --root-user-action=ignore -r requirements.txt
-
-      - name: Calculate code complexity
-        run: |
-          echo "Calculating code complexity..."
-          radon cc src/ --json > complexity-report.json
-          radon mi src/ --json > maintainability-report.json
-
-          echo "Complexity Summary:"
-          radon cc src/ --average
-
-          echo "Maintainability Summary:"
-          radon mi src/
-
-      - name: Find dead code
-        run: |
-          echo "Checking for dead code..."
-          vulture src/ --json > dead-code-report.json || true
-
-      - name: Check for code smells
-        run: |
-          echo "Checking for code smells..."
-          xenon --max-absolute B --max-modules A --max-average A src/ || true
-
-      - name: Upload quality reports
-        uses: actions/upload-artifact@v3
-        with:
-          name: code-quality-reports-${{ github.run_number }}
-          path: |
-            complexity-report.json
-            maintainability-report.json
-            dead-code-report.json
-
-  # Security summary
-  security-summary:
-    name: Security Summary
-    runs-on: ubuntu-latest
-    needs: [dependency-scan, license-check, code-quality]
-    if: always()
-
-    steps:
-      - name: Download all artifacts
-        uses: actions/download-artifact@v3
-
-      - name: Generate security summary
-        run: |
-          echo "# Security Scan Summary" > security-summary.md
-          echo "" >> security-summary.md
-          echo "**Scan Date:** $(date -u)" >> security-summary.md
-          echo "**Repository:** ${{ github.repository }}" >> security-summary.md
-          echo "**Commit:** ${{ github.sha }}" >> security-summary.md
-          echo "" >> security-summary.md
-
-          echo "## Results" >> security-summary.md
-          echo "" >> security-summary.md
-
-          # Dependency scan results
-          if [ -f security-reports-*/safety-report.json ]; then
-            vuln_count=$(jq '.vulnerabilities | length' security-reports-*/safety-report.json 2>/dev/null || echo "0")
-            if [ "$vuln_count" -eq 0 ]; then
-              echo "- Dependency Scan: No vulnerabilities found" >> security-summary.md
-            else
-              echo "- Dependency Scan: $vuln_count vulnerabilities found" >> security-summary.md
-            fi
-          else
-            echo "- Dependency Scan: Results not available" >> security-summary.md
-          fi
-
-          # Docker scan results (removed Trivy)
-          echo "- Docker Scan: Skipped (Trivy removed)" >> security-summary.md
-
-          # License check results
-          if [ -f license-report-*/licenses.json ]; then
-            echo "- License Check: Completed" >> security-summary.md
-          else
-            echo "- License Check: Results not available" >> security-summary.md
-          fi
-
-          # Code quality results
-          if [ -f code-quality-reports-*/complexity-report.json ]; then
-            echo "- Code Quality: Analyzed" >> security-summary.md
-          else
-            echo "- Code Quality: Results not available" >> security-summary.md
-          fi
-
-          echo "" >> security-summary.md
-          echo "## Detailed Reports" >> security-summary.md
-          echo "" >> security-summary.md
-          echo "Detailed reports are available in the workflow artifacts." >> security-summary.md
-
-          cat security-summary.md
-
-      - name: Upload security summary
-        uses: actions/upload-artifact@v3
-        with:
-          name: security-summary-${{ github.run_number }}
-          path: security-summary.md
+            licenses.json

.gitignore

@@ -148,6 +148,28 @@ grafana_data/
 models/*.joblib
 models/cache/
 models/metrics.json
+# scripts/retrain.sh working dirs (staging + one rollback generation)
+models/.staging/
+models/.previous/
 
 # Playwright MCP browser artifacts (screenshots/snapshots from agent sessions)
 .playwright-mcp/
+
+# Agent tooling state (whole dirs; the entries above only covered subpaths)
+.claude/
+.claude-flow/
+.swarm/
+# local MCP server wiring, not project config
+.mcp.json
+# CLAUDE.md is intentionally NOT ignored — track it if you want the agent
+# conventions shared with collaborators; it is untracked today.
+
+# Model evaluation output (regenerate with scripts/evaluate_variants.py)
+models/eval_*.json
+
+# Editor/merge leftovers and stray shell-redirect artifacts. The repo root once
+# collected 56 zero-byte files named after fragments of shell commands.
+*.orig
+*.rej
+*.bak
+*~

.gitlab-ci.yml

@@ -1,129 +0,0 @@
-# GitLab CI/CD Pipeline for Northern Thailand Ping River Monitor
-
-stages:
-  - test
-  - build
-  - deploy
-
-variables:
-  PYTHON_VERSION: "3.11"
-  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
-
-cache:
-  paths:
-    - .cache/pip
-    - venv/
-
-# Test stage
-test:
-  stage: test
-  image: python:${PYTHON_VERSION}-slim
-  before_script:
-    - apt-get update && apt-get install -y build-essential
-    - python -m venv venv
-    - source venv/bin/activate
-    - pip install --upgrade pip
-    - pip install -r requirements-dev.txt
-  script:
-    - python test_integration.py
-    - python test_station_management.py
-    - flake8 src/ --max-line-length=100
-    - mypy src/
-  coverage: '/TOTAL.*\s+(\d+%)$/'
-  artifacts:
-    reports:
-      coverage_report:
-        coverage_format: cobertura
-        path: coverage.xml
-    paths:
-      - htmlcov/
-    expire_in: 1 week
-
-# Code quality
-code_quality:
-  stage: test
-  image: python:${PYTHON_VERSION}-slim
-  before_script:
-    - python -m venv venv
-    - source venv/bin/activate
-    - pip install black isort flake8 mypy
-  script:
-    - black --check src/ *.py
-    - isort --check-only src/ *.py
-    - flake8 src/ --max-line-length=100
-    - mypy src/
-  allow_failure: true
-
-# Security scan
-security_scan:
-  stage: test
-  image: python:${PYTHON_VERSION}-slim
-  before_script:
-    - pip install safety bandit
-  script:
-    - safety check -r requirements.txt
-    - bandit -r src/
-  allow_failure: true
-
-# Build Docker image
-build:
-  stage: build
-  image: docker:latest
-  services:
-    - docker:dind
-  before_script:
-    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
-  script:
-    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
-    - docker build -t $CI_REGISTRY_IMAGE:latest .
-    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-    - docker push $CI_REGISTRY_IMAGE:latest
-  only:
-    - main
-    - develop
-
-# Deploy to staging
-deploy_staging:
-  stage: deploy
-  image: alpine:latest
-  before_script:
-    - apk add --no-cache curl
-  script:
-    - echo "Deploying to staging environment"
-    - curl -X POST "$STAGING_WEBHOOK_URL" -H "Content-Type: application/json" -d '{"image":"'$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA'"}'
-  environment:
-    name: staging
-    url: https://staging.ping-river-monitor.example.com
-  only:
-    - develop
-
-# Deploy to production
-deploy_production:
-  stage: deploy
-  image: alpine:latest
-  before_script:
-    - apk add --no-cache curl
-  script:
-    - echo "Deploying to production environment"
-    - curl -X POST "$PRODUCTION_WEBHOOK_URL" -H "Content-Type: application/json" -d '{"image":"'$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA'"}'
-  environment:
-    name: production
-    url: https://ping-river-monitor.example.com
-  when: manual
-  only:
-    - main
-
-# Health check after deployment
-health_check:
-  stage: deploy
-  image: alpine:latest
-  before_script:
-    - apk add --no-cache curl jq
-  script:
-    - sleep 30  # Wait for deployment
-    - curl -f $HEALTH_CHECK_URL/health
-    - curl -s $HEALTH_CHECK_URL/metrics | jq .
-  dependencies:
-    - deploy_production
-  only:
-    - main

.pre-commit-config.yaml

@@ -19,22 +19,20 @@ repos:
 
   # Python code formatting with Black
   - repo: https://github.com/psf/black
-    rev: 23.11.0
+    rev: 26.5.1
     hooks:
       - id: black
         language_version: python3
-        args: ['--line-length=120']
 
   # Import sorting with isort
   - repo: https://github.com/pycqa/isort
     rev: 5.12.0
     hooks:
       - id: isort
-        args: ['--profile', 'black', '--line-length', '120']
 
   # Linting with flake8
   - repo: https://github.com/pycqa/flake8
     rev: 6.1.0
     hooks:
       - id: flake8
-        args: ['--max-line-length=120', '--extend-ignore=E203,W503']
+        args: ['--max-line-length=100', '--extend-ignore=E203,W503']

CLAUDE.md

@@ -0,0 +1,40 @@
+# CLAUDE.md
+
+Guidance for AI coding agents working in this repository.
+
+## What this is
+
+Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
+API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
+Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
+`water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
+
+## Rules
+
+- Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
+- `make format` (black 88 / isort black profile, config in pyproject.toml) before
+  committing; CI fails on formatting. `make test` must stay green — tests are
+  synthetic-data only, never add one that needs the DB or network.
+- Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
+  parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
+- Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
+  and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive
+  or negative, in `docs/FLOOD_FORECASTING.md` section 5. Do not change what is deployed
+  (`rise_rain` / hgb-v3) without a harness result that beats it on lead.
+- `train_all()` must never silently produce a gauge-only (v2) model; the guard that
+  raises `RainUnavailableError` stays.
+- No `git add -A`: zero-byte shell-accident files (`#`, `$(wc`, ...) have been committed
+  before. Stage files by name.
+- Do not add Co-Authored-By trailers.
+- The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
+  table: every user-visible string needs both languages.
+
+## Where things are
+
+- `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
+  `src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
+  `rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
+- `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
+  promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
+- `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
+  the source catalog.

DEPLOYMENT_CHECKLIST.md

@@ -1,268 +0,0 @@
-# 🚀 Deployment Checklist - Northern Thailand Ping River Monitor
-
-## ✅ Pre-Deployment Checklist
-
-### **Code Quality**
-- [ ] All tests pass (`make test`)
-- [ ] Code formatting applied (`make format`)
-- [ ] Linting checks pass (`make lint`)
-- [ ] No security vulnerabilities (`safety check`)
-- [ ] Documentation updated
-- [ ] Version number updated in `setup.py` and `src/__init__.py`
-
-### **Configuration**
-- [ ] Environment variables configured (`.env` file)
-- [ ] Database connection tested
-- [ ] API endpoints tested
-- [ ] Log levels appropriate for environment
-- [ ] Security settings configured (API keys, secrets)
-- [ ] Resource limits set (memory, CPU)
-
-### **Dependencies**
-- [ ] All required packages in `requirements.txt`
-- [ ] No unused dependencies
-- [ ] Security updates applied
-- [ ] Compatible Python version (3.9+)
-
-## 🐳 Docker Deployment
-
-### **Pre-Docker Checklist**
-- [ ] Dockerfile tested locally
-- [ ] Docker Compose configuration verified
-- [ ] Volume mounts configured correctly
-- [ ] Network settings configured
-- [ ] Health checks working
-- [ ] Resource limits set
-
-### **Docker Commands**
-```bash
-# Build and test locally
-make docker-build
-docker run --rm ping-river-monitor python run.py --test
-
-# Deploy with Docker Compose
-make docker-run
-
-# Verify deployment
-make health-check
-```
-
-### **Post-Docker Checklist**
-- [ ] All services running (`docker-compose ps`)
-- [ ] Health checks passing
-- [ ] Logs showing normal operation
-- [ ] API accessible (`curl http://localhost:8000/health`)
-- [ ] Database connectivity verified
-- [ ] Grafana dashboards loading
-
-## 🌐 Production Deployment
-
-### **Infrastructure Requirements**
-- [ ] Server specifications adequate (CPU, RAM, Storage)
-- [ ] Network connectivity to external APIs
-- [ ] SSL certificates configured (if HTTPS)
-- [ ] Firewall rules configured
-- [ ] Backup strategy implemented
-- [ ] Monitoring alerts configured
-
-### **Security Checklist**
-- [ ] API keys secured (environment variables)
-- [ ] Database credentials secured
-- [ ] HTTPS enabled for web interface
-- [ ] Input validation enabled
-- [ ] Rate limiting configured
-- [ ] Log sanitization enabled
-
-### **Performance Checklist**
-- [ ] Database indexes created
-- [ ] Connection pooling configured
-- [ ] Caching enabled where appropriate
-- [ ] Resource monitoring enabled
-- [ ] Performance baselines established
-
-## 📊 Monitoring Setup
-
-### **Health Monitoring**
-- [ ] Health check endpoints responding
-- [ ] Database health monitoring
-- [ ] API response time monitoring
-- [ ] Memory usage monitoring
-- [ ] Disk space monitoring
-
-### **Alerting**
-- [ ] Critical error alerts configured
-- [ ] Performance degradation alerts
-- [ ] Database connectivity alerts
-- [ ] Disk space alerts
-- [ ] API availability alerts
-
-### **Logging**
-- [ ] Log rotation configured
-- [ ] Log levels appropriate
-- [ ] Structured logging enabled
-- [ ] Log aggregation configured (if applicable)
-- [ ] Log retention policy set
-
-## 🔄 CI/CD Pipeline
-
-### **GitLab CI/CD**
-- [ ] `.gitlab-ci.yml` configured
-- [ ] Pipeline variables set
-- [ ] Test stage passing
-- [ ] Build stage creating artifacts
-- [ ] Deploy stage configured
-- [ ] Rollback procedure documented
-
-### **Pipeline Stages**
-- [ ] **Test**: Unit tests, integration tests, linting
-- [ ] **Build**: Docker image creation, artifact generation
-- [ ] **Deploy**: Staging deployment, production deployment
-- [ ] **Verify**: Health checks, smoke tests
-
-## 🗄️ Database Setup
-
-### **Database Configuration**
-- [ ] Database server running and accessible
-- [ ] Database created with correct permissions
-- [ ] Connection string configured
-- [ ] Migration scripts run (if applicable)
-- [ ] Backup strategy implemented
-- [ ] Performance tuning applied
-
-### **Database-Specific Checklist**
-
-#### **SQLite**
-- [ ] Database file permissions set correctly
-- [ ] WAL mode enabled for better concurrency
-- [ ] Regular backup scheduled
-
-#### **MySQL/PostgreSQL**
-- [ ] User accounts created with minimal privileges
-- [ ] Connection pooling configured
-- [ ] Query performance optimized
-- [ ] Replication configured (if applicable)
-
-#### **InfluxDB**
-- [ ] Retention policies configured
-- [ ] Continuous queries set up (if needed)
-- [ ] Backup strategy implemented
-
-#### **VictoriaMetrics**
-- [ ] Storage configuration optimized
-- [ ] Retention period set
-- [ ] Resource limits configured
-
-## 🌐 Web Interface
-
-### **API Deployment**
-- [ ] FastAPI server running
-- [ ] All endpoints responding correctly
-- [ ] API documentation accessible (`/docs`)
-- [ ] CORS configured correctly
-- [ ] Rate limiting working
-- [ ] Authentication configured (if applicable)
-
-### **Frontend Integration**
-- [ ] Grafana dashboards configured
-- [ ] Data sources connected
-- [ ] Visualizations working
-- [ ] Alerts configured
-- [ ] User access configured
-
-## 📈 Performance Verification
-
-### **Load Testing**
-- [ ] API endpoints tested under load
-- [ ] Database performance under load
-- [ ] Memory usage under load
-- [ ] Response times acceptable
-- [ ] Error rates acceptable
-
-### **Capacity Planning**
-- [ ] Expected data volume calculated
-- [ ] Storage growth projected
-- [ ] Scaling strategy documented
-- [ ] Resource monitoring thresholds set
-
-## 🔧 Operational Procedures
-
-### **Maintenance**
-- [ ] Update procedure documented
-- [ ] Backup and restore procedures tested
-- [ ] Rollback procedure documented
-- [ ] Monitoring runbooks created
-- [ ] Incident response procedures documented
-
-### **Documentation**
-- [ ] Deployment guide updated
-- [ ] API documentation current
-- [ ] Configuration documentation complete
-- [ ] Troubleshooting guide available
-- [ ] Contact information updated
-
-## ✅ Post-Deployment Verification
-
-### **Functional Testing**
-- [ ] Data collection working
-- [ ] API endpoints responding
-- [ ] Database writes successful
-- [ ] Web interface accessible
-- [ ] Station management working
-
-### **Integration Testing**
-- [ ] External API connectivity
-- [ ] Database integration
-- [ ] Monitoring integration
-- [ ] Alert system working
-- [ ] Backup system working
-
-### **Performance Testing**
-- [ ] Response times acceptable
-- [ ] Memory usage normal
-- [ ] CPU usage normal
-- [ ] Disk I/O normal
-- [ ] Network usage normal
-
-## 🚨 Rollback Plan
-
-### **Rollback Triggers**
-- [ ] Critical errors in production
-- [ ] Performance degradation
-- [ ] Data corruption
-- [ ] Security vulnerabilities
-- [ ] Service unavailability
-
-### **Rollback Procedure**
-1. [ ] Stop current deployment
-2. [ ] Restore previous Docker images
-3. [ ] Restore database backup (if needed)
-4. [ ] Verify system functionality
-5. [ ] Update monitoring and alerts
-6. [ ] Document incident and lessons learned
-
-## 📞 Support Information
-
-### **Emergency Contacts**
-- [ ] System administrator contact
-- [ ] Database administrator contact
-- [ ] Network administrator contact
-- [ ] Application developer contact
-
-### **Documentation Links**
-- [ ] Deployment guide
-- [ ] API documentation
-- [ ] Troubleshooting guide
-- [ ] Configuration reference
-- [ ] Monitoring dashboards
-
----
-
-**Deployment Date**: ___________  
-**Deployed By**: ___________  
-**Version**: v3.1.3  
-**Environment**: ___________  
-
-**Sign-off**:
-- [ ] Technical Lead: ___________
-- [ ] Operations Team: ___________
-- [ ] Security Team: ___________

FINAL_CHECKLIST.md

@@ -1,193 +0,0 @@
-# Final GitHub Publication Checklist ✅
-
-This checklist ensures the Thailand Water Level Monitor project is ready for GitHub publication.
-
-## 🎯 **Project Preparation Complete**
-
-### ✅ **Core Repository Files**
-- [x] **README.md** - Comprehensive project documentation with badges and quick start
-- [x] **LICENSE** - MIT License for open source distribution
-- [x] **CONTRIBUTING.md** - Detailed contributor guidelines
-- [x] **.gitignore** - Comprehensive ignore rules for all file types
-- [x] **requirements.txt** - All Python dependencies listed and tested
-
-### ✅ **Source Code Organization**
-- [x] **src/** directory created with clean separation
-- [x] **scripts/** directory for utility scripts and system files
-- [x] **docs/** directory with comprehensive documentation
-- [x] **grafana/** directory with visualization configuration
-- [x] All temporary files removed (*.db, *.log, __pycache__)
-
-### ✅ **Documentation Quality**
-- [x] **Installation guides** for all platforms and databases
-- [x] **Configuration examples** for 5 different database types
-- [x] **Troubleshooting guides** for common deployment issues
-- [x] **Migration guides** for updating existing systems
-- [x] **API references** documenting Thai government data sources
-- [x] **Notable documents** section with official resources
-
-### ✅ **Production Readiness**
-- [x] **Docker support** with Dockerfile and docker-compose
-- [x] **Systemd service** configuration for Linux deployment
-- [x] **Multi-database support** (SQLite, PostgreSQL, MySQL, InfluxDB, VictoriaMetrics)
-- [x] **Geolocation support** for Grafana geomap visualization
-- [x] **Migration scripts** for safe database schema updates
-- [x] **HTTPS configuration** guide for secure deployment
-
-### ✅ **Code Quality**
-- [x] **Modular architecture** with clean separation of concerns
-- [x] **Error handling** and comprehensive logging
-- [x] **Configuration management** via environment variables
-- [x] **Database abstraction** layer for multiple backends
-- [x] **Testing utilities** (demo_databases.py)
-
-### ✅ **Features Verified**
-- [x] **Real-time data collection** from 16 Thai water stations
-- [x] **15-minute scheduling** with intelligent retry logic
-- [x] **Gap filling** for missing historical data
-- [x] **Data validation** and error recovery
-- [x] **Geolocation integration** with sample coordinates
-- [x] **Grafana dashboards** with pre-built visualizations
-
-## 🚀 **Ready for GitHub Publication**
-
-### **Repository Information**
-- **Name**: `thailand-water-monitor`
-- **Description**: "Real-time water level monitoring system for Thailand's Royal Irrigation Department stations with Grafana visualization"
-- **Topics**: `water-monitoring`, `thailand`, `grafana`, `timeseries`, `python`, `iot`, `environmental-monitoring`
-- **License**: MIT
-- **Language**: Python
-
-### **Repository Settings**
-- [x] Enable Issues for bug reports and feature requests
-- [x] Enable Discussions for community support
-- [x] Enable Wiki for extended documentation
-- [x] Set up GitHub Pages for documentation hosting
-- [x] Configure branch protection for main branch
-
-### **Initial Release (v1.0.0)**
-- **Release Title**: "Thailand Water Level Monitor v1.0.0 - Complete Monitoring Solution"
-- **Release Notes**: 
-  - Complete real-time monitoring system
-  - Multi-database backend support
-  - Grafana geomap integration
-  - Production-ready deployment
-  - Comprehensive documentation
-
-## 📊 **Project Statistics**
-
-### **Code Metrics**
-- **Total Files**: 25+ files
-- **Python Source Files**: 4 main modules
-- **Documentation Files**: 12 comprehensive guides
-- **Configuration Files**: 6 deployment configurations
-- **Lines of Code**: ~2,000+ lines of Python
-- **Documentation**: ~15,000+ words
-
-### **Feature Coverage**
-- **Database Backends**: 5 different types supported
-- **Monitoring Stations**: 16 across Thailand
-- **Data Collection**: Every 15 minutes
-- **Data Points**: ~300 measurements per collection cycle
-- **Geolocation**: GPS coordinates and geohash support
-- **Visualization**: Pre-built Grafana dashboards
-
-### **Documentation Coverage**
-- **Installation**: Complete setup for all platforms
-- **Configuration**: All database types documented
-- **Deployment**: Docker, systemd, manual options
-- **Troubleshooting**: Common issues and solutions
-- **Migration**: Safe upgrade procedures
-- **API**: External data source documentation
-
-## 🌟 **Key Selling Points**
-
-### **For Water Management Professionals**
-- Real-time monitoring of 16 stations across Thailand
-- Historical data analysis and trend visualization
-- Alert capabilities for critical water levels
-- Integration with official Thai government data sources
-
-### **For Developers**
-- Clean, modular Python codebase
-- Multiple database backend options
-- Docker containerization for easy deployment
-- Comprehensive API documentation
-
-### **For System Administrators**
-- Production-ready deployment configurations
-- Systemd service integration
-- HTTPS and security configuration
-- Monitoring and logging capabilities
-
-### **For Data Scientists**
-- Time-series data with geolocation
-- Grafana visualization and analysis tools
-- Historical data gap filling
-- Export capabilities for further analysis
-
-## 🎯 **Post-Publication Roadmap**
-
-### **Immediate (Week 1)**
-- [ ] Create GitHub repository and upload files
-- [ ] Set up initial release v1.0.0
-- [ ] Configure repository settings and templates
-- [ ] Create project documentation website
-
-### **Short-term (Month 1)**
-- [ ] Add GitHub Actions for CI/CD
-- [ ] Create issue and PR templates
-- [ ] Set up automated testing
-- [ ] Add code quality badges
-
-### **Medium-term (Quarter 1)**
-- [ ] Community feedback integration
-- [ ] Additional database backends
-- [ ] Mobile app development
-- [ ] Advanced alerting system
-
-### **Long-term (Year 1)**
-- [ ] Predictive analytics features
-- [ ] Machine learning integration
-- [ ] Multi-country expansion
-- [ ] Commercial support options
-
-## 🏆 **Success Metrics**
-
-### **Community Engagement**
-- GitHub stars and forks
-- Issue reports and feature requests
-- Community contributions
-- Documentation feedback
-
-### **Technical Adoption**
-- Download and deployment statistics
-- Database backend usage patterns
-- Performance benchmarks
-- User success stories
-
-### **Impact Measurement**
-- Water management improvements
-- Early warning system effectiveness
-- Data accessibility improvements
-- Research and academic usage
-
----
-
-## ✅ **FINAL VERIFICATION**
-
-**All checklist items completed successfully!**
-
-The Thailand Water Level Monitor project is now:
-- ✅ **Professionally organized** with clean structure
-- ✅ **Comprehensively documented** with guides for all use cases
-- ✅ **Production ready** with multiple deployment options
-- ✅ **Community friendly** with contribution guidelines
-- ✅ **Feature complete** with real-time monitoring capabilities
-
-**🚀 Ready for GitHub publication and community engagement!** 🌊
-
----
-
-*Last updated: July 30, 2025*
-*Project status: Ready for publication*

GITEA_SETUP_SUMMARY.md

@@ -1,233 +0,0 @@
-# 🎉 Gitea Actions Setup Complete!
-
-## 🚀 **What's Been Created**
-
-Your **Northern Thailand Ping River Monitor** now has a complete CI/CD pipeline with Gitea Actions! Here's what's been set up:
-
-### **🔄 Gitea Actions Workflows**
-
-```
-.gitea/workflows/
-├── ci.yml          # Main CI/CD pipeline
-├── release.yml     # Automated releases
-├── security.yml    # Security & dependency scanning
-└── docs.yml        # Documentation generation
-```
-
-### **📊 Workflow Features**
-
-#### **1. CI/CD Pipeline (`ci.yml`)**
-- ✅ **Multi-Python Testing** (3.9, 3.10, 3.11, 3.12)
-- ✅ **Code Quality Checks** (flake8, mypy, black, isort)
-- ✅ **Docker Multi-Arch Builds** (amd64, arm64)
-- ✅ **Integration Testing** with VictoriaMetrics
-- ✅ **Automated Staging Deployment** (develop branch)
-- ✅ **Manual Production Deployment** (main branch)
-- ✅ **Performance Testing** after deployment
-
-#### **2. Release Management (`release.yml`)**
-- 🏷️ **Tag-Based Releases** (`v*.*.*` pattern)
-- 📝 **Automatic Changelog Generation**
-- 🐳 **Multi-Architecture Docker Images**
-- 🔒 **Security Scanning** before release
-- ✅ **Comprehensive Validation** after deployment
-
-#### **3. Security Monitoring (`security.yml`)**
-- 🔒 **Daily Security Scans** (3 AM UTC)
-- 📦 **Dependency Vulnerability Detection**
-- 🐳 **Docker Image Security Scanning**
-- 📄 **License Compliance Checking**
-- 📊 **Code Quality Metrics**
-- 🔄 **Automated Update Notifications**
-
-#### **4. Documentation (`docs.yml`)**
-- 📚 **API Documentation Generation**
-- 🔗 **Link Validation**
-- 📖 **Sphinx Documentation Building**
-- ✅ **Documentation Completeness Checking**
-
-## 🔧 **Setup Instructions**
-
-### **1. Configure Repository Secrets**
-
-In your Gitea repository settings, add these secrets:
-
-```bash
-# Required
-GITEA_TOKEN          # For container registry access
-
-# Optional (for notifications)
-SLACK_WEBHOOK_URL    # Slack notifications
-STAGING_WEBHOOK_URL  # Staging deployment webhook
-PRODUCTION_WEBHOOK_URL # Production deployment webhook
-```
-
-### **2. Enable Actions**
-
-1. Go to your repository settings in Gitea
-2. Enable "Actions" if not already enabled
-3. Configure runners if using self-hosted runners
-
-### **3. Push to Repository**
-
-```bash
-# Initialize and push
-git init
-git remote add origin https://git.b4l.co.th/grabowski/Northern-Thailand-Ping-River-Monitor.git
-git add .
-git commit -m "Initial commit with Gitea Actions workflows"
-git push -u origin main
-```
-
-## 🎯 **Workflow Triggers**
-
-### **Automatic Triggers**
-- **Push to main/develop** → CI/CD Pipeline
-- **Pull Request to main** → Testing & Validation
-- **Daily at 2 AM UTC** → CI/CD Health Check
-- **Daily at 3 AM UTC** → Security Scanning
-- **Git Tag `v*.*.*`** → Release Pipeline
-- **Documentation Changes** → Documentation Build
-
-### **Manual Triggers**
-- **Manual Dispatch** → Any workflow can be triggered manually
-- **Release Creation** → Manual release with custom version
-
-## 📊 **Monitoring & Status**
-
-### **Status Badges**
-Your README now includes comprehensive status badges:
-- CI/CD Pipeline Status
-- Security Scan Status  
-- Documentation Build Status
-- Python Version Support
-- FastAPI Version
-- Docker Ready
-- License Information
-- Current Version
-
-### **Workflow Artifacts**
-Each workflow generates useful artifacts:
-- **Test Results** and coverage reports
-- **Security Scan Reports** (JSON format)
-- **Docker Images** (multi-architecture)
-- **Documentation** (HTML and PDF)
-- **Performance Reports**
-
-## 🚀 **Usage Examples**
-
-### **Development Workflow**
-```bash
-# Create feature branch
-git checkout -b feature/new-station-type
-# Make changes
-git add .
-git commit -m "Add support for new station type"
-git push origin feature/new-station-type
-# Create PR in Gitea → Triggers testing
-```
-
-### **Release Workflow**
-```bash
-# Create and push release tag
-git tag v3.1.1
-git push origin v3.1.1
-# → Triggers automated release pipeline
-```
-
-### **Security Monitoring**
-- **Daily scans** run automatically
-- **Security reports** available in Actions artifacts
-- **Notifications** sent for critical vulnerabilities
-
-## 🔍 **Validation Commands**
-
-Test your setup locally:
-
-```bash
-# Validate workflow syntax
-make validate-workflows
-
-# Test workflow components
-make workflow-test
-
-# Run full test suite
-make test
-
-# Build Docker image
-make docker-build
-```
-
-## 📈 **Performance & Optimization**
-
-### **Caching Strategy**
-- **Pip dependencies** cached across runs
-- **Docker layers** cached for faster builds
-- **Workflow artifacts** retained for analysis
-
-### **Parallel Execution**
-- **Matrix builds** for multiple Python versions
-- **Independent jobs** for security and testing
-- **Conditional execution** to skip unnecessary steps
-
-### **Resource Management**
-- **Appropriate timeouts** prevent hanging workflows
-- **Artifact cleanup** manages storage usage
-- **Efficient Docker builds** with multi-stage approach
-
-## 🔒 **Security Best Practices**
-
-### **Implemented Security**
-- ✅ **Secret management** via Gitea repository secrets
-- ✅ **Multi-stage Docker builds** for minimal attack surface
-- ✅ **Non-root containers** for better security
-- ✅ **Vulnerability scanning** before deployment
-- ✅ **Dependency monitoring** with automated alerts
-
-### **Security Scanning Coverage**
-- **Python dependencies** (Safety, Bandit)
-- **Docker images** (Trivy)
-- **Code quality** (Semgrep)
-- **License compliance** (pip-licenses)
-
-## 📚 **Documentation**
-
-### **Available Documentation**
-- [Gitea Workflows Guide](docs/GITEA_WORKFLOWS.md) - Detailed workflow documentation
-- [Contributing Guide](CONTRIBUTING.md) - How to contribute
-- [Deployment Checklist](DEPLOYMENT_CHECKLIST.md) - Production deployment
-- [Project Structure](docs/PROJECT_STRUCTURE.md) - Architecture overview
-
-### **Generated Documentation**
-- **API Documentation** - Auto-generated from OpenAPI spec
-- **Code Documentation** - Sphinx-generated from docstrings
-- **Security Reports** - Automated vulnerability reports
-
-## 🎉 **Ready for Production!**
-
-Your repository is now equipped with:
-
-- 🔄 **Enterprise-grade CI/CD pipeline**
-- 🔒 **Comprehensive security monitoring**
-- 📊 **Automated quality assurance**
-- 🚀 **Streamlined release management**
-- 📚 **Automated documentation**
-- 🐳 **Multi-architecture Docker support**
-- 📈 **Performance monitoring**
-- 🔍 **Comprehensive testing**
-
-## 🚀 **Next Steps**
-
-1. **Push to Gitea** and watch the workflows run
-2. **Configure deployment environments** (staging/production)
-3. **Set up monitoring dashboards** for workflow metrics
-4. **Configure notifications** for team collaboration
-5. **Create your first release** with `git tag v3.1.3`
-
-Your **Northern Thailand Ping River Monitor** is now ready for professional development and deployment! 🎊
-
----
-
-**Workflow Version**: v3.1.3  
-**Setup Date**: 2025-08-12  
-**Repository**: https://git.b4l.co.th/grabowski/Northern-Thailand-Ping-River-Monitor

GITHUB_PUBLICATION_SUMMARY.md

@@ -1,203 +0,0 @@
-# GitHub Publication Summary
-
-This document summarizes the Thailand Water Level Monitor project preparation for GitHub publication.
-
-## 📁 **Final Project Structure**
-
-```
-thailand-water-monitor/
-├── 📄 README.md                           # Main project documentation
-├── 📄 LICENSE                             # MIT License
-├── 📄 CONTRIBUTING.md                     # Contributor guidelines
-├── 📄 requirements.txt                    # Python dependencies
-├── 📄 .gitignore                          # Git ignore rules
-├── 📄 Dockerfile                          # Container definition
-├── 📄 docker-compose.victoriametrics.yml # Complete stack deployment
-│
-├── 📂 src/                                # Source Code
-│   ├── 🐍 water_scraper_v3.py           # Main application
-│   ├── 🐍 database_adapters.py          # Multi-database support
-│   ├── 🐍 config.py                     # Configuration management
-│   └── 🐍 demo_databases.py             # Database testing utility
-│
-├── 📂 scripts/                           # Utility Scripts
-│   ├── 🐍 migrate_geolocation.py        # Database migration script
-│   └── ⚙️ water-monitor.service          # Systemd service file
-│
-├── 📂 docs/                              # Documentation
-│   ├── 📖 DATABASE_DEPLOYMENT_GUIDE.md   # Complete setup guide
-│   ├── 📖 ENHANCED_SCHEDULER_GUIDE.md    # 15-minute scheduling
-│   ├── 📖 GEOLOCATION_GUIDE.md          # Grafana geomap integration
-│   ├── 📖 GAP_FILLING_GUIDE.md          # Data integrity management
-│   ├── 📖 MIGRATION_QUICKSTART.md       # Quick migration guide
-│   ├── 📖 VICTORIAMETRICS_SETUP.md      # High-performance deployment
-│   ├── 📖 HTTPS_CONFIGURATION.md        # Secure deployment
-│   ├── 📖 DEBIAN_TROUBLESHOOTING.md     # Linux deployment issues
-│   ├── 📖 PROJECT_STATUS.md             # Development status
-│   └── 📂 references/
-│       └── 📖 NOTABLE_DOCUMENTS.md       # Official Thai government resources
-│
-└── 📂 grafana/                          # Grafana Configuration
-    ├── 📂 dashboards/
-    │   └── 📊 water-monitoring-dashboard.json
-    └── 📂 provisioning/
-        ├── 📂 dashboards/
-        │   └── ⚙️ dashboard.yml
-        └── 📂 datasources/
-            └── ⚙️ victoriametrics.yml
-```
-
-## ✅ **GitHub Readiness Checklist**
-
-### **Core Files**
-- ✅ **README.md** - Comprehensive project documentation with badges, features, quick start
-- ✅ **LICENSE** - MIT License for open source distribution
-- ✅ **CONTRIBUTING.md** - Detailed contributor guidelines and development setup
-- ✅ **.gitignore** - Comprehensive ignore rules for Python, databases, logs, IDE files
-- ✅ **requirements.txt** - All Python dependencies listed
-
-### **Source Code Organization**
-- ✅ **src/** directory - Clean separation of source code
-- ✅ **scripts/** directory - Utility scripts and system files
-- ✅ **docs/** directory - Comprehensive documentation
-- ✅ **grafana/** directory - Visualization configuration
-
-### **Documentation Quality**
-- ✅ **Installation guides** - Multiple deployment options
-- ✅ **Configuration examples** - All database types covered
-- ✅ **Troubleshooting guides** - Common issues and solutions
-- ✅ **Migration guides** - Updating existing systems
-- ✅ **API references** - External data sources documented
-
-### **Production Readiness**
-- ✅ **Docker support** - Containerization ready
-- ✅ **Systemd service** - Linux service configuration
-- ✅ **Multi-database support** - 5 different database options
-- ✅ **Geolocation support** - Grafana geomap integration
-- ✅ **Migration scripts** - Safe database updates
-
-## 🌟 **Key Features for GitHub**
-
-### **Real-time Monitoring**
-- 16 water stations across Thailand
-- 15-minute data collection frequency
-- Automatic gap filling and data validation
-- Multi-database backend support
-
-### **Visualization Ready**
-- Pre-built Grafana dashboards
-- Geomap integration with coordinates
-- Real-time alerts and notifications
-- Historical trend analysis
-
-### **Production Deployment**
-- Docker containerization
-- VictoriaMetrics high-performance backend
-- HTTPS and security configuration
-- Comprehensive logging and monitoring
-
-### **Developer Friendly**
-- Clean, modular code structure
-- Comprehensive documentation
-- Multiple database adapters
-- Easy local development setup
-
-## 📊 **Project Statistics**
-
-### **Code Metrics**
-- **Python Files**: 4 main source files
-- **Documentation**: 10+ comprehensive guides
-- **Database Support**: 5 different backends
-- **Monitoring Stations**: 16 across Thailand
-- **Data Points**: ~300 every 15 minutes
-
-### **Documentation Coverage**
-- **Installation**: Complete setup guides for all platforms
-- **Configuration**: All database types documented
-- **Deployment**: Docker, systemd, and manual options
-- **Troubleshooting**: Common issues and solutions
-- **Migration**: Safe upgrade procedures
-
-### **Features Implemented**
-- ✅ Real-time data collection
-- ✅ Multi-database support
-- ✅ Geolocation integration
-- ✅ Gap filling and data validation
-- ✅ Grafana visualization
-- ✅ Docker deployment
-- ✅ Production monitoring
-- ✅ Migration tools
-
-## 🚀 **Ready for GitHub Publication**
-
-### **Repository Setup**
-1. **Create GitHub repository** - "thailand-water-monitor"
-2. **Upload all files** - Complete project structure
-3. **Configure repository settings**:
-   - Add description: "Real-time water level monitoring for Thailand's RID stations"
-   - Add topics: `water-monitoring`, `thailand`, `grafana`, `timeseries`, `python`
-   - Enable Issues and Discussions
-   - Set up GitHub Pages for documentation
-
-### **Initial Release**
-- **Version**: v1.0.0
-- **Release Notes**: Complete feature set with multi-database support
-- **Assets**: Include sample configuration files
-- **Documentation**: Link to comprehensive guides
-
-### **Community Features**
-- **Issues Template**: Bug reports and feature requests
-- **Pull Request Template**: Contribution guidelines
-- **Discussions**: Community support and questions
-- **Wiki**: Extended documentation and tutorials
-
-## 🎯 **Post-Publication Tasks**
-
-### **Community Building**
-- Create detailed issue templates
-- Set up GitHub Actions for CI/CD
-- Add code quality badges
-- Create project roadmap
-
-### **Documentation Enhancement**
-- Add video tutorials
-- Create API documentation
-- Add performance benchmarks
-- Create deployment examples
-
-### **Feature Development**
-- Mobile app integration
-- Additional database backends
-- Advanced alerting system
-- Predictive analytics
-
-## 📞 **Support Channels**
-
-- **GitHub Issues**: Bug reports and feature requests
-- **GitHub Discussions**: Community support and questions
-- **Documentation**: Comprehensive guides in docs/ directory
-- **Examples**: Working configurations and deployments
-
-## 🏆 **Project Highlights**
-
-### **Technical Excellence**
-- Clean, modular architecture
-- Comprehensive error handling
-- Production-ready deployment
-- Multi-database abstraction
-
-### **Documentation Quality**
-- Step-by-step installation guides
-- Troubleshooting for common issues
-- Migration procedures for updates
-- API and configuration references
-
-### **Community Ready**
-- Open source MIT license
-- Contributor guidelines
-- Development setup instructions
-- Code quality standards
-
----
-
-**The Thailand Water Level Monitor project is now fully prepared for GitHub publication with a professional structure, comprehensive documentation, and production-ready features.** 🌊

GITHUB_TOKEN_SETUP.md

@@ -1,114 +0,0 @@
-# 🔑 GitHub Token Setup Guide
-
-## 🎯 **Why You Need This**
-
-The Gitea Actions workflows use Trivy for security scanning, which needs to download vulnerability databases from GitHub. Without a GitHub token, you'll hit rate limits and the security scans will fail.
-
-## 🚀 **Quick Setup (5 minutes)**
-
-### **Step 1: Create GitHub Personal Access Token**
-
-1. **Go to GitHub**: https://github.com/settings/tokens
-2. **Click "Generate new token"** → "Generate new token (classic)"
-3. **Configure the token**:
-   - **Note**: `B4L Ping River Monitor - Gitea Actions`
-   - **Expiration**: `90 days` (or longer)
-   - **Scopes**: Select `public_repo` (for public repositories)
-4. **Click "Generate token"**
-5. **Copy the token** (you won't see it again!)
-
-### **Step 2: Add Token to Gitea Repository**
-
-1. **Go to your repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
-2. **Click "Settings"** (in the repository)
-3. **Click "Secrets"** in the left sidebar
-4. **Click "Add Secret"**
-5. **Configure the secret**:
-   - **Name**: `GITHUB_TOKEN`
-   - **Value**: Paste the token you copied from GitHub
-6. **Click "Add Secret"**
-
-### **Step 3: Verify It's Working**
-
-1. **Trigger a workflow** by pushing a commit or manually running the security workflow
-2. **Check the Actions tab** in your repository
-3. **Look for the message**: `✅ GITHUB_TOKEN is configured`
-
-## 🔒 **Security Best Practices**
-
-### **Token Permissions**
-- **Minimum required**: `public_repo` scope
-- **Never use**: `repo` scope unless you need private repo access
-- **Avoid**: Admin or write permissions
-
-### **Token Management**
-- **Set expiration**: Don't create tokens that never expire
-- **Regular rotation**: Update tokens every 90 days
-- **Monitor usage**: Check GitHub token usage in settings
-
-### **Repository Security**
-- **Only trusted contributors**: Should have access to repository secrets
-- **Audit regularly**: Review who has access to secrets
-- **Use organization secrets**: For multiple repositories
-
-## 🧪 **Testing the Setup**
-
-### **Manual Test**
-```bash
-# Trigger the security workflow manually
-# Go to: Repository → Actions → Security & Dependency Updates → Run workflow
-```
-
-### **Automatic Test**
-```bash
-# Push any change to trigger workflows
-git commit --allow-empty -m "Test GitHub token setup"
-git push
-```
-
-### **Check Workflow Logs**
-1. Go to Actions tab in your repository
-2. Click on the latest "Security & Dependency Updates" run
-3. Click on "Docker Security Scan" job
-4. Look for: `✅ GITHUB_TOKEN is configured`
-
-## ❌ **Troubleshooting**
-
-### **"GITHUB_TOKEN not configured" message**
-- **Problem**: Token not added to repository secrets
-- **Solution**: Follow Step 2 above, ensure exact name `GITHUB_TOKEN`
-
-### **"Bad credentials" error**
-- **Problem**: Token is invalid or expired
-- **Solution**: Generate a new token and update the secret
-
-### **Rate limit errors**
-- **Problem**: Token doesn't have correct permissions
-- **Solution**: Ensure token has `public_repo` scope
-
-### **Trivy still failing**
-- **Problem**: Network issues or GitHub API problems
-- **Solution**: Wait and retry, or check GitHub status page
-
-## 🎉 **Success Indicators**
-
-When everything is working correctly, you'll see:
-
-✅ **In workflow logs**: `✅ GITHUB_TOKEN is configured`  
-✅ **Security scans**: Complete without authentication errors  
-✅ **Trivy reports**: Generated and uploaded as artifacts  
-✅ **No rate limit errors**: In the workflow execution  
-
-## 📚 **Additional Resources**
-
-- [GitHub Personal Access Tokens Documentation](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
-- [Gitea Secrets Documentation](https://docs.gitea.io/en-us/usage/actions/#secrets)
-- [Trivy Action Documentation](https://github.com/aquasecurity/trivy-action)
-
----
-
-**Setup Time**: ~5 minutes  
-**Token Validity**: 90 days (recommended)  
-**Security Level**: High (read-only public repo access)  
-
-Your workflows will now run smoothly with proper GitHub API authentication! 🚀

Makefile

@@ -120,10 +120,6 @@ docker-logs:
 docs:
 	cd docs && make html
 
-# Database management
-db-migrate:
-	uv run python scripts/migrate_geolocation.py
-
 # Monitoring
 health-check:
 	curl -f http://localhost:8000/health || exit 1
@@ -149,9 +145,6 @@ setup-postgres:
 test-postgres:
 	uv run python -c "from scripts.setup_postgres import test_postgres_connection; from src.config import Config; config = Config.get_database_config(); test_postgres_connection(config['connection_string'])"
 
-encode-password:
-	uv run python scripts/encode_password.py
-
 migrate-sqlite:
 	uv run python scripts/migrate_sqlite_to_postgres.py
 
@@ -161,16 +154,6 @@ migrate-fast:
 analyze-sqlite:
 	uv run python scripts/migrate_sqlite_to_postgres.py --dry-run
 
-# Distribution
-build-exe:
-	uv run python build_simple.py
-
-package: build-exe
-	@echo "Creating distribution package..."
-	@if exist dist\ping-river-monitor-distribution.zip del dist\ping-river-monitor-distribution.zip
-	@cd dist && powershell -Command "Compress-Archive -Path * -DestinationPath ping-river-monitor-distribution.zip -Force"
-	@echo "✅ Distribution package created: dist/ping-river-monitor-distribution.zip"
-
 # Git helpers
 git-setup:
 	git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git

README.md

@@ -1,510 +1,157 @@
-# Northern Thailand Ping River Monitor 🏔️
+# Northern Thailand Ping River Monitor
 
-A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.
+Live water levels, discharge, rainfall and machine-learning flood forecasts for the
+Ping River basin around Chiang Mai. Collects hourly gauge data from public sources,
+keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST
+API, and 6/12/24-hour flood-risk forecasts per gauge.
 
-[![CI/CD](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Documentation](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Python](https://img.shields.io/badge/Python-3.9+-blue.svg)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-0.104+-green.svg)](https://fastapi.tiangolo.com) [![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Version](https://img.shields.io/badge/Version-v3.1.3-blue.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/releases)
+**Live: [water.buildfor.life](https://water.buildfor.life/)** · API reference at
+[/docs](https://water.buildfor.life/docs) · built by [buildfor.life](https://buildfor.life)
+after the [October 2024 flood](https://buildfor.life/blog/chiang-mai-flood-2024/) —
+background in [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
 
-## 🌟 Features
+[![CI](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
+[![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
+[![Docs](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
+[![Python 3.11](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
 
-### 📊 **Real-time Data Collection**
-- **16 Monitoring Stations** across Thailand
-- **15-minute Collection Frequency** with intelligent scheduling
-- **Automatic Gap Filling** for missing historical data
-- **Data Validation** and error recovery mechanisms
-- **Rate Limiting** to prevent API abuse
+## What it does
 
-### 🌐 **Web API Interface (NEW!)**
-- **FastAPI-powered REST API** with interactive documentation
-- **Station Management** - Add, update, and remove monitoring stations
-- **Real-time health monitoring** and system status
-- **Manual data collection triggers** via web interface
-- **Comprehensive metrics** and performance monitoring
-- **CORS support** for web applications
+- **Collects** hourly water level and discharge from 16 Royal Irrigation Department
+  (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly
+  rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment
+  rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and
+  its quirks: [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md).
+- **Fills gaps.** The raw RID grid had readings for ~56 % of hours; a full-history
+  re-fetch plus HII cross-fill brought it to ~93 %. `GET /api/stats` reports the
+  current figure.
+- **Forecasts.** Per gauge and horizon, a gradient-boosted model predicts the rise
+  within 6/12/24 h and the probability of crossing the station's warning and danger
+  levels. Trained on the monitor's own history plus catchment rain; evaluated
+  rolling-origin, event by event. On the October 2024 record flood, trained only on
+  data through August 2024, the first alert came **13 hours before** P.1 crossed
+  3.70 m. Everything about the model, including what did not work:
+  [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md).
+- **Shows it.** A Leaflet map with the river drawn as OSM geometry and styled by live
+  discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the
+  forecast card, a replay of the 2024 flood, English/Thai, light/dark.
+- **Notifies.** Public push alerts over a self-hosted [ntfy](https://ntfy.sh): one
+  message when a gauge crosses its warning or danger level, one all-clear on the
+  way down, an opt-in early-warning topic from the model, nothing in between.
+  Subscribe from the free app, no account. Matrix room alerts for a team are
+  also supported.
 
-### 🗄️ **Multi-Database Support**
-- **VictoriaMetrics** (Recommended) - High-performance time-series
-- **InfluxDB** - Purpose-built time-series database
-- **PostgreSQL + TimescaleDB** - Relational with time-series optimization
-- **MySQL** - Traditional relational database
-- **SQLite** - Local development and testing
+## Quick start
 
-### 🗺️ **Geolocation Support**
-- **Grafana Geomap** integration ready
-- **GPS coordinates** and geohash support
-- **Interactive mapping** of water stations
-
-### 📈 **Visualization & Monitoring**
-- **Pre-built Grafana dashboards**
-- **Real-time alerts** and notifications
-- **Historical trend analysis**
-- **Built-in metrics collection** (counters, gauges, histograms)
-- **Health checks** for database, API, and system resources
-
-### 🚀 **Production Ready**
-- **Docker containerization** with multi-service support
-- **Systemd service** configuration
-- **HTTPS support** with SSL certificates
-- **Comprehensive logging** with rotation and colored output
-- **Type safety** with Pydantic models and type hints
-- **Custom exception handling** for better error management
-
-## 🚀 Quick Start
-
-### Prerequisites
-- Python 3.9 or higher
-- Internet connection for data fetching
-- Database server (optional - SQLite works out of the box)
-
-### Installation
+Python **3.11** (3.13 breaks the pinned `psycopg2-binary`), PostgreSQL for anything
+beyond a quick look, [uv](https://docs.astral.sh/uv/).
 
 ```bash
-# Clone the repository
 git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 cd Northern-Thailand-Ping-River-Monitor
-
-# Quick setup with Make
-make dev-setup
-
-# Or manual setup:
-python -m venv venv
-source venv/bin/activate  # Windows: venv\Scripts\activate
-pip install -r requirements.txt
-cp .env.example .env
+uv sync --python 3.11
+cp .env.example .env            # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
+uv run python run.py --web-api  # dashboard + API on http://localhost:8000
 ```
 
-### Basic Usage
+`DB_TYPE=sqlite` works for the dashboard and API; the forecasting path expects the
+PostgreSQL history.
 
 ```bash
-# Test run with SQLite (default)
-make run-test
-# or: python run.py --test
-
-# Run continuous monitoring
-make run
-# or: python run.py
-
-# Start web API server (NEW!)
-make run-api
-# or: python run.py --web-api
-
-# Run all tests
-make test
-
-# Demo different databases
-python src/demo_databases.py
+uv run python run.py --status          # collector status
+uv run python run.py --test            # one collection cycle
+uv run python run.py --fill-gaps 7     # re-fetch the last 7 days from RID
+uv run python run.py --collect-hii     # one ThaiWater/HII collection cycle
+uv run python run.py --alert-check     # evaluate thresholds, notify Matrix
+uv run python scripts/train_flood_model.py --stations all   # retrain (~12 min)
+make test                              # pytest, synthetic data, no network
+make format                            # black + isort (the CI contract)
 ```
 
-### 🌐 Web API Interface (NEW!)
+## API
 
-The system now includes a comprehensive FastAPI web interface:
+Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
+Asia/Bangkok wall-clock without an offset suffix.
+
+| Endpoint | Returns |
+| --- | --- |
+| `GET /stations` | The 16 RID gauges: code, Thai/English names, coordinates |
+| `GET /measurements/latest?limit=N` | Newest reading per station |
+| `GET /measurements/history/{code}?hours=N` | Hourly history; or `?start=YYYY-MM-DD&end=YYYY-MM-DD`; `limit` ≤ 100000 |
+| `GET /forecast` | Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
+| `GET /api/forecast/history/{code}?hours=N&horizon=24` | Forecasts as issued, for auditing lead time after the fact |
+| `GET /api/hii/rainfall/latest`, `/api/hii/waterlevel/latest` | Latest ThaiWater/HII gauge readings |
+| `GET /api/hii/rainfall/catchment?days=N` | HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
+| `GET /api/forecast/skill?station_code=P.1` | Issued forecasts vs what happened, per deployed model version |
+| `GET /api/notifications` | ntfy server and topic names for the subscribe panel |
+| `GET /api/stats` | Row counts per source, date range, coverage |
+| `GET /health` | DB / upstream / memory checks |
+
+Interactive reference with schemas: [water.buildfor.life/docs](https://water.buildfor.life/docs).
+Responses are cached briefly server-side; poll no faster than once a minute — the data
+changes hourly.
+
+## Deployment
+
+Production is a systemd unit on a small VPS behind Cloudflare, updated by `git pull`.
+`scripts/install.sh` (run as root from a checkout) creates the `water-monitor` user,
+deploys to `/opt/thailand-water-monitor`, runs `uv sync` into `.venv`, installs
+`water-monitor.service` and the monthly `water-monitor-retrain.timer`.
 
 ```bash
-# Start the web API
-python run.py --web-api
-
-# Access the API at:
-# - Dashboard: http://localhost:8000
-# - Interactive docs: http://localhost:8000/docs
-# - Health check: http://localhost:8000/health
-# - Latest data: http://localhost:8000/measurements/latest
-```
-
-**Key API Endpoints:**
-- `GET /` - Web dashboard
-- `GET /health` - System health status
-- `GET /metrics` - Application metrics
-- `GET /stations` - List all monitoring stations
-- `POST /stations` - Add new monitoring station
-- `PUT /stations/{id}` - Update station information
-- `DELETE /stations/{id}` - Remove monitoring station
-- `GET /measurements/latest` - Latest measurements
-- `GET /measurements/station/{code}` - Station-specific data
-- `POST /scrape/trigger` - Trigger manual data collection
-
-## 📊 Station Information
-
-The system monitors **16 water stations** along the Ping River Basin in Northern Thailand:
-
-| Station | Thai Name | English Name | Location |
-|---------|-----------|--------------|----------|
-| P.1 | สะพานนวรัฐ | Nawarat Bridge | Nakhon Sawan |
-| P.5 | สะพานท่านาง | Tha Nang Bridge | - |
-| P.20 | บ้านเชียงดาว | Ban Chiang Dao | Chiang Mai |
-| P.21 | บ้านริมใต้ | Ban Rim Tai | - |
-| P.4A | บ้านแม่แตง | Ban Mae Taeng | Chiang Mai |
-| P.67 | บ้านแม่แต | Ban Tae | - |
-| P.75 | บ้านช่อแล | Ban Chai Lat | - |
-| P.76 | บ้านแม่อีไฮ | Banb Mae I Hai | - |
-| P.77 | บ้านสบแม่สะป๊วด | Baan Sop Mae Sapuord | - |
-| P.81 | บ้านโป่ง | Ban Pong | - |
-| P.82 | บ้านสบวิน | Ban Sob win | - |
-| P.84 | บ้านพันตน | Ban Panton | - |
-| P.85 | บ้านหล่ายแก้ว | Baan Lai Kaew | - |
-| P.87 | บ้านป่าซาง | Ban Pa Sang | - |
-| P.92 | บ้านเมืองกึ๊ด | Ban Muang Aut | - |
-| P.103 | สะพานวงแหวนรอบ 3 | Ring Bridge 3 | Bangkok |
-
-### Data Metrics
-- **Water Level**: Measured in meters (m)
-- **Discharge**: Flow rate in cubic meters per second (cms)
-- **Discharge Percentage**: Relative to station capacity
-- **Timestamp**: Thai time (UTC+7) with Buddhist calendar support
-
-## 🗄️ Database Configuration
-
-### VictoriaMetrics (Recommended)
-
-**High-performance time-series database with excellent compression and query speed.**
-
-```bash
-# Environment variables
-export DB_TYPE=victoriametrics
-export VM_HOST=localhost
-export VM_PORT=8428
-
-# Quick start with Docker
-docker run -d \
-  --name victoriametrics \
-  -p 8428:8428 \
-  -v victoria-metrics-data:/victoria-metrics-data \
-  victoriametrics/victoria-metrics:latest \
-  --storageDataPath=/victoria-metrics-data \
-  --retentionPeriod=2y \
-  --httpListenAddr=:8428
-```
-
-### Complete Stack with Grafana
-
-```bash
-# Start the complete monitoring stack
-docker-compose -f docker-compose.victoriametrics.yml up -d
-
-# Access Grafana at http://localhost:3000
-# Username: admin, Password: admin_password
-```
-
-### Other Database Options
-
-<details>
-<summary>InfluxDB Configuration</summary>
-
-```bash
-export DB_TYPE=influxdb
-export INFLUX_HOST=localhost
-export INFLUX_PORT=8086
-export INFLUX_DATABASE=water_monitoring
-export INFLUX_USERNAME=water_user
-export INFLUX_PASSWORD=your_password
-```
-</details>
-
-<details>
-<summary>PostgreSQL Configuration</summary>
-
-```bash
-export DB_TYPE=postgresql
-export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring
-```
-</details>
-
-<details>
-<summary>MySQL Configuration</summary>
-
-```bash
-export DB_TYPE=mysql
-export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring
-```
-</details>
-
-## 📈 Grafana Dashboards
-
-### Pre-built Dashboard Features
-- **Real-time water levels** across all stations
-- **Historical trends** and patterns
-- **Discharge monitoring** with percentage indicators
-- **Station status** and health monitoring
-- **Geomap visualization** of station locations
-- **Alert thresholds** for critical water levels
-
-### Sample Queries
-
-**VictoriaMetrics/Prometheus:**
-```promql
-# Current water levels
-water_level
-
-# High discharge alerts
-water_discharge_percent > 80
-
-# Station-specific data
-water_level{station_code="P.1"}
-```
-
-**SQL Databases:**
-```sql
--- Latest readings from all stations
-SELECT s.station_code, s.english_name, m.water_level, m.discharge
-FROM stations s
-JOIN water_measurements m ON s.id = m.station_id
-WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);
-```
-
-## 🚀 Production Deployment
-
-### Docker Deployment
-
-```bash
-# Build the image
-docker build -t thailand-water-monitor .
-
-# Run with environment variables
-docker run -d \
-  --name water-monitor \
-  -e DB_TYPE=victoriametrics \
-  -e VM_HOST=victoriametrics \
-  thailand-water-monitor
-```
-
-### Systemd Service (Linux)
-
-The install script sets everything up: a dedicated `water-monitor` system user,
-a deploy to `/opt/thailand-water-monitor`, a uv-managed virtualenv, and the
-enabled systemd unit.
-
-```bash
-# From a checkout of the repo, as root:
 sudo bash scripts/install.sh
-
-# Then start and check:
 sudo systemctl start water-monitor.service
-systemctl status water-monitor.service
+systemctl list-timers water-monitor-retrain.timer
 ```
 
-Fill in `/opt/thailand-water-monitor/.env` (Matrix token/room, DB settings)
-before starting if the script reports it is missing.
+The retrain timer runs `scripts/retrain.sh`, which trains into `models/.staging`,
+refuses to promote anything that is not a rain-enabled (`hgb-v3+`) set covering the
+expected stations, and renames the bundles into place. Details and the operations
+runbook: [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) sections 6–8.
 
-<details>
-<summary>Manual setup (if you prefer not to use the script)</summary>
-
-```bash
-sudo useradd --system --no-create-home --shell /usr/sbin/nologin water-monitor
-sudo cp scripts/water-monitor.service /etc/systemd/system/
-sudo systemctl enable water-monitor.service
-sudo systemctl start water-monitor.service
-```
-</details>
-
-### Migration for Existing Systems
-
-If you have an existing installation, use the migration script to add geolocation support:
-
-```bash
-# Stop the service
-sudo systemctl stop water-monitor
-
-# Run migration
-python scripts/migrate_geolocation.py
-
-# Restart the service
-sudo systemctl start water-monitor
-```
-
-## 🔧 Command Line Tools
-
-### Main Application
-```bash
-python src/water_scraper_v3.py                    # Run continuous monitoring
-python src/water_scraper_v3.py --test             # Single test cycle
-python src/water_scraper_v3.py --help             # Show help
-```
-
-### Data Management
-```bash
-python src/water_scraper_v3.py --check-gaps 7     # Check for missing data (7 days)
-python src/water_scraper_v3.py --fill-gaps 7      # Fill missing data gaps
-python src/water_scraper_v3.py --update-data 2    # Update existing data (2 days)
-```
-
-### Database Testing
-```bash
-python src/demo_databases.py                      # SQLite demo
-python src/demo_databases.py victoriametrics      # VictoriaMetrics demo
-python src/demo_databases.py all                  # Test all databases
-```
-
-## 📚 Documentation
-
-### Core Documentation
-- **[Data Sources & API Catalog](docs/DATA_SOURCES.md)** - Every ingested and available data source (RID, ThaiWater/HII, dams, rainfall, forecasts)
-- **[Installation Guide](docs/DATABASE_DEPLOYMENT_GUIDE.md)** - Complete setup instructions
-- **[Scheduler Guide](docs/ENHANCED_SCHEDULER_GUIDE.md)** - 15-minute scheduling system
-- **[Geolocation Guide](docs/GEOLOCATION_GUIDE.md)** - Grafana geomap integration
-- **[Gap Filling Guide](docs/GAP_FILLING_GUIDE.md)** - Data integrity management
-
-### Deployment Guides
-- **[VictoriaMetrics Setup](docs/VICTORIAMETRICS_SETUP.md)** - High-performance deployment
-- **[HTTPS Configuration](docs/HTTPS_CONFIGURATION.md)** - Secure deployment
-- **[Debian Troubleshooting](docs/DEBIAN_TROUBLESHOOTING.md)** - Linux deployment issues
-
-### References
-- **[Notable Documents](docs/references/NOTABLE_DOCUMENTS.md)** - Official Thai government resources
-- **[Migration Guide](docs/MIGRATION_QUICKSTART.md)** - Updating existing systems
-
-## 🔍 Troubleshooting
-
-### Common Issues
-
-**Database Connection Errors:**
-```bash
-# Check database status
-python src/demo_databases.py
-
-# Test specific database
-python src/demo_databases.py victoriametrics
-```
-
-**Missing Data:**
-```bash
-# Check for gaps
-python src/water_scraper_v3.py --check-gaps 7
-
-# Fill missing data
-python src/water_scraper_v3.py --fill-gaps 7
-```
-
-**Service Issues:**
-```bash
-# Check service status
-sudo systemctl status water-monitor
-
-# View logs
-sudo journalctl -u water-monitor -f
-```
-
-### Health Checks
-
-```bash
-# VictoriaMetrics health
-curl http://localhost:8428/health
-
-# Check latest data
-curl "http://localhost:8428/api/v1/query?query=water_level"
-
-# Application logs
-tail -f water_monitor.log
-```
-
-## 🌐 API Integration
-
-### VictoriaMetrics API Examples
-
-```bash
-# Query current water levels
-curl "http://localhost:8428/api/v1/query?query=water_level"
-
-# Query discharge rates for last hour
-curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"
-
-# Query specific station
-curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
-
-# High discharge alerts
-curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
-```
-
-## 📊 Performance
-
-### System Requirements
-- **CPU**: 1-2 cores (minimal load)
-- **RAM**: 512MB - 2GB (depending on database)
-- **Storage**: 1GB+ (for historical data)
-- **Network**: Stable internet connection
-
-### Performance Metrics
-- **Data Collection**: ~300 data points every 15 minutes
-- **Database Write Speed**: 1000+ points/second (VictoriaMetrics)
-- **Query Response**: <100ms for recent data
-- **Storage Efficiency**: 70x compression vs. raw data
-
-## 🤝 Contributing
-
-Contributions are welcome! Please:
-
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Add tests if applicable
-5. Submit a pull request
-
-### Development Setup
-
-```bash
-# Clone your fork
-git clone https://github.com/your-username/thailand-water-monitor.git
-cd thailand-water-monitor
-
-# Install development dependencies
-pip install -r requirements.txt
-pip install pytest black flake8
-
-# Run tests
-pytest
-
-# Format code
-black src/
-```
-
-## 📄 License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## 🙏 Acknowledgments
-
-- **Royal Irrigation Department (RID)** of Thailand for providing the data API
-- **VictoriaMetrics** team for the excellent time-series database
-- **Grafana** team for the visualization platform
-- **Python community** for the amazing libraries and tools
-
-## 📞 Support
-
-- **Issues**: [GitHub Issues](https://github.com/your-username/thailand-water-monitor/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/your-username/thailand-water-monitor/discussions)
-- **Documentation**: [Project Wiki](https://github.com/your-username/thailand-water-monitor/wiki)
-
----
-
-## 📁 Project Structure
+## Repository layout
 
 ```
-Northern-Thailand-Ping-River-Monitor/
-├── src/                    # Main application code
-├── tests/                  # Test suite
-├── docs/                   # Documentation
-├── grafana/               # Grafana dashboards
-├── scripts/               # Utility scripts
-├── docker-compose.yml     # Docker deployment
-├── Makefile              # Development tasks
-└── requirements.txt      # Dependencies
+src/                 collector, API (web_api.py), dashboard (static/dashboard.html)
+src/ml/              features, training, evaluation harness, prediction, rain/dam/HII loaders
+scripts/             train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
+tests/               pytest suite (synthetic data; no DB or network)
+docs/                FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
+models/              trained bundles + metrics.json (gitignored) and evaluation results (tracked)
+.gitea/workflows/    ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
 ```
 
-See [docs/PROJECT_STRUCTURE.md](docs/PROJECT_STRUCTURE.md) for detailed architecture information.
+## Documentation
 
-## 🔄 CI/CD & Automation
+- [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
+- [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) — every ingested and candidate source, endpoints, quirks
+- [docs/STATION_MANAGEMENT_GUIDE.md](docs/STATION_MANAGEMENT_GUIDE.md) — adding/editing gauges
+- [docs/DATABASE_DEPLOYMENT_GUIDE.md](docs/DATABASE_DEPLOYMENT_GUIDE.md), [POSTGRESQL_SETUP.md](POSTGRESQL_SETUP.md) — database setup
+- [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md) — public push alerts: topics, semantics, ntfy deployment
+- [docs/MATRIX_QUICK_START.md](docs/MATRIX_QUICK_START.md) — Matrix room alerts for a team
+- [docs/GAP_FILLING_GUIDE.md](docs/GAP_FILLING_GUIDE.md) — data integrity tooling
+- [docs/references/NOTABLE_DOCUMENTS.md](docs/references/NOTABLE_DOCUMENTS.md) — official Thai government resources
+- Public overview: [buildfor.life/docs/tooling/ping-river-monitor](https://buildfor.life/docs/tooling/ping-river-monitor/)
 
-The project includes comprehensive Gitea Actions workflows:
+Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
+dashboards under `grafana/` are supported by the adapters but not what production
+runs; see [docs/VICTORIAMETRICS_SETUP.md](docs/VICTORIAMETRICS_SETUP.md) if you want them.
 
-- **🧪 CI/CD Pipeline** - Automated testing, building, and deployment
-- **🔒 Security Scanning** - Daily vulnerability and dependency checks
-- **📚 Documentation** - Automated API docs and validation
-- **🚀 Release Management** - Automated releases with multi-arch Docker builds
+## Contributing
 
-See [docs/GITEA_WORKFLOWS.md](docs/GITEA_WORKFLOWS.md) for detailed workflow documentation.
+`make format` before committing (black 88 columns, isort black profile — the CI gate),
+`make test` must stay green, tests use synthetic data only. See
+[CONTRIBUTING.md](CONTRIBUTING.md). Issues and merge requests on
+[git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
 
-## 🔗 Repository
+## Data sources and thanks
 
-- **Main Repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
-- **Issues**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
-- **Actions**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
-- **Documentation**: [docs/](docs/)
+Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) /
+ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry;
+Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All
+instruments are theirs; we aggregate, store, fill gaps and forecast.
 
-**Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin**
+## License
+
+MIT — see [LICENSE](LICENSE).

build_executable.py

@@ -1,311 +0,0 @@
-#!/usr/bin/env python3
-"""
-Build script to create a standalone executable for Northern Thailand Ping River Monitor
-"""
-
-import os
-import shutil
-import sys
-from pathlib import Path
-
-
-def create_spec_file():
-    """Create PyInstaller spec file"""
-    spec_content = """
-# -*- mode: python ; coding: utf-8 -*-
-
-block_cipher = None
-
-# Data files to include
-data_files = [
-    ('.env', '.'),
-    ('sql/*.sql', 'sql'),
-    ('README.md', '.'),
-    ('POSTGRESQL_SETUP.md', '.'),
-    ('SQLITE_MIGRATION.md', '.'),
-]
-
-# Hidden imports that PyInstaller might miss
-hidden_imports = [
-    'psycopg2',
-    'psycopg2-binary',
-    'sqlalchemy.dialects.postgresql',
-    'sqlalchemy.dialects.sqlite',
-    'sqlalchemy.dialects.mysql',
-    'influxdb',
-    'pymysql',
-    'dotenv',
-    'pydantic',
-    'fastapi',
-    'uvicorn',
-    'schedule',
-    'pandas',
-    'requests',
-    'psutil',
-]
-
-a = Analysis(
-    ['run.py'],
-    pathex=['.'],
-    binaries=[],
-    datas=data_files,
-    hiddenimports=hidden_imports,
-    hookspath=[],
-    hooksconfig={},
-    runtime_hooks=[],
-    excludes=[
-        'tkinter',
-        'matplotlib',
-        'PIL',
-        'jupyter',
-        'notebook',
-        'IPython',
-    ],
-    win_no_prefer_redirects=False,
-    win_private_assemblies=False,
-    cipher=block_cipher,
-    noarchive=False,
-)
-
-pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
-
-exe = EXE(
-    pyz,
-    a.scripts,
-    a.binaries,
-    a.zipfiles,
-    a.datas,
-    [],
-    name='ping-river-monitor',
-    debug=False,
-    bootloader_ignore_signals=False,
-    strip=False,
-    upx=True,
-    upx_exclude=[],
-    runtime_tmpdir=None,
-    console=True,
-    disable_windowed_traceback=False,
-    argv_emulation=False,
-    target_arch=None,
-    codesign_identity=None,
-    entitlements_file=None,
-    icon='icon.ico' if os.path.exists('icon.ico') else None,
-)
-"""
-
-    with open("ping-river-monitor.spec", "w") as f:
-        f.write(spec_content.strip())
-
-    print("[OK] Created ping-river-monitor.spec")
-
-
-def install_pyinstaller():
-    """Install PyInstaller if not present"""
-    try:
-        import PyInstaller
-
-        print("[OK] PyInstaller already installed")
-    except ImportError:
-        print("Installing PyInstaller...")
-        os.system("uv add --dev pyinstaller")
-        print("[OK] PyInstaller installed")
-
-
-def build_executable():
-    """Build the executable"""
-    print("🔨 Building executable...")
-
-    # Clean previous builds
-    if os.path.exists("dist"):
-        shutil.rmtree("dist")
-    if os.path.exists("build"):
-        shutil.rmtree("build")
-
-    # Build with PyInstaller using uv
-    result = os.system("uv run pyinstaller ping-river-monitor.spec --clean --noconfirm")
-
-    if result == 0:
-        print("✅ Executable built successfully!")
-
-        # Copy additional files to dist directory
-        dist_dir = Path("dist")
-        if dist_dir.exists():
-            # Copy .env file if it exists
-            if os.path.exists(".env"):
-                shutil.copy2(".env", dist_dir / ".env")
-                print("✅ Copied .env file")
-
-            # Copy documentation
-            for doc in ["README.md", "POSTGRESQL_SETUP.md", "SQLITE_MIGRATION.md"]:
-                if os.path.exists(doc):
-                    shutil.copy2(doc, dist_dir / doc)
-                    print(f"✅ Copied {doc}")
-
-            # Copy SQL files
-            if os.path.exists("sql"):
-                shutil.copytree("sql", dist_dir / "sql", dirs_exist_ok=True)
-                print("✅ Copied SQL files")
-
-            print(f"\n🎉 Executable created: {dist_dir / 'ping-river-monitor.exe'}")
-            print(f"📁 All files in: {dist_dir.absolute()}")
-
-    else:
-        print("❌ Build failed!")
-        return False
-
-    return True
-
-
-def create_batch_files():
-    """Create convenient batch files"""
-    batch_files = {
-        "start.bat": """@echo off
-echo Starting Ping River Monitor...
-ping-river-monitor.exe
-pause
-""",
-        "start-api.bat": """@echo off
-echo Starting Ping River Monitor Web API...
-ping-river-monitor.exe --web-api
-pause
-""",
-        "test.bat": """@echo off
-echo Running Ping River Monitor test...
-ping-river-monitor.exe --test
-pause
-""",
-        "status.bat": """@echo off
-echo Checking Ping River Monitor status...
-ping-river-monitor.exe --status
-pause
-""",
-    }
-
-    dist_dir = Path("dist")
-    for filename, content in batch_files.items():
-        batch_file = dist_dir / filename
-        with open(batch_file, "w") as f:
-            f.write(content)
-        print(f"✅ Created {filename}")
-
-
-def create_readme():
-    """Create deployment README"""
-    readme_content = """# Ping River Monitor - Standalone Executable
-
-This is a standalone executable version of the Northern Thailand Ping River Monitor.
-
-## Quick Start
-
-1. **Configure Database**: Edit `.env` file with your PostgreSQL settings
-2. **Test Connection**: Double-click `test.bat`
-3. **Start Monitoring**: Double-click `start.bat`
-4. **Web Interface**: Double-click `start-api.bat`
-
-## Files Included
-
-- `ping-river-monitor.exe` - Main executable
-- `.env` - Configuration file (EDIT THIS!)
-- `start.bat` - Start continuous monitoring
-- `start-api.bat` - Start web API server
-- `test.bat` - Run a test cycle
-- `status.bat` - Check system status
-- `README.md`, `POSTGRESQL_SETUP.md` - Documentation
-- `sql/` - Database initialization scripts
-
-## Configuration
-
-Edit `.env` file:
-```
-DB_TYPE=postgresql
-POSTGRES_HOST=your-server-ip
-POSTGRES_PORT=5432
-POSTGRES_DB=water_monitoring
-POSTGRES_USER=your-username
-POSTGRES_PASSWORD=your-password
-```
-
-## Usage
-
-### Command Line
-```cmd
-# Continuous monitoring
-ping-river-monitor.exe
-
-# Single test run
-ping-river-monitor.exe --test
-
-# Web API server
-ping-river-monitor.exe --web-api
-
-# Check status
-ping-river-monitor.exe --status
-```
-
-### Batch Files
-- Just double-click the `.bat` files for easy operation
-
-## Troubleshooting
-
-1. **Database Connection Issues**
-   - Check `.env` file settings
-   - Verify PostgreSQL server is accessible
-   - Test with `test.bat`
-
-2. **Permission Issues**
-   - Run as administrator if needed
-   - Check firewall settings for API mode
-
-3. **Log Files**
-   - Check `water_monitor.log` for detailed logs
-   - Logs are created in the same directory as the executable
-
-## Support
-
-For issues or questions, check the documentation files included.
-"""
-
-    with open("dist/DEPLOYMENT_README.txt", "w") as f:
-        f.write(readme_content)
-
-    print("✅ Created DEPLOYMENT_README.txt")
-
-
-def main():
-    """Main build process"""
-    print("Building Ping River Monitor Executable")
-    print("=" * 50)
-
-    # Check if we're in the right directory
-    if not os.path.exists("run.py"):
-        print(
-            "❌ Error: run.py not found. Please run this from the project root directory."
-        )
-        return False
-
-    # Install PyInstaller
-    install_pyinstaller()
-
-    # Create spec file
-    create_spec_file()
-
-    # Build executable
-    if not build_executable():
-        return False
-
-    # Create convenience files
-    create_batch_files()
-    create_readme()
-
-    print("\n" + "=" * 50)
-    print("🎉 BUILD COMPLETE!")
-    print("📁 Check the 'dist' folder for your executable")
-    print("💡 Edit the .env file before distributing")
-    print("🚀 Ready for deployment!")
-
-    return True
-
-
-if __name__ == "__main__":
-    success = main()
-    sys.exit(0 if success else 1)

build_simple.py

@@ -1,112 +0,0 @@
-#!/usr/bin/env python3
-"""
-Simple build script for standalone executable
-"""
-
-import os
-import shutil
-import sys
-from pathlib import Path
-
-
-def main():
-    print("Building Ping River Monitor Executable")
-    print("=" * 50)
-
-    # Check if PyInstaller is installed
-    try:
-        import PyInstaller
-
-        print("[OK] PyInstaller available")
-    except ImportError:
-        print("[INFO] Installing PyInstaller...")
-        os.system("uv add --dev pyinstaller")
-
-    # Clean previous builds
-    if os.path.exists("dist"):
-        shutil.rmtree("dist")
-        print("[CLEAN] Removed old dist directory")
-    if os.path.exists("build"):
-        shutil.rmtree("build")
-        print("[CLEAN] Removed old build directory")
-
-    # Build command with all necessary options
-    cmd = [
-        "uv",
-        "run",
-        "pyinstaller",
-        "--onefile",
-        "--console",
-        "--name=ping-river-monitor",
-        "--add-data=.env;.",
-        "--add-data=sql;sql",
-        "--add-data=README.md;.",
-        "--add-data=POSTGRESQL_SETUP.md;.",
-        "--add-data=SQLITE_MIGRATION.md;.",
-        "--hidden-import=psycopg2",
-        "--hidden-import=sqlalchemy.dialects.postgresql",
-        "--hidden-import=sqlalchemy.dialects.sqlite",
-        "--hidden-import=dotenv",
-        "--hidden-import=pydantic",
-        "--hidden-import=fastapi",
-        "--hidden-import=uvicorn",
-        "--hidden-import=schedule",
-        "--hidden-import=pandas",
-        "--clean",
-        "--noconfirm",
-        "run.py",
-    ]
-
-    print("[BUILD] Running PyInstaller...")
-    print("[CMD] " + " ".join(cmd))
-
-    result = os.system(" ".join(cmd))
-
-    if result == 0:
-        print("[SUCCESS] Executable built successfully!")
-
-        # Copy .env file to dist if it exists
-        if os.path.exists(".env") and os.path.exists("dist"):
-            shutil.copy2(".env", "dist/.env")
-            print("[COPY] .env file copied to dist/")
-
-        # Create batch files for easy usage
-        batch_files = {
-            "start.bat": """@echo off
-echo Starting Ping River Monitor...
-ping-river-monitor.exe
-pause
-""",
-            "start-api.bat": """@echo off
-echo Starting Web API...
-ping-river-monitor.exe --web-api
-pause
-""",
-            "test.bat": """@echo off
-echo Running test...
-ping-river-monitor.exe --test
-pause
-""",
-        }
-
-        for filename, content in batch_files.items():
-            if os.path.exists("dist"):
-                with open(f"dist/{filename}", "w") as f:
-                    f.write(content)
-                print(f"[CREATE] {filename}")
-
-        print("\n" + "=" * 50)
-        print("BUILD COMPLETE!")
-        print(f"Executable: dist/ping-river-monitor.exe")
-        print("Batch files: start.bat, start-api.bat, test.bat")
-        print("Don't forget to edit .env file before using!")
-
-        return True
-    else:
-        print("[ERROR] Build failed!")
-        return False
-
-
-if __name__ == "__main__":
-    success = main()
-    sys.exit(0 if success else 1)

docs/ENHANCED_SCHEDULER_GUIDE.md

@@ -1,293 +0,0 @@
-# Enhanced Scheduler Guide
-
-This guide explains the new 15-minute scheduling system that runs continuously throughout each hour to ensure comprehensive data coverage.
-
-## ✅ **New Scheduling Behavior**
-
-### **15-Minute Schedule Pattern**
-- **Timing**: Runs every 15 minutes: 1:00, 1:15, 1:30, 1:45, 2:00, 2:15, 2:30, 2:45, etc.
-- **Hourly Full Checks**: At :00 minutes (includes gap filling and data updates)
-- **Quarter-Hour Quick Checks**: At :15, :30, :45 minutes (data fetch only)
-- **Continuous Coverage**: Ensures no data is missed throughout each hour
-
-### **Operation Types**
-- **Full Operations** (at :00): Data fetching + gap filling + data updates
-- **Quick Operations** (at :15, :30, :45): Data fetching only for performance
-
-## 🔧 **Technical Implementation**
-
-### **Scheduler States**
-```python
-# State tracking variables
-self.last_successful_update = None  # Timestamp of last successful data update
-self.retry_mode = False            # Whether in quick check mode (skip gap filling)
-self.next_hourly_check = None      # Next scheduled hourly check
-```
-
-### **Quarter-Hour Check Process**
-```python
-def quarter_hour_check(self):
-    """15-minute check for new data"""
-    current_time = datetime.datetime.now()
-    minute = current_time.minute
-    
-    # Determine if this is a full hourly check (at :00) or a quarter-hour check
-    if minute == 0:
-        logging.info("=== HOURLY CHECK (00:00) ===")
-        self.retry_mode = False  # Full check with gap filling and updates
-    else:
-        logging.info(f"=== 15-MINUTE CHECK ({minute:02d}:00) ===")
-        self.retry_mode = True   # Skip gap filling and updates on 15-min checks
-    
-    new_data_found = self.run_scraping_cycle()
-    
-    if new_data_found:
-        self.last_successful_update = datetime.datetime.now()
-        if minute == 0:
-            logging.info("New data found during hourly check")
-        else:
-            logging.info(f"New data found during 15-minute check at :{minute:02d}")
-    else:
-        if minute == 0:
-            logging.info("No new data found during hourly check")
-        else:
-            logging.info(f"No new data found during 15-minute check at :{minute:02d}")
-```
-
-### **Scheduler Setup**
-```python
-def start_scheduler(self):
-    """Start enhanced scheduler with 15-minute checks"""
-    # Schedule checks every 15 minutes (at :00, :15, :30, :45)
-    schedule.every().hour.at(":00").do(self.quarter_hour_check)
-    schedule.every().hour.at(":15").do(self.quarter_hour_check)
-    schedule.every().hour.at(":30").do(self.quarter_hour_check)
-    schedule.every().hour.at(":45").do(self.quarter_hour_check)
-    
-    while True:
-        schedule.run_pending()
-        time.sleep(30)  # Check every 30 seconds
-```
-
-## 📊 **New Data Detection Logic**
-
-### **Smart Detection Algorithm**
-```python
-def has_new_data(self) -> bool:
-    """Check if there is new data available since last successful update"""
-    # Get most recent timestamp from database
-    latest_data = self.get_latest_data(limit=1)
-    
-    # Check if we should have newer data by now
-    now = datetime.datetime.now()
-    expected_latest = now.replace(minute=0, second=0, microsecond=0)
-    
-    # If current time is past 5 minutes after the hour, we should have data
-    if now.minute >= 5:
-        if latest_timestamp < expected_latest:
-            return True  # New data expected
-    
-    # Check if we have data for the previous hour
-    previous_hour = expected_latest - datetime.timedelta(hours=1)
-    if latest_timestamp < previous_hour:
-        return True  # Missing recent data
-    
-    return False  # Data is up to date
-```
-
-### **Actual Data Verification**
-```python
-# Compare timestamps before and after scraping
-initial_timestamp = get_latest_timestamp_before_scraping()
-# ... perform scraping ...
-latest_timestamp = get_latest_timestamp_after_scraping()
-
-if initial_timestamp is None or latest_timestamp > initial_timestamp:
-    new_data_found = True
-    self.last_successful_update = datetime.datetime.now()
-```
-
-## 🚀 **Operational Modes**
-
-### **Mode 1: Full Hourly Operation (at :00)**
-- **Schedule**: Every hour at :00 minutes (1:00, 2:00, 3:00, etc.)
-- **Operations**: 
-  - ✅ Fetch current data
-  - ✅ Fill data gaps (last 7 days)
-  - ✅ Update existing data (last 2 days)
-- **Purpose**: Comprehensive data collection and maintenance
-
-### **Mode 2: Quick 15-Minute Checks (at :15, :30, :45)**
-- **Schedule**: Every 15 minutes at quarter-hour marks
-- **Operations**:
-  - ✅ Fetch current data only
-  - ❌ Skip gap filling (performance optimization)
-  - ❌ Skip data updates (performance optimization)
-- **Purpose**: Ensure no new data is missed between hourly checks
-
-## 📋 **Logging Output Examples**
-
-### **Successful Hourly Check (at :00)**
-```
-2025-07-26 01:00:00,123 - INFO - === HOURLY CHECK (00:00) ===
-2025-07-26 01:00:00,124 - INFO - Starting scraping cycle...
-2025-07-26 01:00:01,456 - INFO - Successfully fetched 384 data points from API
-2025-07-26 01:00:02,789 - INFO - New data found: 2025-07-26 01:00:00
-2025-07-26 01:00:03,012 - INFO - Filled 5 data gaps
-2025-07-26 01:00:04,234 - INFO - Updated 2 existing measurements
-2025-07-26 01:00:04,235 - INFO - New data found during hourly check
-```
-
-### **15-Minute Quick Check (at :15, :30, :45)**
-```
-2025-07-26 01:15:00,123 - INFO - === 15-MINUTE CHECK (15:00) ===
-2025-07-26 01:15:00,124 - INFO - Starting scraping cycle...
-2025-07-26 01:15:01,456 - INFO - Successfully fetched 299 data points from API
-2025-07-26 01:15:02,789 - INFO - New data found: 2025-07-26 01:00:00
-2025-07-26 01:15:02,790 - INFO - New data found during 15-minute check at :15
-```
-
-### **Continuous 15-Minute Pattern**
-```
-2025-07-26 01:00:00,123 - INFO - === HOURLY CHECK (00:00) ===
-2025-07-26 01:00:04,235 - INFO - New data found during hourly check
-
-2025-07-26 01:15:00,123 - INFO - === 15-MINUTE CHECK (15:00) ===
-2025-07-26 01:15:02,790 - INFO - No new data found during 15-minute check at :15
-
-2025-07-26 01:30:00,123 - INFO - === 15-MINUTE CHECK (30:00) ===
-2025-07-26 01:30:02,790 - INFO - No new data found during 15-minute check at :30
-
-2025-07-26 01:45:00,123 - INFO - === 15-MINUTE CHECK (45:00) ===
-2025-07-26 01:45:02,790 - INFO - No new data found during 15-minute check at :45
-
-2025-07-26 02:00:00,123 - INFO - === HOURLY CHECK (00:00) ===
-2025-07-26 02:00:04,235 - INFO - New data found during hourly check
-```
-
-## ⚙️ **Configuration Options**
-
-### **Environment Variables**
-```bash
-# Retry interval (default: 5 minutes)
-export RETRY_INTERVAL_MINUTES=5
-
-# Data availability buffer (default: 5 minutes after hour)
-export DATA_BUFFER_MINUTES=5
-
-# Gap filling days (default: 7 days)
-export GAP_FILL_DAYS=7
-
-# Update check days (default: 2 days)
-export UPDATE_DAYS=2
-```
-
-### **Scheduler Timing**
-```python
-# Hourly checks at top of hour
-schedule.every().hour.at(":00").do(self.hourly_check)
-
-# 5-minute retries (dynamically scheduled)
-schedule.every(5).minutes.do(self.retry_check).tag('retry')
-
-# Check every 30 seconds for responsive retry scheduling
-time.sleep(30)
-```
-
-## 🔍 **Performance Optimizations**
-
-### **Retry Mode Optimizations**
-- **Skip Gap Filling**: Avoids expensive historical data fetching during retries
-- **Skip Data Updates**: Avoids comparison operations during retries
-- **Focused API Calls**: Only fetches current day data during retries
-- **Reduced Database Queries**: Minimal database operations during retries
-
-### **Resource Management**
-- **API Rate Limiting**: 1-second delays between API calls
-- **Database Connection Pooling**: Efficient connection reuse
-- **Memory Efficiency**: Selective data processing
-- **Error Recovery**: Automatic retry with exponential backoff
-
-## 🛠️ **Troubleshooting**
-
-### **Common Scenarios**
-
-#### **Stuck in Retry Mode**
-```
-# Check if API is returning data
-curl -X POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx
-
-# Check database connectivity
-python water_scraper_v3.py --check-gaps 1
-
-# Manual data fetch test
-python water_scraper_v3.py --test
-```
-
-#### **Missing Hourly Triggers**
-```
-# Check system time synchronization
-timedatectl status
-
-# Verify scheduler is running
-ps aux | grep water_scraper
-
-# Check logs for scheduler activity
-tail -f water_monitor.log | grep "HOURLY CHECK"
-```
-
-#### **False New Data Detection**
-```
-# Check latest data in database
-sqlite3 water_monitoring.db "SELECT MAX(timestamp) FROM water_measurements;"
-
-# Verify timestamp parsing
-python -c "
-import datetime
-print('Current hour:', datetime.datetime.now().replace(minute=0, second=0, microsecond=0))
-"
-```
-
-## 📈 **Monitoring and Alerts**
-
-### **Key Metrics to Monitor**
-- **Hourly Success Rate**: Percentage of hourly checks that find new data
-- **Retry Duration**: How long system stays in retry mode
-- **Data Freshness**: Time since last successful data update
-- **API Response Time**: Performance of data fetching operations
-
-### **Alert Conditions**
-- **Extended Retry Mode**: System in retry mode for > 30 minutes
-- **No Data for 2+ Hours**: No new data found for extended period
-- **High Error Rate**: Multiple consecutive API failures
-- **Database Issues**: Connection or save failures
-
-### **Health Check Script**
-```bash
-#!/bin/bash
-# Check if system is stuck in retry mode
-RETRY_COUNT=$(tail -n 100 water_monitor.log | grep -c "RETRY CHECK")
-if [ $RETRY_COUNT -gt 6 ]; then
-    echo "WARNING: System may be stuck in retry mode ($RETRY_COUNT retries in last 100 log entries)"
-fi
-
-# Check data freshness
-LATEST_DATA=$(sqlite3 water_monitoring.db "SELECT MAX(timestamp) FROM water_measurements;")
-echo "Latest data timestamp: $LATEST_DATA"
-```
-
-## 🎯 **Best Practices**
-
-### **Production Deployment**
-1. **Monitor Logs**: Watch for retry mode patterns
-2. **Set Alerts**: Configure notifications for extended retry periods
-3. **Regular Maintenance**: Weekly gap filling and data validation
-4. **Backup Strategy**: Regular database backups before major operations
-
-### **Performance Tuning**
-1. **Adjust Buffer Time**: Modify data availability buffer based on API patterns
-2. **Optimize Retry Interval**: Balance between responsiveness and API load
-3. **Database Indexing**: Ensure proper indexes for timestamp queries
-4. **Connection Pooling**: Configure appropriate database connection limits
-
-This enhanced scheduler ensures reliable, efficient, and intelligent water level monitoring with automatic adaptation to data availability patterns.

docs/ENHANCEMENT_SUMMARY.md

@@ -1,227 +0,0 @@
-# 🚀 Northern Thailand Ping River Monitor - Enhancement Summary
-
-## 🎯 **What We've Accomplished**
-
-We've successfully transformed your water monitoring system from a simple scraper into a **production-ready, enterprise-grade monitoring platform** focused on the Ping River Basin in Northern Thailand, with modern web interfaces, station management capabilities, and comprehensive observability.
-
-## 🌟 **Major New Features Added**
-
-### 1. **FastAPI Web Interface** 🌐
-- **Interactive Dashboard** at `http://localhost:8000`
-- **REST API** with comprehensive endpoints
-- **Station Management** - Add, update, delete monitoring stations
-- **Real-time Health Monitoring** 
-- **Manual Data Collection Triggers**
-- **Interactive API Documentation** at `/docs`
-- **CORS Support** for web applications
-
-### 2. **Enhanced Architecture** 🏗️
-- **Type Safety** with Pydantic models and comprehensive type hints
-- **Data Validation Layer** with range checking and error handling
-- **Custom Exception Classes** for better error management
-- **Modular Design** with separated concerns
-
-### 3. **Observability & Monitoring** 📊
-- **Metrics Collection System** (counters, gauges, histograms)
-- **Health Checks** for database, API, and system resources
-- **Performance Tracking** with response times and success rates
-- **Enhanced Logging** with colors, rotation, and performance logs
-
-### 4. **Production Features** 🚀
-- **Rate Limiting** to prevent API abuse
-- **Request Tracking** with detailed statistics
-- **Configuration Validation** on startup
-- **Graceful Error Handling** and recovery
-- **Background Task Management**
-
-## 📁 **New Files Created**
-
-```
-src/
-├── models.py              # Data models and type definitions
-├── exceptions.py          # Custom exception classes
-├── validators.py          # Data validation layer
-├── metrics.py            # Metrics collection system
-├── health_check.py       # Health monitoring system
-├── rate_limiter.py       # Rate limiting and request tracking
-├── logging_config.py     # Enhanced logging configuration
-├── web_api.py           # FastAPI web interface
-├── main.py              # Enhanced CLI with multiple modes
-└── __init__.py          # Package initialization
-
-# Root files
-├── run.py                # Simple startup script
-├── test_integration.py   # Integration test suite
-├── test_api.py          # API endpoint tests
-└── ENHANCEMENT_SUMMARY.md # This file
-```
-
-## 🔧 **Enhanced Existing Files**
-
-- **`src/water_scraper_v3.py`** - Integrated new features, metrics, validation
-- **`src/config.py`** - Added configuration validation
-- **`requirements.txt`** - Added FastAPI, Pydantic, and monitoring dependencies
-- **`docker-compose.victoriametrics.yml`** - Added web API service
-- **`Dockerfile`** - Updated for new startup script
-- **`README.md`** - Updated with new features and usage instructions
-
-## 🌐 **Web API Endpoints**
-
-| Endpoint | Method | Description |
-|----------|--------|-------------|
-| `/` | GET | Interactive dashboard |
-| `/docs` | GET | API documentation |
-| `/health` | GET | System health status |
-| `/metrics` | GET | Application metrics |
-| `/stations` | GET | List all monitoring stations |
-| `/measurements/latest` | GET | Latest measurements |
-| `/measurements/station/{code}` | GET | Station-specific data |
-| `/scrape/trigger` | POST | Trigger manual data collection |
-| `/scraping/status` | GET | Scraping status and statistics |
-| `/config` | GET | Current configuration (masked) |
-
-## 🚀 **Usage Examples**
-
-### **Traditional Mode (Enhanced)**
-```bash
-# Test single cycle
-python run.py --test
-
-# Continuous monitoring
-python run.py
-
-# Fill data gaps
-python run.py --fill-gaps 7
-
-# Show system status
-python run.py --status
-```
-
-### **Web API Mode (NEW!)**
-```bash
-# Start web API server
-python run.py --web-api
-
-# Access dashboard
-open http://localhost:8000
-
-# View API documentation
-open http://localhost:8000/docs
-```
-
-### **Docker Deployment**
-```bash
-# Start complete stack
-docker-compose -f docker-compose.victoriametrics.yml up -d
-
-# Services available:
-# - Water API: http://localhost:8000
-# - Grafana: http://localhost:3000
-# - VictoriaMetrics: http://localhost:8428
-```
-
-## 📊 **Monitoring & Observability**
-
-### **Built-in Metrics**
-- API request counts and response times
-- Database connection status and save operations
-- Scraping cycle success/failure rates
-- System resource usage (memory, etc.)
-
-### **Health Checks**
-- Database connectivity and data freshness
-- External API availability
-- Memory usage monitoring
-- Overall system health status
-
-### **Enhanced Logging**
-- Colored console output for better readability
-- File rotation to prevent disk space issues
-- Performance logging for optimization
-- Structured logging with proper levels
-
-## 🔒 **Production Ready Features**
-
-### **Security & Reliability**
-- Rate limiting to prevent API abuse
-- Input validation and sanitization
-- Graceful error handling and recovery
-- Configuration validation on startup
-
-### **Performance**
-- Efficient metrics collection with minimal overhead
-- Background task management
-- Connection pooling and resource management
-- Optimized database operations
-
-### **Scalability**
-- Modular architecture for easy extension
-- Async support for high concurrency
-- Configurable resource limits
-- Health checks for load balancer integration
-
-## 🧪 **Testing**
-
-### **Integration Tests**
-```bash
-# Run all integration tests
-python test_integration.py
-```
-
-### **API Tests**
-```bash
-# Test API endpoints (server must be running)
-python test_api.py
-```
-
-## 📈 **Performance Improvements**
-
-1. **Request Tracking** - Monitor API performance and success rates
-2. **Rate Limiting** - Prevent API abuse and ensure stability
-3. **Data Validation** - Catch errors early and improve data quality
-4. **Metrics Collection** - Identify bottlenecks and optimization opportunities
-5. **Health Monitoring** - Proactive issue detection and alerting
-
-## 🎉 **Benefits Achieved**
-
-### **For Developers**
-- **Better Developer Experience** with type hints and validation
-- **Easier Debugging** with enhanced logging and error messages
-- **Comprehensive Testing** with integration and API tests
-- **Modern Architecture** following best practices
-
-### **For Operations**
-- **Web Dashboard** for easy monitoring and management
-- **Health Checks** for automated monitoring integration
-- **Metrics Collection** for performance analysis
-- **Production-Ready** deployment with Docker support
-
-### **For Users**
-- **REST API** for integration with other systems
-- **Real-time Data Access** via web interface
-- **Manual Controls** for triggering data collection
-- **Status Monitoring** for system visibility
-
-## 🔮 **Future Enhancement Opportunities**
-
-1. **Authentication & Authorization** - Add user management and API keys
-2. **Real-time WebSocket Updates** - Live data streaming to web clients
-3. **Advanced Analytics** - Trend analysis and forecasting
-4. **Alert System** - Email/SMS notifications for critical conditions
-5. **Multi-tenant Support** - Support for multiple organizations
-6. **Data Export** - CSV, Excel, and other format exports
-7. **Mobile App** - React Native or Flutter mobile interface
-
-## 🏆 **Summary**
-
-Your Thailand Water Monitor has been transformed from a simple data scraper into a **comprehensive, enterprise-grade monitoring platform** that includes:
-
-- ✅ **Modern Web Interface** with FastAPI
-- ✅ **Production-Ready Architecture** with proper error handling
-- ✅ **Comprehensive Monitoring** with metrics and health checks
-- ✅ **Type Safety** and data validation
-- ✅ **Enhanced Logging** and observability
-- ✅ **Docker Support** for easy deployment
-- ✅ **Extensive Testing** for reliability
-
-The system is now ready for production deployment and can serve as a foundation for further enhancements and integrations!

docs/FLOOD_FORECASTING.md

@@ -68,7 +68,7 @@ environment variable, then `Config.get_database_config()` when `DB_TYPE` is
 1. **PostgreSQL** (`_fetch_from_db`) — the primary path. NULL discharge stays
    NULL, which matters because the models must learn from the real missingness
    pattern.
-2. **HTTP API** (`_fetch_from_api`, default `http://100.81.167.42:8000`) — a
+2. **HTTP API** (`_fetch_from_api`, default `https://water.buildfor.life`) — a
    fallback for running off-server. **Caveat:** the public history endpoint
    backfills missing discharge with a synthetic rating-curve estimate, so this
    path is not equivalent to the DB path. It is flagged as
@@ -536,6 +536,54 @@ exists alongside its flood events.
 (`fill_from_hii`, +9,341 h at P.81, +682 h at P.92, +810 h at P.20) is
 lead-neutral — the gate holds at 13 h with fill on — and ships enabled.
 
+### 2026-09-12: three candidates on top of hgb-v3 — two rejected, one deferred
+
+Same rolling-origin harness (`src/ml/evaluate.py`, five monsoon folds
+2021–2025, P.1 and P.103), all variants run from the identical
+`models/cache/` snapshot (`--from-cache`), results in
+`models/eval_2026-09-12*.json`, tables via `scripts/summarize_eval.py`.
+Baseline is `rise_rain`, the deployed configuration.
+
+**Quantile regression heads (`rise_rain_quantile`, `_uw`) — rejected.** The
+August result that quantile loss beat L2 on MAE held with rain in the model
+(P.1 0.083/0.081 vs 0.087; P.103 0.143/0.124 vs 0.152), and Brier improved a
+hair, but the operational numbers went the wrong way: at P.103 the 2022-08-14
+crossing dropped from +6 h to +1 h lead, 2022-10-02 from +9 h to +5/+3 h, and
+the 2024-09-30 event from +9 h to +4 h; at P.1 2022 dropped +5 → +3/+2 h and
+2025 +2 → +1 h, with one false-alarm episode where the baseline had none. A
+median predicts the *typical* rise, and on the run-up to a crossing the typical
+rise is not the one that matters. MAE is not the objective; lead is.
+
+**Quantile heads for sigma only (`rise_rain_qsigma`) — no effect.** The
+hybrid keeps the L2 point prediction (so every lead is identical to the
+baseline by construction — p≥0.5 alerts are sigma-independent) and derives a
+per-row sigma from q90−q50. Brier moved 0.0031 → 0.0029 at P.1 and
+0.0061 → 0.0060 at P.103, i.e. within noise, at the cost of three fitted
+heads per horizon instead of one. Per-row uncertainty from this family of
+models is not informative enough here to be worth the training time; the
+0.15 m floor stays.
+
+**Forward-48 h forecast rain (`rise_rain_fc48`) — deferred.** Adding the
+`(t, t+48]` Open-Meteo sum alongside `rain_fc24` left MAE, Brier and false
+alarms unchanged and every event lead within ±1 h of baseline, *except* the
+2024-10-03 P.1 record crossing, which went from +21 h to +72 h (and +55 → +69 h
+at P.103). That is one event with the highest stakes in the record, on the
+same feature family that already produced the 2024 gain, but n=1 is not
+evidence: the P.103 2025-09-26 event lost 2 h in the same run. Rerun after the
+2026 season adds events; if the 48 h window still moves only the biggest
+onsets, promote it. Serving would need no new data source (`fetch_forecast`
+already pulls `forecast_days=2`).
+
+**HII gauge rain — not evaluable yet.** `hii_rainfall` (~130 gauges in the
+upper-Ping box, DWR/FOP/HII/RID/TMD) is the obvious independent rain source,
+but the table only exists since 2026-08-11 and the api-v3 archive endpoint
+ignores its date range (see `docs/DATA_SOURCES.md` §2.1), so every training
+row before that is NaN and no fold in the harness has gauge data in its test
+span. `src/ml/hii_rain.py` builds the catchment mean and
+`GET /api/hii/rainfall/catchment` exposes it next to the Open-Meteo series with
+a 24 h-sum bias/MAE/correlation, so the two sources' relationship is on record
+by the time the 2027 fold (train ≤ 2027-04-30, test Jun–Nov 2027) can test it.
+
 ## 6. Deployment
 
 ### API
@@ -733,6 +781,32 @@ timestamp in every bundle. Both are echoed in every `/forecast` row, so you can
 tell from the API response alone which code produced a forecast and how old the
 model is.
 
+**Scheduled retrain (since 2026-09-12).** `scripts/water-monitor-retrain.timer`
+fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
+time (`Persistent=true`, so a missed run catches up at boot). The unit runs
+`scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
+
+1. trains all stations into `models/.staging/` (the API keeps serving the old
+   bundles throughout);
+2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
+   least 14 trained stations (exit 3, staging discarded, old models untouched);
+3. renames the new bundles into `models/`, moving the previous generation to
+   `models/.previous/` for rollback.
+
+No API restart: `predict.py` reloads bundles by mtime on the next hourly
+precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
+run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
+flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
+(`scripts/install.sh`) enables the timer.
+
+**Why the trainer refuses to run without rain (since 2026-09-12).** On
+2026-09-01 the server retrain could not reach the Open-Meteo archive on a
+checkout with no `models/cache/`, logged a warning, and quietly overwrote the
+v3 bundles with gauge-only v2 ones — the 13-hour early warning on the 2024 flood
+became an 18-hour late one and nothing on the dashboard said so. `train_all()`
+now raises `RainUnavailableError` (CLI exit 2) in that situation. Gauge-only
+bundles are still available, but only by asking for them: `--no-rain`.
+
 ## 8. Operations runbook
 
 All commands assume the project virtualenv is active (`.venv` locally).
@@ -786,10 +860,12 @@ that went quiet, or a bad backfill) rather than a modelling one.
 python -m pytest tests/test_flood_forecast.py -v
 ```
 
-Seven tests covering leakage, label alignment, the coverage gate, forward-fill and
-staleness, a train/predict round trip, the heuristic fallback, and feature-name
-stability. The whole suite runs in about 8 seconds, so there is no excuse for
-skipping it before a deploy.
+Tests cover leakage, label alignment, the coverage gate, forward-fill and
+staleness, a train/predict round trip, the heuristic fallback, feature-name
+stability, and the rain-downgrade guard (no rain series → `RainUnavailableError`,
+nothing written; `--no-rain` → v2; rain present → v3 with the rain columns in
+`feature_names`). The file runs in well under a minute, so there is no excuse
+for skipping it before a deploy.
 
 **Understanding graceful degradation.** Three things can make a forecast row
 non-model-backed, and all of them are visible in the payload:

docs/GEOLOCATION_GUIDE.md

@@ -1,475 +0,0 @@
-# Geolocation Support for Grafana Geomap
-
-This guide explains the geolocation functionality added to the Thailand Water Monitor for use with Grafana's geomap visualization.
-
-## ✅ **Implemented Features**
-
-### **Database Schema Updates**
-All database adapters now support geolocation fields:
-- **latitude**: Decimal latitude coordinates (DECIMAL(10,8) for SQL, REAL for SQLite)
-- **longitude**: Decimal longitude coordinates (DECIMAL(11,8) for SQL, REAL for SQLite)  
-- **geohash**: Geohash string for efficient spatial indexing (VARCHAR(20)/TEXT)
-
-### **Station Data Enhancement**
-Station mapping now includes geolocation fields:
-```python
-'8': {
-    'code': 'P.1', 
-    'thai_name': 'สะพานนวรัฐ', 
-    'english_name': 'Nawarat Bridge',
-    'latitude': 15.6944,    # Decimal degrees
-    'longitude': 100.2028,  # Decimal degrees
-    'geohash': 'w5q6uuhvfcfp25'  # Geohash for P.1
-}
-```
-
-## 🗄️ **Database Schema**
-
-### **Updated Stations Table**
-```sql
-CREATE TABLE stations (
-    id INTEGER PRIMARY KEY,
-    station_code TEXT UNIQUE NOT NULL,
-    thai_name TEXT NOT NULL,
-    english_name TEXT NOT NULL,
-    latitude REAL,           -- NEW: Latitude coordinate
-    longitude REAL,          -- NEW: Longitude coordinate
-    geohash TEXT,           -- NEW: Geohash for spatial indexing
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-);
-```
-
-### **Database Support**
-- ✅ **SQLite**: REAL columns for coordinates, TEXT for geohash
-- ✅ **PostgreSQL**: DECIMAL(10,8) and DECIMAL(11,8) for coordinates, VARCHAR(20) for geohash
-- ✅ **MySQL**: DECIMAL(10,8) and DECIMAL(11,8) for coordinates, VARCHAR(20) for geohash
-- ✅ **VictoriaMetrics**: Geolocation data included in metric labels
-
-## 📊 **Current Station Data**
-
-### **P.1 - Nawarat Bridge (Sample)**
-- **Station Code**: P.1
-- **Thai Name**: สะพานนวรัฐ
-- **English Name**: Nawarat Bridge
-- **Latitude**: 15.6944
-- **Longitude**: 100.2028
-- **Geohash**: w5q6uuhvfcfp25
-
-### **Remaining Stations**
-The following stations are ready for geolocation data when coordinates become available:
-- P.20 - บ้านเชียงดาว (Ban Chiang Dao)
-- P.75 - บ้านช่อแล (Ban Chai Lat)
-- P.92 - บ้านเมืองกึ๊ด (Ban Muang Aut)
-- P.4A - บ้านแม่แตง (Ban Mae Taeng)
-- P.67 - บ้านแม่แต (Ban Tae)
-- P.21 - บ้านริมใต้ (Ban Rim Tai)
-- P.103 - สะพานวงแหวนรอบ 3 (Ring Bridge 3)
-- P.82 - บ้านสบวิน (Ban Sob win)
-- P.84 - บ้านพันตน (Ban Panton)
-- P.81 - บ้านโป่ง (Ban Pong)
-- P.5 - สะพานท่านาง (Tha Nang Bridge)
-- P.77 - บ้านสบแม่สะป๊วด (Baan Sop Mae Sapuord)
-- P.87 - บ้านป่าซาง (Ban Pa Sang)
-- P.76 - บ้านแม่อีไฮ (Banb Mae I Hai)
-- P.85 - บ้านหล่ายแก้ว (Baan Lai Kaew)
-
-## 🗺️ **Grafana Geomap Integration**
-
-### **Data Source Configuration**
-The geolocation data is automatically included in all database queries and can be used directly in Grafana:
-
-#### **SQLite/PostgreSQL/MySQL Query Example**
-```sql
-SELECT 
-    m.timestamp,
-    s.station_code,
-    s.english_name,
-    s.thai_name,
-    s.latitude,
-    s.longitude,
-    s.geohash,
-    m.water_level,
-    m.discharge,
-    m.discharge_percent
-FROM water_measurements m
-JOIN stations s ON m.station_id = s.id
-WHERE s.latitude IS NOT NULL 
-  AND s.longitude IS NOT NULL
-ORDER BY m.timestamp DESC
-```
-
-#### **VictoriaMetrics Query Example**
-```promql
-water_level{latitude!="",longitude!=""}
-```
-
-### **Geomap Panel Configuration**
-
-#### **1. Create Geomap Panel**
-1. Add new panel in Grafana
-2. Select "Geomap" visualization
-3. Configure data source (SQLite/PostgreSQL/MySQL/VictoriaMetrics)
-
-#### **2. Configure Location Fields**
-- **Latitude Field**: `latitude`
-- **Longitude Field**: `longitude`
-- **Alternative**: Use `geohash` field for geohash-based positioning
-
-#### **3. Configure Display Options**
-- **Station Labels**: Use `station_code` or `english_name`
-- **Tooltip Information**: Include `thai_name`, `water_level`, `discharge`
-- **Color Mapping**: Map to `water_level` or `discharge_percent`
-
-#### **4. Sample Geomap Configuration**
-```json
-{
-  "type": "geomap",
-  "title": "Thailand Water Stations",
-  "targets": [
-    {
-      "rawSql": "SELECT latitude, longitude, station_code, english_name, water_level, discharge_percent FROM stations s JOIN water_measurements m ON s.id = m.station_id WHERE s.latitude IS NOT NULL AND m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id)",
-      "format": "table"
-    }
-  ],
-  "fieldConfig": {
-    "defaults": {
-      "custom": {
-        "hideFrom": {
-          "legend": false,
-          "tooltip": false,
-          "vis": false
-        }
-      },
-      "mappings": [],
-      "color": {
-        "mode": "continuous-GrYlRd",
-        "field": "water_level"
-      }
-    }
-  },
-  "options": {
-    "view": {
-      "id": "coords",
-      "lat": 15.6944,
-      "lon": 100.2028,
-      "zoom": 8
-    },
-    "controls": {
-      "mouseWheelZoom": true,
-      "showZoom": true,
-      "showAttribution": true
-    },
-    "layers": [
-      {
-        "type": "markers",
-        "config": {
-          "size": {
-            "field": "discharge_percent",
-            "min": 5,
-            "max": 20
-          },
-          "color": {
-            "field": "water_level"
-          },
-          "showLegend": true
-        }
-      }
-    ]
-  }
-}
-```
-
-## 🔧 **Adding New Station Coordinates**
-
-### **Method 1: Update Station Mapping**
-Edit `water_scraper_v3.py` and add coordinates to the station mapping:
-```python
-'1': {
-    'code': 'P.20', 
-    'thai_name': 'บ้านเชียงดาว', 
-    'english_name': 'Ban Chiang Dao',
-    'latitude': 19.3056,    # Add actual coordinates
-    'longitude': 98.9264,   # Add actual coordinates
-    'geohash': 'w4r6...'    # Add actual geohash
-}
-```
-
-### **Method 2: Direct Database Update**
-```sql
-UPDATE stations 
-SET latitude = 19.3056, longitude = 98.9264, geohash = 'w4r6uuhvfcfp25'
-WHERE station_code = 'P.20';
-```
-
-### **Method 3: Bulk Update Script**
-```python
-import sqlite3
-
-coordinates = {
-    'P.20': {'lat': 19.3056, 'lon': 98.9264, 'geohash': 'w4r6uuhvfcfp25'},
-    'P.75': {'lat': 18.7756, 'lon': 99.1234, 'geohash': 'w4r5uuhvfcfp25'},
-    # Add more stations...
-}
-
-conn = sqlite3.connect('water_monitoring.db')
-cursor = conn.cursor()
-
-for station_code, coords in coordinates.items():
-    cursor.execute("""
-        UPDATE stations 
-        SET latitude = ?, longitude = ?, geohash = ?
-        WHERE station_code = ?
-    """, (coords['lat'], coords['lon'], coords['geohash'], station_code))
-
-conn.commit()
-conn.close()
-```
-
-## 🌐 **Geohash Information**
-
-### **What is Geohash?**
-Geohash is a geocoding system that represents geographic coordinates as a short alphanumeric string. It provides:
-- **Spatial Indexing**: Efficient spatial queries
-- **Proximity**: Similar geohashes indicate nearby locations
-- **Hierarchical**: Longer geohashes provide more precision
-
-### **Geohash Precision Levels**
-- **5 characters**: ~2.4km precision
-- **6 characters**: ~610m precision  
-- **7 characters**: ~76m precision
-- **8 characters**: ~19m precision
-- **9+ characters**: <5m precision
-
-### **Example: P.1 Geohash**
-- **Geohash**: `w5q6uuhvfcfp25`
-- **Length**: 14 characters
-- **Precision**: Sub-meter accuracy
-- **Location**: Nawarat Bridge, Thailand
-
-## 📈 **Grafana Visualization Examples**
-
-### **1. Station Location Map**
-- **Type**: Geomap with markers
-- **Data**: Current station locations
-- **Color**: Water level or discharge percentage
-- **Size**: Discharge volume
-
-### **2. Regional Water Levels**
-- **Type**: Geomap with heatmap
-- **Data**: Water level data across regions
-- **Visualization**: Color-coded intensity map
-- **Filters**: Time range, station groups
-
-### **3. Alert Zones**
-- **Type**: Geomap with threshold markers
-- **Data**: Stations exceeding alert thresholds
-- **Visualization**: Red markers for high water levels
-- **Alerts**: Automated notifications for critical levels
-
-## 🔄 **Updating a Running System**
-
-### **Automated Migration Script**
-Use the provided migration script to safely add geolocation columns to your existing database:
-
-```bash
-# Stop the water monitoring service first
-sudo systemctl stop water-monitor
-
-# Run the migration script
-python migrate_geolocation.py
-
-# Restart the service
-sudo systemctl start water-monitor
-```
-
-### **Migration Script Features**
-- ✅ **Auto-detects database type** from environment variables
-- ✅ **Checks existing columns** to avoid conflicts
-- ✅ **Supports all database types** (SQLite, PostgreSQL, MySQL)
-- ✅ **Adds sample data** for P.1 station
-- ✅ **Safe operation** - won't break existing data
-
-### **Step-by-Step Migration Process**
-
-#### **1. Stop the Application**
-```bash
-# If running as systemd service
-sudo systemctl stop water-monitor
-
-# If running in screen/tmux
-# Use Ctrl+C to stop the process
-
-# If running as Docker container
-docker stop water-monitor
-```
-
-#### **2. Backup Your Database**
-```bash
-# SQLite backup
-cp water_monitoring.db water_monitoring.db.backup
-
-# PostgreSQL backup
-pg_dump water_monitoring > water_monitoring_backup.sql
-
-# MySQL backup
-mysqldump water_monitoring > water_monitoring_backup.sql
-```
-
-#### **3. Run Migration Script**
-```bash
-# Default (uses environment variables)
-python migrate_geolocation.py
-
-# Or specify database path for SQLite
-SQLITE_DB_PATH=/path/to/water_monitoring.db python migrate_geolocation.py
-```
-
-#### **4. Verify Migration**
-```bash
-# Check SQLite schema
-sqlite3 water_monitoring.db ".schema stations"
-
-# Check PostgreSQL schema
-psql -d water_monitoring -c "\d stations"
-
-# Check MySQL schema
-mysql -e "DESCRIBE water_monitoring.stations"
-```
-
-#### **5. Update Application Code**
-Ensure you have the latest version of the application with geolocation support:
-```bash
-# Pull latest code
-git pull origin main
-
-# Install any new dependencies
-pip install -r requirements.txt
-```
-
-#### **6. Restart Application**
-```bash
-# Systemd service
-sudo systemctl start water-monitor
-
-# Docker container
-docker start water-monitor
-
-# Manual execution
-python water_scraper_v3.py
-```
-
-### **Migration Output Example**
-```
-2025-07-28 17:30:00,123 - INFO - Starting geolocation column migration...
-2025-07-28 17:30:00,124 - INFO - Detected database type: SQLITE
-2025-07-28 17:30:00,125 - INFO - Migrating SQLite database: water_monitoring.db
-2025-07-28 17:30:00,126 - INFO - Current columns in stations table: ['id', 'station_code', 'thai_name', 'english_name', 'created_at', 'updated_at']
-2025-07-28 17:30:00,127 - INFO - Added latitude column
-2025-07-28 17:30:00,128 - INFO - Added longitude column
-2025-07-28 17:30:00,129 - INFO - Added geohash column
-2025-07-28 17:30:00,130 - INFO - Successfully added columns: latitude, longitude, geohash
-2025-07-28 17:30:00,131 - INFO - Updated P.1 station with sample geolocation data
-2025-07-28 17:30:00,132 - INFO - P.1 station geolocation: ('P.1', 15.6944, 100.2028, 'w5q6uuhvfcfp25')
-2025-07-28 17:30:00,133 - INFO - ✅ Migration completed successfully!
-2025-07-28 17:30:00,134 - INFO - You can now restart your water monitoring application
-2025-07-28 17:30:00,135 - INFO - The system will automatically use the new geolocation columns
-```
-
-## 🔍 **Troubleshooting**
-
-### **Migration Issues**
-
-#### **Database Locked Error**
-```bash
-# Stop all processes using the database
-sudo systemctl stop water-monitor
-pkill -f water_scraper
-
-# Wait a few seconds, then run migration
-sleep 5
-python migrate_geolocation.py
-```
-
-#### **Permission Denied**
-```bash
-# Check database file permissions
-ls -la water_monitoring.db
-
-# Fix permissions if needed
-sudo chown $USER:$USER water_monitoring.db
-chmod 664 water_monitoring.db
-```
-
-#### **Missing Dependencies**
-```bash
-# For PostgreSQL
-pip install psycopg2-binary
-
-# For MySQL
-pip install pymysql
-
-# For all databases
-pip install -r requirements.txt
-```
-
-### **Verification Issues**
-
-#### **Missing Coordinates**
-If stations don't appear on the geomap:
-1. Check if latitude/longitude are NULL in database
-2. Verify geolocation data in station mapping
-3. Ensure database schema includes geolocation columns
-4. Run migration script if columns are missing
-
-#### **Incorrect Positioning**
-If stations appear in wrong locations:
-1. Verify coordinate format (decimal degrees)
-2. Check latitude/longitude order (lat first, lon second)
-3. Validate geohash accuracy
-
-### **Rollback Procedure**
-If migration causes issues:
-
-#### **SQLite Rollback**
-```bash
-# Stop application
-sudo systemctl stop water-monitor
-
-# Restore backup
-cp water_monitoring.db.backup water_monitoring.db
-
-# Restart with old version
-sudo systemctl start water-monitor
-```
-
-#### **PostgreSQL Rollback**
-```sql
--- Remove added columns
-ALTER TABLE stations DROP COLUMN IF EXISTS latitude;
-ALTER TABLE stations DROP COLUMN IF EXISTS longitude;
-ALTER TABLE stations DROP COLUMN IF EXISTS geohash;
-```
-
-#### **MySQL Rollback**
-```sql
--- Remove added columns
-ALTER TABLE stations DROP COLUMN latitude;
-ALTER TABLE stations DROP COLUMN longitude;
-ALTER TABLE stations DROP COLUMN geohash;
-```
-
-## 🎯 **Next Steps**
-
-### **Immediate Actions**
-1. **Gather Coordinates**: Collect GPS coordinates for all 16 stations
-2. **Update Database**: Add coordinates to remaining stations
-3. **Create Dashboards**: Build Grafana geomap visualizations
-
-### **Future Enhancements**
-1. **Automatic Geocoding**: API integration for address-to-coordinate conversion
-2. **Mobile GPS**: Mobile app for field coordinate collection
-3. **Satellite Integration**: Satellite imagery overlay in Grafana
-4. **Geofencing**: Alert zones based on geographic boundaries
-
-The geolocation functionality is now fully implemented and ready for use with Grafana's geomap visualization. Station P.1 (Nawarat Bridge) serves as a working example with complete coordinate data.

docs/GITEA_WORKFLOWS.md

@@ -286,8 +286,6 @@ make validate-workflows
 
 ### **Project-Specific Resources**
 - [Contributing Guide](../CONTRIBUTING.md)
-- [Deployment Checklist](../DEPLOYMENT_CHECKLIST.md)
-- [Project Structure](PROJECT_STRUCTURE.md)
 
 ### **Monitoring and Alerts**
 - Workflow status badges in README

docs/GRAFANA_MATRIX_ALERTING.md

@@ -1,168 +0,0 @@
-# Grafana Matrix Alerting Setup
-
-## Overview
-Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.
-
-## Prerequisites
-- Grafana instance with your PostgreSQL data source
-- Matrix account and access token
-- Matrix room for alerts
-
-## Step 1: Configure Matrix Contact Point
-
-1. **In Grafana, go to Alerting → Contact Points**
-2. **Add new contact point:**
-   ```
-   Name: matrix-water-alerts
-   Integration: Webhook
-   URL: https://matrix.org/_matrix/client/v3/rooms/!ROOM_ID:matrix.org/send/m.room.message
-   HTTP Method: POST
-   ```
-
-3. **Add Headers:**
-   ```
-   Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
-   Content-Type: application/json
-   ```
-
-4. **Message Template:**
-   ```json
-   {
-     "msgtype": "m.text",
-     "body": "🌊 WATER ALERT: {{ .CommonLabels.alertname }}\n\nStation: {{ .CommonLabels.station_code }}\nLevel: {{ .CommonAnnotations.water_level }}m\nStatus: {{ .CommonLabels.severity }}\n\nTime: {{ .CommonAnnotations.time }}"
-   }
-   ```
-
-## Step 2: Create Alert Rules
-
-### High Water Level Alert
-```yaml
-Rule Name: high-water-level
-Query: water_level > 6.0
-Condition: IS ABOVE 6.0 FOR 5m
-Labels:
-  - severity: critical
-  - station_code: {{ .station_code }}
-Annotations:
-  - water_level: {{ .water_level }}
-  - summary: "Critical water level at {{ .station_code }}"
-```
-
-### Low Water Level Alert
-```yaml
-Rule Name: low-water-level
-Query: water_level < 1.0
-Condition: IS BELOW 1.0 FOR 10m
-Labels:
-  - severity: warning
-  - station_code: {{ .station_code }}
-```
-
-### Data Gap Alert
-```yaml
-Rule Name: data-gap
-Query: increase(measurements_total[1h]) == 0
-Condition: IS EQUAL TO 0 FOR 30m
-Labels:
-  - severity: warning
-  - issue: data-gap
-```
-
-## Step 3: Matrix Setup
-
-### Get Matrix Access Token
-```bash
-curl -X POST https://matrix.org/_matrix/client/v3/login \
-  -H "Content-Type: application/json" \
-  -d '{
-    "type": "m.login.password",
-    "user": "your_username",
-    "password": "your_password"
-  }'
-```
-
-### Create Alert Room
-```bash
-curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
-  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "name": "Water Level Alerts - Northern Thailand",
-    "topic": "Automated alerts for Ping River water monitoring",
-    "preset": "trusted_private_chat"
-  }'
-```
-
-## Example Alert Queries
-
-### Critical Water Levels
-```promql
-# High water alert
-water_level{station_code=~"P.1|P.4A|P.20"} > 6.0
-
-# Dangerous discharge
-discharge{station_code=~".*"} > 500
-
-# Rapid level change
-increase(water_level[15m]) > 0.5
-```
-
-### System Health
-```promql
-# No data received
-up{job="water-monitor"} == 0
-
-# Old data
-(time() - timestamp) > 7200
-```
-
-## Alert Notification Format
-
-Your Matrix messages will look like:
-```
-🌊 WATER ALERT: High Water Level
-
-Station: P.1 (Chiang Mai)
-Level: 6.2m (CRITICAL)
-Discharge: 450 cms
-Status: DANGER
-
-Time: 2025-09-26 14:30:00
-Trend: Rising (+0.3m in 30min)
-
-📍 Location: 18.7883°N, 98.9853°E
-```
-
-## Advanced Features
-
-### Escalation Rules
-```yaml
-# Send to different rooms based on severity
-- if: severity == "critical"
-  receiver: matrix-emergency
-- if: severity == "warning"
-  receiver: matrix-alerts
-- if: time_of_day() outside "08:00-20:00"
-  receiver: matrix-night-duty
-```
-
-### Rate Limiting
-```yaml
-group_wait: 5m
-group_interval: 10m
-repeat_interval: 30m
-```
-
-## Testing Alerts
-
-1. **Test Contact Point** - Use Grafana's test button
-2. **Simulate Alert** - Manually trigger with test data
-3. **Verify Matrix** - Check message formatting and delivery
-
-## Troubleshooting
-
-### Common Issues
-- **403 Forbidden**: Check Matrix access token
-- **Room not found**: Verify room ID format
-- **No alerts**: Check query syntax and thresholds
-- **Spam**: Configure proper grouping and intervals

docs/GRAFANA_MATRIX_SETUP.md

@@ -1,351 +0,0 @@
-# Complete Grafana Matrix Alerting Setup Guide
-
-## Overview
-Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.
-
-## Prerequisites
-- Grafana instance running (v8.0+)
-- PostgreSQL data source configured in Grafana
-- Matrix account
-- Matrix room for alerts
-
-## Step 1: Get Matrix Access Token
-
-### Method 1: Using curl
-```bash
-curl -X POST https://matrix.org/_matrix/client/v3/login \
-  -H "Content-Type: application/json" \
-  -d '{
-    "type": "m.login.password",
-    "user": "your_username",
-    "password": "your_password"
-  }'
-```
-
-### Method 2: Using Element Web Client
-1. Open Element in browser: https://app.element.io
-2. Login to your account
-3. Go to Settings → Help & About → Advanced
-4. Copy your Access Token
-
-### Method 3: Using Matrix Admin Panel
-- If you have admin access to your homeserver, generate token via admin API
-
-## Step 2: Create Alert Room
-
-```bash
-curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
-  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "name": "Water Level Alerts - Northern Thailand",
-    "topic": "Automated alerts for Ping River water monitoring",
-    "preset": "private_chat"
-  }'
-```
-
-Save the `room_id` from the response (format: !roomid:homeserver.com)
-
-## Step 3: Configure Grafana Contact Point
-
-### Navigate to Alerting
-1. In Grafana, go to **Alerting → Contact Points**
-2. Click **Add contact point**
-
-### Contact Point Settings
-```
-Name: matrix-water-alerts
-Integration: Webhook
-URL: https://matrix.org/_matrix/client/v3/rooms/!YOUR_ROOM_ID:matrix.org/send/m.room.message/{{ .GroupLabels.alertname }}_{{ .GroupLabels.severity }}_{{ now.Unix }}
-HTTP Method: POST
-```
-
-### Headers
-```
-Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
-Content-Type: application/json
-```
-
-### Message Template (JSON Body)
-```json
-{
-  "msgtype": "m.text",
-  "body": "🌊 **PING RIVER WATER ALERT**\n\n**Alert:** {{ .GroupLabels.alertname }}\n**Severity:** {{ .GroupLabels.severity | toUpper }}\n**Station:** {{ .GroupLabels.station_code }} ({{ .GroupLabels.station_name }})\n\n{{ range .Alerts }}**Status:** {{ .Status | toUpper }}\n**Water Level:** {{ .Annotations.water_level }}m\n**Threshold:** {{ .Annotations.threshold }}m\n**Time:** {{ .StartsAt.Format \"2006-01-02 15:04:05\" }}\n{{ if .Annotations.discharge }}**Discharge:** {{ .Annotations.discharge }} cms\n{{ end }}{{ if .Annotations.message }}**Details:** {{ .Annotations.message }}\n{{ end }}{{ end }}\n📈 **Dashboard:** {{ .ExternalURL }}\n📍 **Location:** Northern Thailand Ping River"
-}
-```
-
-## Step 4: Create Alert Rules
-
-### High Water Level Alert
-```yaml
-# Rule Configuration
-Rule Name: high-water-level
-Evaluation Group: water-level-alerts
-Folder: Water Monitoring
-
-# Query A
-SELECT
-  station_code,
-  station_name_th as station_name,
-  water_level,
-  discharge,
-  timestamp
-FROM water_measurements
-WHERE
-  timestamp > now() - interval '5 minutes'
-  AND water_level > 6.0
-
-# Condition
-IS ABOVE 6.0 FOR 5 minutes
-
-# Labels
-severity: critical
-alertname: High Water Level
-station_code: {{ $labels.station_code }}
-station_name: {{ $labels.station_name }}
-
-# Annotations
-water_level: {{ $values.water_level }}
-threshold: 6.0
-discharge: {{ $values.discharge }}
-summary: Critical water level detected at {{ $labels.station_code }}
-```
-
-### Emergency Water Level Alert
-```yaml
-Rule Name: emergency-water-level
-Query: water_level > 8.0
-Condition: IS ABOVE 8.0 FOR 2 minutes
-Labels:
-  severity: emergency
-  alertname: Emergency Water Level
-Annotations:
-  threshold: 8.0
-  message: IMMEDIATE ACTION REQUIRED - Flood risk imminent
-```
-
-### Low Water Level Alert
-```yaml
-Rule Name: low-water-level
-Query: water_level < 1.0
-Condition: IS BELOW 1.0 FOR 15 minutes
-Labels:
-  severity: warning
-  alertname: Low Water Level
-Annotations:
-  threshold: 1.0
-  message: Drought conditions detected
-```
-
-### Data Gap Alert
-```yaml
-Rule Name: data-gap
-Query:
-  SELECT
-    station_code,
-    MAX(timestamp) as last_seen
-  FROM water_measurements
-  GROUP BY station_code
-  HAVING MAX(timestamp) < now() - interval '2 hours'
-
-Condition: HAS NO DATA FOR 30 minutes
-Labels:
-  severity: warning
-  alertname: Data Gap
-  issue: missing-data
-```
-
-### Rapid Level Change Alert
-```yaml
-Rule Name: rapid-level-change
-Query:
-  SELECT
-    station_code,
-    water_level,
-    LAG(water_level, 1) OVER (PARTITION BY station_code ORDER BY timestamp) as prev_level
-  FROM water_measurements
-  WHERE timestamp > now() - interval '15 minutes'
-  HAVING ABS(water_level - prev_level) > 0.5
-
-Condition: CHANGE > 0.5m FOR 1 minute
-Labels:
-  severity: warning
-  alertname: Rapid Water Level Change
-```
-
-## Step 5: Configure Notification Policy
-
-### Create Notification Policy
-```yaml
-# Policy Tree
-- receiver: matrix-water-alerts
-  match:
-    severity: emergency|critical
-  group_wait: 10s
-  group_interval: 5m
-  repeat_interval: 30m
-
-- receiver: matrix-water-alerts
-  match:
-    severity: warning
-  group_wait: 30s
-  group_interval: 10m
-  repeat_interval: 2h
-```
-
-### Grouping Rules
-```yaml
-group_by: [alertname, station_code]
-group_wait: 10s
-group_interval: 5m
-repeat_interval: 1h
-```
-
-## Step 6: Station-Specific Thresholds
-
-Create separate rules for each station with appropriate thresholds:
-
-```sql
--- P.1 (Chiang Mai) - Urban area, higher thresholds
-SELECT * FROM water_measurements
-WHERE station_code = 'P.1' AND water_level > 6.5
-
--- P.4A (Mae Ping) - Agricultural area
-SELECT * FROM water_measurements
-WHERE station_code = 'P.4A' AND water_level > 5.0
-
--- P.20 (Downstream) - Lower threshold
-SELECT * FROM water_measurements
-WHERE station_code = 'P.20' AND water_level > 4.0
-```
-
-## Step 7: Advanced Features
-
-### Time-Based Routing
-```yaml
-# Different receivers for day/night
-time_intervals:
-  - name: working_hours
-    time_intervals:
-      - times:
-        - start_time: '08:00'
-          end_time: '20:00'
-        weekdays: ['monday:friday']
-
-routes:
-  - receiver: matrix-alerts-day
-    match:
-      severity: warning
-    active_time_intervals: [working_hours]
-
-  - receiver: matrix-alerts-night
-    match:
-      severity: warning
-    active_time_intervals: ['!working_hours']
-```
-
-### Multi-Channel Alerts
-```yaml
-# Send critical alerts to multiple rooms
-- receiver: matrix-emergency
-  webhook_configs:
-    - url: https://matrix.org/_matrix/client/v3/rooms/!emergency:matrix.org/send/m.room.message
-      http_config:
-        authorization:
-          credentials: "Bearer EMERGENCY_TOKEN"
-    - url: https://matrix.org/_matrix/client/v3/rooms/!general:matrix.org/send/m.room.message
-      http_config:
-        authorization:
-          credentials: "Bearer GENERAL_TOKEN"
-```
-
-## Step 8: Testing
-
-### Test Contact Point
-1. Go to Contact Points in Grafana
-2. Select your Matrix contact point
-3. Click "Test" button
-4. Check Matrix room for test message
-
-### Test Alert Rules
-1. Temporarily lower thresholds
-2. Wait for condition to trigger
-3. Verify alert appears in Grafana
-4. Verify Matrix message received
-5. Reset thresholds
-
-### Manual Alert Trigger
-```bash
-# Simulate high water level in database
-INSERT INTO water_measurements (station_code, water_level, timestamp)
-VALUES ('P.1', 7.5, NOW());
-```
-
-## Troubleshooting
-
-### Common Issues
-
-#### 403 Forbidden
-- **Cause**: Invalid Matrix access token
-- **Fix**: Regenerate token or check permissions
-
-#### Room Not Found
-- **Cause**: Incorrect room ID format
-- **Fix**: Ensure room ID starts with ! and includes homeserver
-
-#### No Alerts Firing
-- **Cause**: Query returns no results
-- **Fix**: Test queries in Grafana Explore, check data availability
-
-#### Alert Spam
-- **Cause**: No grouping configured
-- **Fix**: Configure proper group_by and intervals
-
-#### Messages Not Formatted
-- **Cause**: Template syntax errors
-- **Fix**: Validate JSON template, check Grafana template docs
-
-### Debug Steps
-1. Check Grafana alert rule status
-2. Verify contact point test succeeds
-3. Check Grafana logs: `/var/log/grafana/grafana.log`
-4. Test Matrix API directly with curl
-5. Verify database connectivity and query results
-
-## Environment Variables
-
-Add to your `.env`:
-```bash
-MATRIX_HOMESERVER=https://matrix.org
-MATRIX_ACCESS_TOKEN=your_access_token_here
-MATRIX_ROOM_ID=!your_room_id:matrix.org
-GRAFANA_URL=http://your-grafana-host:3000
-```
-
-## Example Alert Message
-Your Matrix messages will appear as:
-```
-🌊 **PING RIVER WATER ALERT**
-
-**Alert:** High Water Level
-**Severity:** CRITICAL
-**Station:** P.1 (สถานีเชียงใหม่)
-
-**Status:** FIRING
-**Water Level:** 6.75m
-**Threshold:** 6.0m
-**Time:** 2025-09-26 14:30:00
-**Discharge:** 450.2 cms
-
-📈 **Dashboard:** http://grafana:3000
-📍 **Location:** Northern Thailand Ping River
-```
-
-## Security Notes
-- Store Matrix tokens securely (environment variables)
-- Use room-specific tokens when possible
-- Enable rate limiting to prevent spam
-- Consider using dedicated alerting user account
-- Regularly rotate access tokens
-
-This setup provides comprehensive water level monitoring with immediate Matrix notifications when thresholds are exceeded.

docs/HTTPS_CONFIGURATION.md

@@ -1,389 +0,0 @@
-# HTTPS VictoriaMetrics Configuration Guide
-
-This guide explains how to configure the Thailand Water Monitor to connect to VictoriaMetrics through HTTPS and reverse proxies.
-
-## Configuration Options
-
-### 1. Environment Variables for HTTPS
-
-```bash
-# Option 1: Full HTTPS URL (Recommended)
-export DB_TYPE=victoriametrics
-export VM_HOST=https://vm.example.com
-export VM_PORT=443
-
-# Option 2: Host and port separately
-export DB_TYPE=victoriametrics
-export VM_HOST=vm.example.com
-export VM_PORT=443
-
-# Option 3: Custom port with HTTPS
-export DB_TYPE=victoriametrics
-export VM_HOST=https://vm.example.com
-export VM_PORT=8443
-```
-
-### 2. Windows PowerShell Configuration
-
-```powershell
-# Set environment variables for HTTPS
-$env:DB_TYPE="victoriametrics"
-$env:VM_HOST="https://vm.example.com"
-$env:VM_PORT="443"
-
-# Run the water monitor
-python water_scraper_v3.py
-```
-
-### 3. Linux/Mac Configuration
-
-```bash
-# Set environment variables for HTTPS
-export DB_TYPE=victoriametrics
-export VM_HOST=https://vm.example.com
-export VM_PORT=443
-
-# Run the water monitor
-python water_scraper_v3.py
-```
-
-## Reverse Proxy Examples
-
-### 1. Nginx Reverse Proxy
-
-```nginx
-server {
-    listen 443 ssl http2;
-    server_name vm.example.com;
-    
-    # SSL Configuration
-    ssl_certificate /path/to/certificate.crt;
-    ssl_certificate_key /path/to/private.key;
-    ssl_protocols TLSv1.2 TLSv1.3;
-    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
-    
-    # Security headers
-    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
-    add_header X-Frame-Options DENY always;
-    add_header X-Content-Type-Options nosniff always;
-    
-    # Optional: Basic authentication
-    # auth_basic "VictoriaMetrics";
-    # auth_basic_user_file /etc/nginx/.htpasswd;
-    
-    location / {
-        proxy_pass http://localhost:8428;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        
-        # WebSocket support (if needed)
-        proxy_http_version 1.1;
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-        
-        # Timeouts
-        proxy_connect_timeout 60s;
-        proxy_send_timeout 60s;
-        proxy_read_timeout 60s;
-    }
-}
-
-# Redirect HTTP to HTTPS
-server {
-    listen 80;
-    server_name vm.example.com;
-    return 301 https://$server_name$request_uri;
-}
-```
-
-### 2. Apache Reverse Proxy
-
-```apache
-<VirtualHost *:443>
-    ServerName vm.example.com
-    
-    # SSL Configuration
-    SSLEngine on
-    SSLCertificateFile /path/to/certificate.crt
-    SSLCertificateKeyFile /path/to/private.key
-    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
-    SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
-    
-    # Security headers
-    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
-    Header always set X-Frame-Options DENY
-    Header always set X-Content-Type-Options nosniff
-    
-    # Reverse proxy configuration
-    ProxyPreserveHost On
-    ProxyPass / http://localhost:8428/
-    ProxyPassReverse / http://localhost:8428/
-    
-    # Optional: Basic authentication
-    # AuthType Basic
-    # AuthName "VictoriaMetrics"
-    # AuthUserFile /etc/apache2/.htpasswd
-    # Require valid-user
-</VirtualHost>
-
-<VirtualHost *:80>
-    ServerName vm.example.com
-    Redirect permanent / https://vm.example.com/
-</VirtualHost>
-```
-
-### 3. Traefik Reverse Proxy
-
-```yaml
-# docker-compose.yml with Traefik
-version: '3.8'
-
-services:
-  traefik:
-    image: traefik:v2.10
-    command:
-      - --api.dashboard=true
-      - --entrypoints.web.address=:80
-      - --entrypoints.websecure.address=:443
-      - --providers.docker=true
-      - --certificatesresolvers.letsencrypt.acme.tlschallenge=true
-      - --certificatesresolvers.letsencrypt.acme.email=admin@example.com
-      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
-    ports:
-      - "80:80"
-      - "443:443"
-    volumes:
-      - /var/run/docker.sock:/var/run/docker.sock
-      - letsencrypt:/letsencrypt
-    labels:
-      - traefik.http.routers.api.rule=Host(`traefik.example.com`)
-      - traefik.http.routers.api.tls.certresolver=letsencrypt
-
-  victoriametrics:
-    image: victoriametrics/victoria-metrics:latest
-    command:
-      - '--storageDataPath=/victoria-metrics-data'
-      - '--retentionPeriod=2y'
-      - '--httpListenAddr=:8428'
-    volumes:
-      - vm_data:/victoria-metrics-data
-    labels:
-      - traefik.enable=true
-      - traefik.http.routers.vm.rule=Host(`vm.example.com`)
-      - traefik.http.routers.vm.tls.certresolver=letsencrypt
-      - traefik.http.services.vm.loadbalancer.server.port=8428
-
-volumes:
-  vm_data:
-  letsencrypt:
-```
-
-## Testing HTTPS Configuration
-
-### 1. Test Connection
-
-```bash
-# Test HTTPS connection
-curl -k https://vm.example.com/health
-
-# Test with specific port
-curl -k https://vm.example.com:8443/health
-
-# Test API endpoint
-curl -k "https://vm.example.com/api/v1/query?query=up"
-```
-
-### 2. Test with Water Monitor
-
-```bash
-# Set environment variables
-export DB_TYPE=victoriametrics
-export VM_HOST=https://vm.example.com
-export VM_PORT=443
-
-# Test with demo script
-python demo_databases.py victoriametrics
-
-# Run full water monitor
-python water_scraper_v3.py
-```
-
-### 3. Verify SSL Certificate
-
-```bash
-# Check SSL certificate
-openssl s_client -connect vm.example.com:443 -servername vm.example.com
-
-# Check certificate expiration
-echo | openssl s_client -connect vm.example.com:443 2>/dev/null | openssl x509 -noout -dates
-```
-
-## Configuration Examples
-
-### 1. Production HTTPS Setup
-
-```bash
-# Environment variables for production
-export DB_TYPE=victoriametrics
-export VM_HOST=https://metrics.company.com
-export VM_PORT=443
-export LOG_LEVEL=INFO
-export SCRAPING_INTERVAL_HOURS=1
-
-# Run water monitor
-python water_scraper_v3.py
-```
-
-### 2. Development with Self-Signed Certificate
-
-```bash
-# For development with self-signed certificates
-export DB_TYPE=victoriametrics
-export VM_HOST=https://dev-vm.local
-export VM_PORT=443
-export PYTHONHTTPSVERIFY=0  # Disable SSL verification (dev only)
-
-python water_scraper_v3.py
-```
-
-### 3. Custom Port Configuration
-
-```bash
-# Custom HTTPS port
-export DB_TYPE=victoriametrics
-export VM_HOST=https://vm.example.com
-export VM_PORT=8443
-
-python water_scraper_v3.py
-```
-
-## Troubleshooting HTTPS Issues
-
-### 1. SSL Certificate Errors
-
-```bash
-# Error: SSL certificate verify failed
-# Solution: Check certificate validity
-openssl x509 -in certificate.crt -text -noout
-
-# Temporary workaround (not recommended for production)
-export PYTHONHTTPSVERIFY=0
-```
-
-### 2. Connection Timeout
-
-```bash
-# Error: Connection timeout
-# Check firewall and network connectivity
-telnet vm.example.com 443
-nc -zv vm.example.com 443
-```
-
-### 3. DNS Resolution Issues
-
-```bash
-# Error: Name resolution failed
-# Check DNS resolution
-nslookup vm.example.com
-dig vm.example.com
-```
-
-### 4. Proxy Configuration Issues
-
-```bash
-# Check proxy logs
-# Nginx
-tail -f /var/log/nginx/error.log
-
-# Apache
-tail -f /var/log/apache2/error.log
-
-# Test direct connection to backend
-curl http://localhost:8428/health
-```
-
-## Security Best Practices
-
-### 1. SSL/TLS Configuration
-
-- Use TLS 1.2 or higher
-- Disable weak ciphers
-- Enable HSTS headers
-- Use strong SSL certificates
-
-### 2. Authentication
-
-```nginx
-# Basic authentication in Nginx
-auth_basic "VictoriaMetrics Access";
-auth_basic_user_file /etc/nginx/.htpasswd;
-
-# Create password file
-htpasswd -c /etc/nginx/.htpasswd username
-```
-
-### 3. Network Security
-
-- Use firewall rules to restrict access
-- Consider VPN for internal access
-- Implement rate limiting
-- Monitor access logs
-
-### 4. Certificate Management
-
-```bash
-# Auto-renewal with Let's Encrypt
-certbot renew --dry-run
-
-# Certificate monitoring
-echo | openssl s_client -connect vm.example.com:443 2>/dev/null | \
-openssl x509 -noout -dates | grep notAfter
-```
-
-## Docker Configuration for HTTPS
-
-### 1. Docker Compose with HTTPS
-
-```yaml
-version: '3.8'
-
-services:
-  water-monitor:
-    build: .
-    environment:
-      - DB_TYPE=victoriametrics
-      - VM_HOST=https://vm.example.com
-      - VM_PORT=443
-    restart: unless-stopped
-    depends_on:
-      - victoriametrics
-
-  victoriametrics:
-    image: victoriametrics/victoria-metrics:latest
-    ports:
-      - "8428:8428"
-    volumes:
-      - vm_data:/victoria-metrics-data
-    command:
-      - '--storageDataPath=/victoria-metrics-data'
-      - '--retentionPeriod=2y'
-      - '--httpListenAddr=:8428'
-
-volumes:
-  vm_data:
-```
-
-### 2. Environment File (.env)
-
-```bash
-# .env file
-DB_TYPE=victoriametrics
-VM_HOST=https://vm.example.com
-VM_PORT=443
-LOG_LEVEL=INFO
-SCRAPING_INTERVAL_HOURS=1
-```
-
-This configuration guide provides comprehensive instructions for setting up HTTPS connectivity to VictoriaMetrics through reverse proxies, ensuring secure and reliable data transmission for the Thailand Water Monitor.

docs/MIGRATION_QUICKSTART.md

@@ -1,136 +0,0 @@
-# Geolocation Migration Quick Start
-
-This is a quick reference guide for updating a running Thailand Water Monitor system to add geolocation support for Grafana geomap.
-
-## 🚀 **Quick Migration (5 minutes)**
-
-### **Step 1: Stop Application**
-```bash
-# Stop the service (choose your method)
-sudo systemctl stop water-monitor
-# OR
-docker stop water-monitor
-# OR use Ctrl+C if running manually
-```
-
-### **Step 2: Backup Database**
-```bash
-# SQLite backup
-cp water_monitoring.db water_monitoring.db.backup
-
-# PostgreSQL backup
-pg_dump water_monitoring > backup.sql
-
-# MySQL backup
-mysqldump water_monitoring > backup.sql
-```
-
-### **Step 3: Run Migration**
-```bash
-# Run the automated migration script
-python migrate_geolocation.py
-```
-
-### **Step 4: Restart Application**
-```bash
-# Restart the service
-sudo systemctl start water-monitor
-# OR
-docker start water-monitor
-# OR
-python water_scraper_v3.py
-```
-
-## ✅ **Expected Output**
-```
-2025-07-28 17:30:00,123 - INFO - Starting geolocation column migration...
-2025-07-28 17:30:00,124 - INFO - Detected database type: SQLITE
-2025-07-28 17:30:00,127 - INFO - Added latitude column
-2025-07-28 17:30:00,128 - INFO - Added longitude column
-2025-07-28 17:30:00,129 - INFO - Added geohash column
-2025-07-28 17:30:00,133 - INFO - ✅ Migration completed successfully!
-```
-
-## 🗺️ **Verify Geolocation Works**
-
-### **Check Database**
-```bash
-# SQLite
-sqlite3 water_monitoring.db "SELECT station_code, latitude, longitude, geohash FROM stations WHERE station_code = 'P.1';"
-
-# Expected output: P.1|15.6944|100.2028|w5q6uuhvfcfp25
-```
-
-### **Test Application**
-```bash
-# Run a test cycle
-python water_scraper_v3.py --test
-
-# Should complete without errors
-```
-
-## 🔧 **Grafana Setup**
-
-### **Query for Geomap**
-```sql
-SELECT 
-    s.latitude, s.longitude, s.station_code, s.english_name,
-    m.water_level, m.discharge_percent
-FROM stations s 
-JOIN water_measurements m ON s.id = m.station_id
-WHERE s.latitude IS NOT NULL 
-  AND m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id)
-```
-
-### **Geomap Configuration**
-1. Create new panel → Select "Geomap"
-2. Set **Latitude field**: `latitude`
-3. Set **Longitude field**: `longitude`
-4. Set **Color field**: `water_level`
-5. Set **Size field**: `discharge_percent`
-
-## 🚨 **Troubleshooting**
-
-### **Database Locked**
-```bash
-sudo systemctl stop water-monitor
-pkill -f water_scraper
-sleep 5
-python migrate_geolocation.py
-```
-
-### **Permission Error**
-```bash
-sudo chown $USER:$USER water_monitoring.db
-chmod 664 water_monitoring.db
-```
-
-### **Missing Dependencies**
-```bash
-pip install psycopg2-binary pymysql
-```
-
-## 🔄 **Rollback (if needed)**
-```bash
-# Stop application
-sudo systemctl stop water-monitor
-
-# Restore backup
-cp water_monitoring.db.backup water_monitoring.db
-
-# Restart
-sudo systemctl start water-monitor
-```
-
-## 📚 **More Information**
-- **Full Guide**: See `GEOLOCATION_GUIDE.md`
-- **Migration Script**: `migrate_geolocation.py`
-- **Database Schema**: Updated with latitude, longitude, geohash columns
-
-## 🎯 **What You Get**
-- ✅ **P.1 Station** ready for geomap (Nawarat Bridge)
-- ✅ **Database Schema** updated for all 16 stations
-- ✅ **Grafana Compatible** data structure
-- ✅ **Backward Compatible** - existing data preserved
-
-**Total Time**: ~5 minutes for complete migration

docs/NOTIFICATIONS.md

@@ -0,0 +1,132 @@
+# Flood notifications (ntfy)
+
+Public push notifications for threshold crossings, without accounts, mailing
+lists or app-store review: the monitor publishes to a self-hosted
+[ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
+about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
+
+ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
+runs on the same VPS as the monitor.
+
+## What subscribers get
+
+Every message is a **transition**, never a state. Crossing up into a level sends
+one message; dropping back below it (with 0.10 m hysteresis) sends one
+all-clear. A river that sits at 3.9 m for three days produces two messages, not
+seventy-two. In a quiet season a subscriber hears nothing.
+
+| Topic | Trigger | Priority |
+|---|---|---|
+| `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
+| `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
+| `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
+| `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
+| `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
+| `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
+
+Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
+Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
+4.20 m, P.103 5.95 / 6.75 m, and so on.
+
+The outlook topic is opt-in for a reason: it is model output, and the message
+says so. Observed-crossing topics only ever report a gauge reading.
+
+Each message carries a click-through and an "Open dashboard" action button to
+the public dashboard.
+
+## How it runs
+
+`src/notify.py` is called once per collection cycle inside the API process
+(leader only), right after the forecast precompute, so it sees exactly the
+readings and forecasts the dashboard shows. Per-key last-sent state is stored
+in the `notification_state` table of the monitor's own database, so a restart
+or redeploy never re-sends and never misses a crossing that happened while
+the service was down (the next cycle compares against the persisted state).
+
+If ntfy is unreachable the transition is **not** recorded, so it is retried
+on the next cycle rather than silently lost. Any other failure in the notify
+step is logged and never reaches the collection loop.
+
+The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
+set; it reads `GET /api/notifications` and renders subscribe links
+(`ntfy://` deep links for the app, https links for the web UI).
+
+## Deployment
+
+On the monitor VPS, as root:
+
+```bash
+cd /opt/thailand-water-monitor
+NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
+```
+
+This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
+host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
+message cache, signup/login/metrics off, tight visitor limits), enables the
+systemd unit,
+creates the `monitor` user with **write-only access to `ping-*`**, mints a
+token, and appends `NTFY_SERVER` (public URL for subscribers),
+`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
+and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
+
+```bash
+systemctl restart water-monitor
+journalctl -u water-monitor -n 20 | grep ntfy      # "ntfy notifications: https://... topics ping-*"
+curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1'   # anonymous read works
+```
+
+The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
+the monitor host's Tailscale address and nothing is exposed on a public
+interface. On the Caddy machine:
+
+```caddyfile
+ntfy.buildfor.life {
+    reverse_proxy <monitor tailscale ip>:2586
+}
+```
+
+Caddy proxies websockets and keeps long-poll connections open by default;
+subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
+`X-Forwarded-For` rather than treating every subscriber as the proxy.
+
+Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
+monitor at the Tailscale address directly, so a DNS or proxy problem never
+holds back an alert. Test the pipeline before the domain is live with
+`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
+
+## Configuration
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
+| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
+| `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
+| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
+| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
+
+Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
+(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
+
+## Testing
+
+`tests/test_notify.py` covers the state machine: quiet river sends nothing;
+crossing once, then silence while above, then all-clear; hysteresis on the way
+down; escalation to danger and back; basin digest grouping; outlook on/off;
+heuristic forecasts ignored; stale feed and recovery; state survives a restart
+through sqlite; a failed publish is retried next cycle.
+
+To exercise the real path against a real ntfy locally: run `ntfy serve` (any
+platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
+poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
+template.
+
+## Why ntfy and not …
+
+- **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
+  subscriber and a room invite; fine for a team, wrong for the public.
+- **Gotify**: also self-hosted and light, but Android-only client and one
+  account per subscriber.
+- **Email / SMS**: deliverability work, cost per message, no priority
+  semantics; ntfy can forward to email per subscription if someone wants it.
+- **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
+  added later as ntfy→webhook fan-out without touching the monitor.

docs/PROJECT_STATUS.md

@@ -1,206 +0,0 @@
-# Thailand Water Monitor - Current Project Status
-
-## 📁 **Clean Project Structure**
-
-The project has been cleaned up and organized with the following structure:
-
-```
-water_level_monitor/
-├── 📄 .gitignore                             # Git ignore rules
-├── 📄 README.md                              # Main project documentation
-├── 📄 requirements.txt                       # Python dependencies
-├── 📄 config.py                              # Configuration management
-├── 📄 water_scraper_v3.py                    # Main application (15-min scheduler)
-├── 📄 database_adapters.py                   # Multi-database support
-├── 📄 demo_databases.py                      # Database demonstration
-├── 📄 Dockerfile                             # Container configuration
-├── 📄 docker-compose.victoriametrics.yml     # VictoriaMetrics stack
-├── 📚 Documentation/
-│   ├── 📄 DATABASE_DEPLOYMENT_GUIDE.md       # Multi-database setup guide
-│   ├── 📄 DEBIAN_TROUBLESHOOTING.md          # Linux deployment guide
-│   ├── 📄 ENHANCED_SCHEDULER_GUIDE.md        # 15-minute scheduler guide
-│   ├── 📄 GAP_FILLING_GUIDE.md              # Data gap filling guide
-│   ├── 📄 HTTPS_CONFIGURATION.md            # HTTPS setup guide
-│   └── 📄 VICTORIAMETRICS_SETUP.md          # VictoriaMetrics guide
-└── 📁 grafana/                              # Grafana configuration
-    ├── 📁 provisioning/
-    │   ├── 📁 datasources/
-    │   │   └── 📄 victoriametrics.yml        # VictoriaMetrics data source
-    │   └── 📁 dashboards/
-    │       └── 📄 dashboard.yml              # Dashboard provider config
-    └── 📁 dashboards/
-        └── 📄 water-monitoring-dashboard.json # Pre-built dashboard
-```
-
-## 🧹 **Files Removed During Cleanup**
-
-### **Old Data Files**
-- ❌ `thailand_water_data_v2.csv` - Old CSV export
-- ❌ `water_monitor.log` - Log file (regenerated automatically)
-- ❌ `water_monitoring.db` - SQLite database (recreated automatically)
-
-### **Outdated Documentation**
-- ❌ `FINAL_SUMMARY.md` - Contained references to non-existent v2 files
-- ❌ `PROJECT_SUMMARY.md` - Outdated project information
-
-### **System Files**
-- ❌ `__pycache__/` - Python compiled files directory
-
-## ✅ **Current Features**
-
-### **Enhanced 15-Minute Scheduler**
-- **Timing**: Runs every 15 minutes (1:00, 1:15, 1:30, 1:45, 2:00, etc.)
-- **Full Checks**: At :00 minutes (gap filling + data updates)
-- **Quick Checks**: At :15, :30, :45 minutes (data fetch only)
-- **Gap Filling**: Automatically fills missing historical data
-- **Data Updates**: Updates existing records when values change
-
-### **Multi-Database Support**
-- **VictoriaMetrics** (Recommended) - High-performance time-series
-- **InfluxDB** - Purpose-built time-series database
-- **PostgreSQL + TimescaleDB** - Relational with time-series optimization
-- **MySQL** - Traditional relational database
-- **SQLite** - Local development and testing
-
-### **Production Features**
-- **Docker Support**: Complete containerization
-- **Grafana Integration**: Pre-built dashboards
-- **HTTPS Configuration**: Secure deployment options
-- **Health Monitoring**: Comprehensive logging and error handling
-- **Gap Detection**: Automatic identification of missing data
-- **Retry Logic**: Database lock handling and network error recovery
-
-## 🚀 **Quick Start**
-
-### **1. Basic Setup (SQLite)**
-```bash
-cd water_level_monitor
-pip install -r requirements.txt
-python water_scraper_v3.py
-```
-
-### **2. VictoriaMetrics Setup**
-```bash
-# Start VictoriaMetrics + Grafana
-docker-compose -f docker-compose.victoriametrics.yml up -d
-
-# Configure environment
-export DB_TYPE=victoriametrics
-export VM_HOST=localhost
-export VM_PORT=8428
-
-# Run monitor
-python water_scraper_v3.py
-```
-
-### **3. Test Different Databases**
-```bash
-# Test all supported databases
-python demo_databases.py all
-
-# Test specific database
-python demo_databases.py victoriametrics
-```
-
-## 📊 **Data Collection**
-
-### **Station Coverage**
-- **16 Water Monitoring Stations** across Thailand
-- **Accurate Station Codes**: P.1, P.20, P.21, P.4A, P.5, P.67, P.75, P.76, P.77, P.81, P.82, P.84, P.85, P.87, P.92, P.103
-- **Bilingual Names**: Thai and English station identification
-
-### **Metrics Collected**
-- 🌊 **Water Level**: Measured in meters (m)
-- 💧 **Discharge**: Measured in cubic meters per second (cms)
-- 📊 **Discharge Percentage**: Relative to station capacity
-- ⏰ **Timestamp**: Hour 24 handling (midnight = 00:00 next day)
-
-### **Data Frequency**
-- **Every 15 Minutes**: Continuous monitoring
-- **~300+ Data Points**: Per collection cycle
-- **Automatic Gap Filling**: Historical data recovery
-- **Data Updates**: Changed values detection and correction
-
-## 🔧 **Command Line Tools**
-
-### **Main Application**
-```bash
-python water_scraper_v3.py                    # Run continuous monitoring
-python water_scraper_v3.py --test             # Single test cycle
-python water_scraper_v3.py --help             # Show help
-```
-
-### **Gap Management**
-```bash
-python water_scraper_v3.py --check-gaps [days]  # Check for missing data
-python water_scraper_v3.py --fill-gaps [days]   # Fill missing data gaps
-python water_scraper_v3.py --update-data [days] # Update existing data
-```
-
-### **Database Testing**
-```bash
-python demo_databases.py                      # SQLite demo
-python demo_databases.py victoriametrics      # VictoriaMetrics demo
-python demo_databases.py all                  # Test all databases
-```
-
-## 📈 **Monitoring & Visualization**
-
-### **Grafana Dashboard**
-- **URL**: http://localhost:3000 (when using docker-compose)
-- **Username**: admin
-- **Password**: admin_password
-- **Features**: Time series charts, status tables, gauges, alerts
-
-### **VictoriaMetrics API**
-- **URL**: http://localhost:8428
-- **Health**: http://localhost:8428/health
-- **Metrics**: http://localhost:8428/metrics
-- **Query API**: http://localhost:8428/api/v1/query
-
-## 🛡️ **Security & Production**
-
-### **HTTPS Configuration**
-- Complete guide in `HTTPS_CONFIGURATION.md`
-- SSL certificate setup
-- Reverse proxy configuration
-- Security best practices
-
-### **Deployment Options**
-- **Docker**: Containerized deployment
-- **Systemd**: Linux service configuration
-- **Cloud**: AWS, GCP, Azure deployment guides
-- **Monitoring**: Health checks and alerting
-
-## 📚 **Documentation**
-
-### **Available Guides**
-1. **README.md** - Main project documentation
-2. **DATABASE_DEPLOYMENT_GUIDE.md** - Multi-database setup
-3. **ENHANCED_SCHEDULER_GUIDE.md** - 15-minute scheduler details
-4. **GAP_FILLING_GUIDE.md** - Data integrity and gap filling
-5. **DEBIAN_TROUBLESHOOTING.md** - Linux deployment troubleshooting
-6. **VICTORIAMETRICS_SETUP.md** - VictoriaMetrics configuration
-7. **HTTPS_CONFIGURATION.md** - Secure deployment setup
-
-### **Key Features Documented**
-- ✅ Installation and configuration
-- ✅ Multi-database support
-- ✅ 15-minute scheduling system
-- ✅ Gap filling and data integrity
-- ✅ Production deployment
-- ✅ Monitoring and troubleshooting
-- ✅ Security configuration
-
-## 🎯 **Project Status: PRODUCTION READY**
-
-The Thailand Water Monitor is now:
-- ✅ **Clean**: All old and redundant files removed
-- ✅ **Organized**: Clear project structure with proper documentation
-- ✅ **Enhanced**: 15-minute scheduling with gap filling
-- ✅ **Scalable**: Multi-database support with VictoriaMetrics
-- ✅ **Secure**: HTTPS configuration and security best practices
-- ✅ **Monitored**: Comprehensive logging and Grafana dashboards
-- ✅ **Documented**: Complete guides for all features and deployment options
-
-The project is ready for production deployment with professional-grade monitoring capabilities.

docs/PROJECT_STRUCTURE.md

@@ -1,272 +0,0 @@
-# 🏗️ Project Structure - Northern Thailand Ping River Monitor
-
-## 📁 Directory Layout
-
-```
-Northern-Thailand-Ping-River-Monitor/
-├── 📁 src/                          # Main application source code
-│   ├── __init__.py                  # Package initialization
-│   ├── main.py                      # CLI entry point and main application
-│   ├── water_scraper_v3.py          # Core data collection engine
-│   ├── web_api.py                   # FastAPI web interface
-│   ├── config.py                    # Configuration management
-│   ├── database_adapters.py         # Database abstraction layer
-│   ├── models.py                    # Data models and type definitions
-│   ├── exceptions.py                # Custom exception classes
-│   ├── validators.py                # Data validation layer
-│   ├── metrics.py                   # Metrics collection system
-│   ├── health_check.py              # Health monitoring system
-│   ├── rate_limiter.py              # Rate limiting and request tracking
-│   └── logging_config.py            # Enhanced logging configuration
-├── 📁 docs/                         # Documentation files
-│   ├── STATION_MANAGEMENT_GUIDE.md  # Station management documentation
-│   ├── ENHANCEMENT_SUMMARY.md       # Feature enhancement summary
-│   └── PROJECT_STRUCTURE.md         # This file
-├── 📁 scripts/                      # Utility scripts
-│   └── migrate_geolocation.py       # Database migration script
-├── 📁 grafana/                      # Grafana configuration
-│   ├── dashboards/                  # Dashboard definitions
-│   └── provisioning/                # Grafana provisioning config
-├── 📁 tests/                        # Test files
-│   ├── test_integration.py          # Integration test suite
-│   ├── test_station_management.py   # Station management tests
-│   └── test_api.py                  # API endpoint tests
-├── 📄 run.py                        # Simple startup script
-├── 📄 requirements.txt              # Production dependencies
-├── 📄 requirements-dev.txt          # Development dependencies
-├── 📄 setup.py                      # Package installation script
-├── 📄 Dockerfile                    # Docker container definition
-├── 📄 docker-compose.victoriametrics.yml  # Complete stack deployment
-├── 📄 Makefile                      # Common development tasks
-├── 📄 .env.example                  # Environment configuration template
-├── 📄 .gitignore                    # Git ignore patterns
-├── 📄 .gitlab-ci.yml                # CI/CD pipeline configuration
-├── 📄 LICENSE                       # MIT license
-├── 📄 README.md                     # Main project documentation
-└── 📄 CONTRIBUTING.md               # Contribution guidelines
-```
-
-## 🔧 Core Components
-
-### **Application Layer**
-- **`src/main.py`** - Command-line interface and application orchestration
-- **`src/web_api.py`** - FastAPI web interface with REST endpoints
-- **`src/water_scraper_v3.py`** - Core data collection and processing engine
-
-### **Data Layer**
-- **`src/database_adapters.py`** - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
-- **`src/models.py`** - Pydantic data models and type definitions
-- **`src/validators.py`** - Data validation and sanitization
-
-### **Infrastructure Layer**
-- **`src/config.py`** - Configuration management with environment variable support
-- **`src/logging_config.py`** - Structured logging with rotation and colors
-- **`src/metrics.py`** - Application metrics collection (counters, gauges, histograms)
-- **`src/health_check.py`** - System health monitoring and status checks
-
-### **Utility Layer**
-- **`src/exceptions.py`** - Custom exception hierarchy
-- **`src/rate_limiter.py`** - API rate limiting and request tracking
-
-## 🌐 Web API Structure
-
-### **Endpoints Organization**
-```
-/                           # Dashboard homepage
-├── /health                 # System health status
-├── /metrics               # Application metrics
-├── /config                # Configuration (masked)
-├── /stations              # Station management
-│   ├── GET /              # List all stations
-│   ├── POST /             # Create new station
-│   ├── GET /{id}          # Get specific station
-│   ├── PUT /{id}          # Update station
-│   └── DELETE /{id}       # Delete station
-├── /measurements          # Data access
-│   ├── /latest            # Latest measurements
-│   └── /station/{code}    # Station-specific data
-└── /scraping              # Data collection control
-    ├── /trigger           # Manual data collection
-    └── /status            # Scraping status
-```
-
-### **API Models**
-- **Request Models**: Station creation/update, query parameters
-- **Response Models**: Station info, measurements, health status
-- **Error Models**: Standardized error responses
-
-## 🗄️ Database Architecture
-
-### **Supported Databases**
-1. **SQLite** - Local development and testing
-2. **MySQL** - Traditional relational database
-3. **PostgreSQL** - Advanced relational with TimescaleDB support
-4. **InfluxDB** - Purpose-built time-series database
-5. **VictoriaMetrics** - High-performance metrics storage
-
-### **Schema Design**
-```sql
--- Stations table
-stations (
-    id INTEGER PRIMARY KEY,
-    station_code VARCHAR(10) UNIQUE,
-    thai_name VARCHAR(255),
-    english_name VARCHAR(255),
-    latitude DECIMAL(10,8),
-    longitude DECIMAL(11,8),
-    geohash VARCHAR(20),
-    status VARCHAR(20),
-    created_at TIMESTAMP,
-    updated_at TIMESTAMP
-)
-
--- Measurements table
-water_measurements (
-    id BIGINT PRIMARY KEY,
-    timestamp DATETIME,
-    station_id INTEGER,
-    water_level DECIMAL(10,3),
-    discharge DECIMAL(10,2),
-    discharge_percent DECIMAL(5,2),
-    status VARCHAR(20),
-    created_at TIMESTAMP,
-    FOREIGN KEY (station_id) REFERENCES stations(id),
-    UNIQUE(timestamp, station_id)
-)
-```
-
-## 🐳 Docker Architecture
-
-### **Multi-Stage Build**
-1. **Builder Stage** - Compile dependencies and build artifacts
-2. **Production Stage** - Minimal runtime environment
-
-### **Service Composition**
-- **ping-river-monitor** - Data collection service
-- **ping-river-api** - Web API service
-- **victoriametrics** - Time-series database
-- **grafana** - Visualization dashboard
-
-## 📊 Monitoring Architecture
-
-### **Metrics Collection**
-- **Counters** - API requests, database operations, scraping cycles
-- **Gauges** - Current values, connection status, resource usage
-- **Histograms** - Response times, processing durations
-
-### **Health Checks**
-- **Database Health** - Connection status, data freshness
-- **API Health** - External API availability, response times
-- **System Health** - Memory usage, disk space, CPU load
-
-### **Logging Levels**
-- **DEBUG** - Detailed execution information
-- **INFO** - General operational messages
-- **WARNING** - Potential issues and recoverable errors
-- **ERROR** - Serious problems requiring attention
-- **CRITICAL** - System-threatening issues
-
-## 🔧 Configuration Management
-
-### **Environment Variables**
-```bash
-# Database
-DB_TYPE=victoriametrics
-VM_HOST=localhost
-VM_PORT=8428
-
-# Application
-SCRAPING_INTERVAL_HOURS=1
-LOG_LEVEL=INFO
-DATA_RETENTION_DAYS=365
-
-# Security
-SECRET_KEY=your-secret-key
-API_KEY=your-api-key
-```
-
-### **Configuration Hierarchy**
-1. Environment variables (highest priority)
-2. .env file
-3. Default values in config.py (lowest priority)
-
-## 🧪 Testing Architecture
-
-### **Test Categories**
-- **Unit Tests** - Individual component testing
-- **Integration Tests** - System component interaction
-- **API Tests** - Endpoint functionality and responses
-- **Performance Tests** - Load and stress testing
-
-### **Test Data**
-- **Mock Data** - Simulated API responses
-- **Test Database** - Isolated test environment
-- **Fixtures** - Reusable test data sets
-
-## 📦 Deployment Architecture
-
-### **Development**
-```bash
-python run.py --web-api    # Local development server
-```
-
-### **Production**
-```bash
-docker-compose up -d       # Full stack deployment
-```
-
-### **CI/CD Pipeline**
-1. **Test Stage** - Run all tests and quality checks
-2. **Build Stage** - Create Docker images
-3. **Deploy Stage** - Deploy to staging/production
-4. **Health Check** - Verify deployment success
-
-## 🔒 Security Architecture
-
-### **Input Validation**
-- Pydantic models for API requests
-- Data range validation for measurements
-- SQL injection prevention through ORM
-
-### **Authentication** (Future)
-- API key authentication
-- JWT token support
-- Role-based access control
-
-### **Data Protection**
-- Environment variable configuration
-- Sensitive data masking in logs
-- HTTPS support for production
-
-## 📈 Performance Architecture
-
-### **Optimization Strategies**
-- Database connection pooling
-- Query optimization and indexing
-- Response caching for static data
-- Async processing for I/O operations
-
-### **Scalability Considerations**
-- Horizontal scaling with load balancers
-- Database read replicas
-- Microservice architecture readiness
-- Container orchestration support
-
-## 🔄 Data Flow Architecture
-
-### **Collection Flow**
-```
-External API → Rate Limiter → Data Validator → Database Adapter → Database
-```
-
-### **API Flow**
-```
-HTTP Request → FastAPI → Business Logic → Database Adapter → HTTP Response
-```
-
-### **Monitoring Flow**
-```
-Application Events → Metrics Collector → Health Checks → Monitoring Dashboard
-```
-
-This architecture provides a solid foundation for a production-ready water monitoring system with excellent maintainability, scalability, and observability.

models/eval_2026-09-12.json

@@ -0,0 +1,723 @@
+[
+ {
+  "station": "P.1",
+  "warn_thr": 3.7,
+  "folds": [
+   {
+    "year": 2021,
+    "n_train": 20024,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.07375926701460789,
+      "mae_above_2p5": null,
+      "brier_warn": 0.0,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.07332598842242062,
+      "mae_above_2p5": null,
+      "brier_warn": 0.0,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.07383022350644125,
+      "mae_above_2p5": null,
+      "brier_warn": 1.0850721383440065e-12,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.06977345537342049,
+      "mae_above_2p5": null,
+      "brier_warn": 7.128994064266462e-16,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2022,
+    "n_train": 28784,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2022-10-02T19:00:00",
+      "peak_ts": "2022-10-03T15:00:00",
+      "peak_level": 4.65
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.08452847354970178,
+      "mae_above_2p5": 0.25082252888260664,
+      "brier_warn": 0.004300908725927739,
+      "events": [
+       {
+        "crossing": "2022-10-02T19:00:00",
+        "lead_h": 5.0,
+        "peak_level": 4.65,
+        "peak_pred_24h_before": 3.8173954245046406
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.08369773912557228,
+      "mae_above_2p5": 0.24702357745371217,
+      "brier_warn": 0.004325741658698973,
+      "events": [
+       {
+        "crossing": "2022-10-02T19:00:00",
+        "lead_h": 5.0,
+        "peak_level": 4.65,
+        "peak_pred_24h_before": 3.8114190118860223
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.08230165237819607,
+      "mae_above_2p5": 0.2599241552494541,
+      "brier_warn": 0.004191550822623699,
+      "events": [
+       {
+        "crossing": "2022-10-02T19:00:00",
+        "lead_h": 3.0,
+        "peak_level": 4.65,
+        "peak_pred_24h_before": 3.693734826616603
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.08181109208140971,
+      "mae_above_2p5": 0.2639358812546371,
+      "brier_warn": 0.004581181754314636,
+      "events": [
+       {
+        "crossing": "2022-10-02T19:00:00",
+        "lead_h": 2.0,
+        "peak_level": 4.65,
+        "peak_pred_24h_before": 3.620470606696475
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2023,
+    "n_train": 37539,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.07696637416933236,
+      "mae_above_2p5": 0.10643655855133666,
+      "brier_warn": 1.919860722404625e-15,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.07708101918232377,
+      "mae_above_2p5": 0.09974894450126857,
+      "brier_warn": 1.7028444765622288e-15,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.06968496247786034,
+      "mae_above_2p5": 0.10210094973854984,
+      "brier_warn": 1.0987601008721297e-09,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.06726603345661021,
+      "mae_above_2p5": 0.09716644572204487,
+      "brier_warn": 2.7085590803510675e-09,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2024,
+    "n_train": 46323,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2024-09-24T17:00:00",
+      "peak_ts": "2024-09-26T02:00:00",
+      "peak_level": 4.93
+     },
+     {
+      "crossing": "2024-10-03T09:00:00",
+      "peak_ts": "2024-10-05T12:00:00",
+      "peak_level": 5.3
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.0890019157543212,
+      "mae_above_2p5": 0.2093245927883516,
+      "brier_warn": 0.005826169840715695,
+      "events": [
+       {
+        "crossing": "2024-09-24T17:00:00",
+        "lead_h": 10.0,
+        "peak_level": 4.93,
+        "peak_pred_24h_before": 4.817519939833057
+       },
+       {
+        "crossing": "2024-10-03T09:00:00",
+        "lead_h": 21.0,
+        "peak_level": 5.3,
+        "peak_pred_24h_before": 5.546588884631041
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.08875916089292855,
+      "mae_above_2p5": 0.20975114218621593,
+      "brier_warn": 0.006061154593275248,
+      "events": [
+       {
+        "crossing": "2024-09-24T17:00:00",
+        "lead_h": 11.0,
+        "peak_level": 4.93,
+        "peak_pred_24h_before": 4.800924141216692
+       },
+       {
+        "crossing": "2024-10-03T09:00:00",
+        "lead_h": 72.0,
+        "peak_level": 5.3,
+        "peak_pred_24h_before": 5.577164984770105
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.08341835741803395,
+      "mae_above_2p5": 0.18688839374651373,
+      "brier_warn": 0.003188229521816099,
+      "events": [
+       {
+        "crossing": "2024-09-24T17:00:00",
+        "lead_h": 17.0,
+        "peak_level": 4.93,
+        "peak_pred_24h_before": 5.058029430632501
+       },
+       {
+        "crossing": "2024-10-03T09:00:00",
+        "lead_h": 21.0,
+        "peak_level": 5.3,
+        "peak_pred_24h_before": 5.3311787370709975
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.08412336465267245,
+      "mae_above_2p5": 0.19173218504592401,
+      "brier_warn": 0.0034722638888286116,
+      "events": [
+       {
+        "crossing": "2024-09-24T17:00:00",
+        "lead_h": 15.0,
+        "peak_level": 4.93,
+        "peak_pred_24h_before": 5.0118284217314
+       },
+       {
+        "crossing": "2024-10-03T09:00:00",
+        "lead_h": 21.0,
+        "peak_level": 5.3,
+        "peak_pred_24h_before": 5.3520431553190155
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2025,
+    "n_train": 55083,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2025-09-27T18:00:00",
+      "peak_ts": "2025-09-27T22:00:00",
+      "peak_level": 3.93
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.11202835461695825,
+      "mae_above_2p5": 0.2456782496767171,
+      "brier_warn": 0.005178356039745929,
+      "events": [
+       {
+        "crossing": "2025-09-27T18:00:00",
+        "lead_h": 2.0,
+        "peak_level": 3.93,
+        "peak_pred_24h_before": 3.23
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.11083068059655521,
+      "mae_above_2p5": 0.23787013498709667,
+      "brier_warn": 0.005228043825289021,
+      "events": [
+       {
+        "crossing": "2025-09-27T18:00:00",
+        "lead_h": 2.0,
+        "peak_level": 3.93,
+        "peak_pred_24h_before": 3.23
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.10372526043263834,
+      "mae_above_2p5": 0.23846363852091013,
+      "brier_warn": 0.005898259026876986,
+      "events": [
+       {
+        "crossing": "2025-09-27T18:00:00",
+        "lead_h": 1.0,
+        "peak_level": 3.93,
+        "peak_pred_24h_before": 3.23
+       }
+      ],
+      "false_alarm_episodes": 1
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.10267267834487567,
+      "mae_above_2p5": 0.2507951319537275,
+      "brier_warn": 0.005020467408392599,
+      "events": [
+       {
+        "crossing": "2025-09-27T18:00:00",
+        "lead_h": 2.0,
+        "peak_level": 3.93,
+        "peak_pred_24h_before": 3.2417205711942434
+       }
+      ],
+      "false_alarm_episodes": 1
+     }
+    }
+   }
+  ]
+ },
+ {
+  "station": "P.103",
+  "warn_thr": 5.95,
+  "folds": [
+   {
+    "year": 2021,
+    "n_train": 20009,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.14438683486790602,
+      "mae_above_2p5": null,
+      "brier_warn": 0.0,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.14305613024043953,
+      "mae_above_2p5": null,
+      "brier_warn": 0.0,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.13512418754716052,
+      "mae_above_2p5": null,
+      "brier_warn": 6.937198832251013e-10,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.10270057685061172,
+      "mae_above_2p5": null,
+      "brier_warn": 6.565945930999852e-14,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2022,
+    "n_train": 28769,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2022-08-14T04:00:00",
+      "peak_ts": "2022-08-14T08:00:00",
+      "peak_level": 6.09
+     },
+     {
+      "crossing": "2022-10-02T16:00:00",
+      "peak_ts": "2022-10-03T16:00:00",
+      "peak_level": 7.54
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.14392334606606189,
+      "mae_above_2p5": 0.41075382386412596,
+      "brier_warn": 0.00764679988213082,
+      "events": [
+       {
+        "crossing": "2022-08-14T04:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.09,
+        "peak_pred_24h_before": 4.825270553204425
+       },
+       {
+        "crossing": "2022-10-02T16:00:00",
+        "lead_h": 9.0,
+        "peak_level": 7.54,
+        "peak_pred_24h_before": 7.194064117976157
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.14284722696431765,
+      "mae_above_2p5": 0.39401132538096667,
+      "brier_warn": 0.007427375799450148,
+      "events": [
+       {
+        "crossing": "2022-08-14T04:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.09,
+        "peak_pred_24h_before": 4.845896993132048
+       },
+       {
+        "crossing": "2022-10-02T16:00:00",
+        "lead_h": 9.0,
+        "peak_level": 7.54,
+        "peak_pred_24h_before": 7.194661123502819
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.1458954690776336,
+      "mae_above_2p5": 0.4674051188369517,
+      "brier_warn": 0.00899991317044724,
+      "events": [
+       {
+        "crossing": "2022-08-14T04:00:00",
+        "lead_h": 1.0,
+        "peak_level": 6.09,
+        "peak_pred_24h_before": 4.82714042795344
+       },
+       {
+        "crossing": "2022-10-02T16:00:00",
+        "lead_h": 5.0,
+        "peak_level": 7.54,
+        "peak_pred_24h_before": 6.59871451008115
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.13203958422483053,
+      "mae_above_2p5": 0.4902773906613983,
+      "brier_warn": 0.008514527559601331,
+      "events": [
+       {
+        "crossing": "2022-08-14T04:00:00",
+        "lead_h": 4.0,
+        "peak_level": 6.09,
+        "peak_pred_24h_before": 4.895369771408423
+       },
+       {
+        "crossing": "2022-10-02T16:00:00",
+        "lead_h": 3.0,
+        "peak_level": 7.54,
+        "peak_pred_24h_before": 6.294082735853337
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2023,
+    "n_train": 37524,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.16195301512514512,
+      "mae_above_2p5": null,
+      "brier_warn": 3.0208441147560167e-07,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.16334224973870387,
+      "mae_above_2p5": null,
+      "brier_warn": 1.1626187140381066e-08,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.14755024879522577,
+      "mae_above_2p5": null,
+      "brier_warn": 3.850104714388072e-05,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.10589805327026759,
+      "mae_above_2p5": null,
+      "brier_warn": 1.8896757611081001e-07,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2024,
+    "n_train": 46308,
+    "n_test": 4058,
+    "events": [
+     {
+      "crossing": "2024-09-24T10:00:00",
+      "peak_ts": "2024-09-26T00:00:00",
+      "peak_level": 8.27
+     },
+     {
+      "crossing": "2024-09-30T03:00:00",
+      "peak_ts": "2024-09-30T06:00:00",
+      "peak_level": 5.99
+     },
+     {
+      "crossing": "2024-10-03T06:00:00",
+      "peak_ts": "2024-10-05T07:00:00",
+      "peak_level": 9.93
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.13901842325128816,
+      "mae_above_2p5": 0.2887416310966684,
+      "brier_warn": 0.0072040857753137046,
+      "events": [
+       {
+        "crossing": "2024-09-24T10:00:00",
+        "lead_h": 19.0,
+        "peak_level": 8.27,
+        "peak_pred_24h_before": 8.212734363860193
+       },
+       {
+        "crossing": "2024-09-30T03:00:00",
+        "lead_h": 9.0,
+        "peak_level": 5.99,
+        "peak_pred_24h_before": 5.47565489914091
+       },
+       {
+        "crossing": "2024-10-03T06:00:00",
+        "lead_h": 55.0,
+        "peak_level": 9.93,
+        "peak_pred_24h_before": 8.780278464915938
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_fc48": {
+      "mae": 0.13850733053801131,
+      "mae_above_2p5": 0.29817392926900865,
+      "brier_warn": 0.007928846574280278,
+      "events": [
+       {
+        "crossing": "2024-09-24T10:00:00",
+        "lead_h": 19.0,
+        "peak_level": 8.27,
+        "peak_pred_24h_before": 8.166020251020889
+       },
+       {
+        "crossing": "2024-09-30T03:00:00",
+        "lead_h": 10.0,
+        "peak_level": 5.99,
+        "peak_pred_24h_before": 5.524584037259288
+       },
+       {
+        "crossing": "2024-10-03T06:00:00",
+        "lead_h": 69.0,
+        "peak_level": 9.93,
+        "peak_pred_24h_before": 8.595120917488185
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile": {
+      "mae": 0.12640546558686208,
+      "mae_above_2p5": 0.27944115421093924,
+      "brier_warn": 0.007730268539879654,
+      "events": [
+       {
+        "crossing": "2024-09-24T10:00:00",
+        "lead_h": 20.0,
+        "peak_level": 8.27,
+        "peak_pred_24h_before": 8.336487732683672
+       },
+       {
+        "crossing": "2024-09-30T03:00:00",
+        "lead_h": 4.0,
+        "peak_level": 5.99,
+        "peak_pred_24h_before": 5.45284915024346
+       },
+       {
+        "crossing": "2024-10-03T06:00:00",
+        "lead_h": 69.0,
+        "peak_level": 9.93,
+        "peak_pred_24h_before": 8.575915226697406
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.12480975852168202,
+      "mae_above_2p5": 0.2857575557415695,
+      "brier_warn": 0.005894928998647136,
+      "events": [
+       {
+        "crossing": "2024-09-24T10:00:00",
+        "lead_h": 21.0,
+        "peak_level": 8.27,
+        "peak_pred_24h_before": 8.444720326882825
+       },
+       {
+        "crossing": "2024-09-30T03:00:00",
+        "lead_h": 9.0,
+        "peak_level": 5.99,
+        "peak_pred_24h_before": 5.42
+       },
+       {
+        "crossing": "2024-10-03T06:00:00",
+        "lead_h": 32.0,
+        "peak_level": 9.93,
+        "peak_pred_24h_before": 8.781001847167515
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2025,
+    "n_train": 54577,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2025-09-26T06:00:00",
+      "peak_ts": "2025-09-27T21:00:00",
+      "peak_level": 6.64
+     },
+     {
+      "crossing": "2025-10-03T06:00:00",
+      "peak_ts": "2025-10-03T12:00:00",
+      "peak_level": 6.14
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.17206685418009837,
+      "mae_above_2p5": 0.3840520085986099,
+      "brier_warn": 0.015824852891785323,
+      "events": [
+       {
+        "crossing": "2025-09-26T06:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.64,
+        "peak_pred_24h_before": 5.73043665401637
+       },
+       {
+        "crossing": "2025-10-03T06:00:00",
+        "lead_h": 8.0,
+        "peak_level": 6.14,
+        "peak_pred_24h_before": 5.609919760117381
+       }
+      ],
+      "false_alarm_episodes": 2
+     },
+     "rise_rain_fc48": {
+      "mae": 0.1722274451362234,
+      "mae_above_2p5": 0.3872216974630654,
+      "brier_warn": 0.016245727130063177,
+      "events": [
+       {
+        "crossing": "2025-09-26T06:00:00",
+        "lead_h": 4.0,
+        "peak_level": 6.64,
+        "peak_pred_24h_before": 5.7312263707793685
+       },
+       {
+        "crossing": "2025-10-03T06:00:00",
+        "lead_h": 8.0,
+        "peak_level": 6.14,
+        "peak_pred_24h_before": 5.685511824758637
+       }
+      ],
+      "false_alarm_episodes": 2
+     },
+     "rise_rain_quantile": {
+      "mae": 0.15912275848686555,
+      "mae_above_2p5": 0.3845695612674703,
+      "brier_warn": 0.014797606329811499,
+      "events": [
+       {
+        "crossing": "2025-09-26T06:00:00",
+        "lead_h": 11.0,
+        "peak_level": 6.64,
+        "peak_pred_24h_before": 5.73
+       },
+       {
+        "crossing": "2025-10-03T06:00:00",
+        "lead_h": 10.0,
+        "peak_level": 6.14,
+        "peak_pred_24h_before": 5.58133012298031
+       }
+      ],
+      "false_alarm_episodes": 2
+     },
+     "rise_rain_quantile_uw": {
+      "mae": 0.15385355845103685,
+      "mae_above_2p5": 0.41314645854725585,
+      "brier_warn": 0.016528116336109958,
+      "events": [
+       {
+        "crossing": "2025-09-26T06:00:00",
+        "lead_h": 9.0,
+        "peak_level": 6.64,
+        "peak_pred_24h_before": 5.741254234340573
+       },
+       {
+        "crossing": "2025-10-03T06:00:00",
+        "lead_h": 8.0,
+        "peak_level": 6.14,
+        "peak_pred_24h_before": 5.418105405306207
+       }
+      ],
+      "false_alarm_episodes": 2
+     }
+    }
+   }
+  ]
+ }
+]

models/eval_2026-09-12_qsigma.json

@@ -0,0 +1,439 @@
+[
+ {
+  "station": "P.1",
+  "warn_thr": 3.7,
+  "folds": [
+   {
+    "year": 2021,
+    "n_train": 20024,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.07375926701460789,
+      "mae_above_2p5": null,
+      "brier_warn": 0.0,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.07375926701460789,
+      "mae_above_2p5": null,
+      "brier_warn": 7.852802408474157e-14,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2022,
+    "n_train": 28784,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2022-10-02T19:00:00",
+      "peak_ts": "2022-10-03T15:00:00",
+      "peak_level": 4.65
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.08452847354970178,
+      "mae_above_2p5": 0.25082252888260664,
+      "brier_warn": 0.004300908725927739,
+      "events": [
+       {
+        "crossing": "2022-10-02T19:00:00",
+        "lead_h": 5.0,
+        "peak_level": 4.65,
+        "peak_pred_24h_before": 3.8173954245046406
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.08452847354970178,
+      "mae_above_2p5": 0.25082252888260664,
+      "brier_warn": 0.0039815091186836665,
+      "events": [
+       {
+        "crossing": "2022-10-02T19:00:00",
+        "lead_h": 5.0,
+        "peak_level": 4.65,
+        "peak_pred_24h_before": 3.8173954245046406
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2023,
+    "n_train": 37539,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.07696637416933236,
+      "mae_above_2p5": 0.10643655855133666,
+      "brier_warn": 1.919860722404625e-15,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.07696637416933236,
+      "mae_above_2p5": 0.10643655855133666,
+      "brier_warn": 4.0844243581898366e-08,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2024,
+    "n_train": 46323,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2024-09-24T17:00:00",
+      "peak_ts": "2024-09-26T02:00:00",
+      "peak_level": 4.93
+     },
+     {
+      "crossing": "2024-10-03T09:00:00",
+      "peak_ts": "2024-10-05T12:00:00",
+      "peak_level": 5.3
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.0890019157543212,
+      "mae_above_2p5": 0.2093245927883516,
+      "brier_warn": 0.005826169840715695,
+      "events": [
+       {
+        "crossing": "2024-09-24T17:00:00",
+        "lead_h": 10.0,
+        "peak_level": 4.93,
+        "peak_pred_24h_before": 4.817519939833057
+       },
+       {
+        "crossing": "2024-10-03T09:00:00",
+        "lead_h": 21.0,
+        "peak_level": 5.3,
+        "peak_pred_24h_before": 5.546588884631041
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.0890019157543212,
+      "mae_above_2p5": 0.2093245927883516,
+      "brier_warn": 0.005475787026695418,
+      "events": [
+       {
+        "crossing": "2024-09-24T17:00:00",
+        "lead_h": 10.0,
+        "peak_level": 4.93,
+        "peak_pred_24h_before": 4.817519939833057
+       },
+       {
+        "crossing": "2024-10-03T09:00:00",
+        "lead_h": 21.0,
+        "peak_level": 5.3,
+        "peak_pred_24h_before": 5.546588884631041
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2025,
+    "n_train": 55083,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2025-09-27T18:00:00",
+      "peak_ts": "2025-09-27T22:00:00",
+      "peak_level": 3.93
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.11202835461695825,
+      "mae_above_2p5": 0.2456782496767171,
+      "brier_warn": 0.005178356039745929,
+      "events": [
+       {
+        "crossing": "2025-09-27T18:00:00",
+        "lead_h": 2.0,
+        "peak_level": 3.93,
+        "peak_pred_24h_before": 3.23
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.11202835461695825,
+      "mae_above_2p5": 0.2456782496767171,
+      "brier_warn": 0.00491346243876589,
+      "events": [
+       {
+        "crossing": "2025-09-27T18:00:00",
+        "lead_h": 2.0,
+        "peak_level": 3.93,
+        "peak_pred_24h_before": 3.23
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   }
+  ]
+ },
+ {
+  "station": "P.103",
+  "warn_thr": 5.95,
+  "folds": [
+   {
+    "year": 2021,
+    "n_train": 20009,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.14438683486790602,
+      "mae_above_2p5": null,
+      "brier_warn": 0.0,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.14438683486790602,
+      "mae_above_2p5": null,
+      "brier_warn": 3.5349670949872053e-13,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2022,
+    "n_train": 28769,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2022-08-14T04:00:00",
+      "peak_ts": "2022-08-14T08:00:00",
+      "peak_level": 6.09
+     },
+     {
+      "crossing": "2022-10-02T16:00:00",
+      "peak_ts": "2022-10-03T16:00:00",
+      "peak_level": 7.54
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.14392334606606189,
+      "mae_above_2p5": 0.41075382386412596,
+      "brier_warn": 0.00764679988213082,
+      "events": [
+       {
+        "crossing": "2022-08-14T04:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.09,
+        "peak_pred_24h_before": 4.825270553204425
+       },
+       {
+        "crossing": "2022-10-02T16:00:00",
+        "lead_h": 9.0,
+        "peak_level": 7.54,
+        "peak_pred_24h_before": 7.194064117976157
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.14392334606606189,
+      "mae_above_2p5": 0.41075382386412596,
+      "brier_warn": 0.006975493312169236,
+      "events": [
+       {
+        "crossing": "2022-08-14T04:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.09,
+        "peak_pred_24h_before": 4.825270553204425
+       },
+       {
+        "crossing": "2022-10-02T16:00:00",
+        "lead_h": 9.0,
+        "peak_level": 7.54,
+        "peak_pred_24h_before": 7.194064117976157
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2023,
+    "n_train": 37524,
+    "n_test": 4392,
+    "events": [],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.16195301512514512,
+      "mae_above_2p5": null,
+      "brier_warn": 3.0208441147560167e-07,
+      "events": [],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.16195301512514512,
+      "mae_above_2p5": null,
+      "brier_warn": 3.2176039819636275e-05,
+      "events": [],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2024,
+    "n_train": 46308,
+    "n_test": 4058,
+    "events": [
+     {
+      "crossing": "2024-09-24T10:00:00",
+      "peak_ts": "2024-09-26T00:00:00",
+      "peak_level": 8.27
+     },
+     {
+      "crossing": "2024-09-30T03:00:00",
+      "peak_ts": "2024-09-30T06:00:00",
+      "peak_level": 5.99
+     },
+     {
+      "crossing": "2024-10-03T06:00:00",
+      "peak_ts": "2024-10-05T07:00:00",
+      "peak_level": 9.93
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.13901842325128816,
+      "mae_above_2p5": 0.2887416310966684,
+      "brier_warn": 0.0072040857753137046,
+      "events": [
+       {
+        "crossing": "2024-09-24T10:00:00",
+        "lead_h": 19.0,
+        "peak_level": 8.27,
+        "peak_pred_24h_before": 8.212734363860193
+       },
+       {
+        "crossing": "2024-09-30T03:00:00",
+        "lead_h": 9.0,
+        "peak_level": 5.99,
+        "peak_pred_24h_before": 5.47565489914091
+       },
+       {
+        "crossing": "2024-10-03T06:00:00",
+        "lead_h": 55.0,
+        "peak_level": 9.93,
+        "peak_pred_24h_before": 8.780278464915938
+       }
+      ],
+      "false_alarm_episodes": 0
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.13901842325128816,
+      "mae_above_2p5": 0.2887416310966684,
+      "brier_warn": 0.006910847607098417,
+      "events": [
+       {
+        "crossing": "2024-09-24T10:00:00",
+        "lead_h": 19.0,
+        "peak_level": 8.27,
+        "peak_pred_24h_before": 8.212734363860193
+       },
+       {
+        "crossing": "2024-09-30T03:00:00",
+        "lead_h": 9.0,
+        "peak_level": 5.99,
+        "peak_pred_24h_before": 5.47565489914091
+       },
+       {
+        "crossing": "2024-10-03T06:00:00",
+        "lead_h": 55.0,
+        "peak_level": 9.93,
+        "peak_pred_24h_before": 8.780278464915938
+       }
+      ],
+      "false_alarm_episodes": 0
+     }
+    }
+   },
+   {
+    "year": 2025,
+    "n_train": 54577,
+    "n_test": 4392,
+    "events": [
+     {
+      "crossing": "2025-09-26T06:00:00",
+      "peak_ts": "2025-09-27T21:00:00",
+      "peak_level": 6.64
+     },
+     {
+      "crossing": "2025-10-03T06:00:00",
+      "peak_ts": "2025-10-03T12:00:00",
+      "peak_level": 6.14
+     }
+    ],
+    "variants": {
+     "rise_rain": {
+      "mae": 0.17206685418009837,
+      "mae_above_2p5": 0.3840520085986099,
+      "brier_warn": 0.015824852891785323,
+      "events": [
+       {
+        "crossing": "2025-09-26T06:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.64,
+        "peak_pred_24h_before": 5.73043665401637
+       },
+       {
+        "crossing": "2025-10-03T06:00:00",
+        "lead_h": 8.0,
+        "peak_level": 6.14,
+        "peak_pred_24h_before": 5.609919760117381
+       }
+      ],
+      "false_alarm_episodes": 2
+     },
+     "rise_rain_qsigma": {
+      "mae": 0.17206685418009837,
+      "mae_above_2p5": 0.3840520085986099,
+      "brier_warn": 0.015889933586185904,
+      "events": [
+       {
+        "crossing": "2025-09-26T06:00:00",
+        "lead_h": 6.0,
+        "peak_level": 6.64,
+        "peak_pred_24h_before": 5.73043665401637
+       },
+       {
+        "crossing": "2025-10-03T06:00:00",
+        "lead_h": 8.0,
+        "peak_level": 6.14,
+        "peak_pred_24h_before": 5.609919760117381
+       }
+      ],
+      "false_alarm_episodes": 2
+     }
+    }
+   }
+  ]
+ }
+]

ping-river-monitor.spec

@@ -1,38 +0,0 @@
-# -*- mode: python ; coding: utf-8 -*-
-
-
-a = Analysis(
-    ['run.py'],
-    pathex=[],
-    binaries=[],
-    datas=[('.env', '.'), ('sql', 'sql'), ('README.md', '.'), ('POSTGRESQL_SETUP.md', '.'), ('SQLITE_MIGRATION.md', '.')],
-    hiddenimports=['psycopg2', 'sqlalchemy.dialects.postgresql', 'sqlalchemy.dialects.sqlite', 'dotenv', 'pydantic', 'fastapi', 'uvicorn', 'schedule', 'pandas'],
-    hookspath=[],
-    hooksconfig={},
-    runtime_hooks=[],
-    excludes=[],
-    noarchive=False,
-    optimize=0,
-)
-pyz = PYZ(a.pure)
-
-exe = EXE(
-    pyz,
-    a.scripts,
-    a.binaries,
-    a.datas,
-    [],
-    name='ping-river-monitor',
-    debug=False,
-    bootloader_ignore_signals=False,
-    strip=False,
-    upx=True,
-    upx_exclude=[],
-    runtime_tmpdir=None,
-    console=True,
-    disable_windowed_traceback=False,
-    argv_emulation=False,
-    target_arch=None,
-    codesign_identity=None,
-    entitlements_file=None,
-)

pyproject.toml

@@ -34,23 +34,23 @@ classifiers = [
     "Environment :: Web Environment",
     "Framework :: FastAPI"
 ]
-requires-python = ">=3.11"
+requires-python = ">=3.11,<3.12"
 dependencies = [
     # Core dependencies
-    "requests==2.31.0",
+    "requests==2.34.2",
     "schedule==1.2.0",
     "pandas==2.0.3",
     "numpy>=1.24,<2",
     # Flood forecasting (ML)
     "scikit-learn==1.9.0",
     # Web API framework
-    "fastapi==0.104.1",
-    "uvicorn[standard]==0.24.0",
-    "pydantic==2.5.0",
+    "fastapi==0.141.1",
+    "uvicorn[standard]==0.52.4",
+    "pydantic==2.13.5",
     # Database adapters
     "sqlalchemy==2.0.23",
     "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
     "psycopg2-binary==2.9.9",
     # Monitoring and metrics
     "psutil==5.9.6"
@@ -59,11 +59,11 @@ dependencies = [
 [project.optional-dependencies]
 dev = [
     # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
     "pytest-cov==4.1.0",
     "pytest-asyncio==0.21.1",
     # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
     "flake8==6.1.0",
     "isort==5.12.0",
     "mypy==1.7.1",
@@ -73,7 +73,7 @@ dev = [
     "ipython==8.17.2",
     "jupyter==1.0.0",
     # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
     "types-python-dateutil==2.8.19.14"
 ]
 docs = [
@@ -83,7 +83,7 @@ docs = [
 ]
 all = [
     "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
     "psycopg2-binary==2.9.9"
 ]
 
@@ -100,11 +100,11 @@ Documentation = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/
 [dependency-groups]
 dev = [
     # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
     "pytest-cov==4.1.0",
     "pytest-asyncio==0.21.1",
     # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
     "flake8==6.1.0",
     "isort==5.12.0",
     "mypy==1.7.1",
@@ -114,7 +114,7 @@ dev = [
     "ipython==8.17.2",
     "jupyter==1.0.0",
     # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
     "types-python-dateutil==2.8.19.14",
     # Documentation
     "sphinx==7.2.6",
@@ -128,3 +128,16 @@ where = ["src"]
 
 [tool.setuptools.package-dir]
 "" = "src"
+
+# One formatting contract for CI, pre-commit and editors. Black's default 88
+# columns; isort in black-compatible mode. Run `make format` before committing.
+[tool.black]
+line-length = 88
+target-version = ["py311"]
+extend-exclude = '/(\.venv|venv|models|\.claude-flow|\.swarm)/'
+
+[tool.isort]
+profile = "black"
+line_length = 88
+known_first_party = ["src"]
+skip_gitignore = true

requirements-dev.txt

@@ -2,12 +2,12 @@
 -r requirements.txt
 
 # Testing
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
 pytest-asyncio==0.21.1
 
 # Code formatting and linting
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 isort==5.12.0
 mypy==1.7.1
@@ -25,5 +25,5 @@ ipython==8.17.2
 jupyter==1.0.0
 
 # Type stubs
-types-requests==2.31.0.10
+types-requests==2.33.0.20260906
 types-python-dateutil==2.8.19.14

requirements.txt

@@ -1,5 +1,5 @@
 # Core dependencies
-requests==2.31.0
+requests==2.34.2
 schedule==1.2.0
 pandas==2.0.3
 numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
@@ -8,23 +8,23 @@ numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
 scikit-learn==1.9.0
 
 # Web API framework
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-pydantic==2.5.0
+fastapi==0.141.1
+uvicorn[standard]==0.52.4
+pydantic==2.13.5
 
 # Database adapters
 sqlalchemy==2.0.23
 influxdb==5.3.1
-pymysql==1.1.0
+pymysql==1.2.0
 psycopg2-binary==2.9.9
 
 # Monitoring and metrics
 psutil==5.9.6
 
 # Development dependencies (optional)
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 mypy==1.7.1
 pre-commit==3.5.0

scripts/dev_proxy.py

@@ -0,0 +1,58 @@
+"""Serve the working-copy dashboard locally with API calls proxied to the
+live server, so browser-side changes can be checked against real data
+before deploy. Usage: python scripts/dev_proxy.py [port]"""
+
+import http.server
+import os
+import sys
+import urllib.request
+from pathlib import Path
+
+UPSTREAM = "https://water.buildfor.life"
+STATIC = Path(__file__).resolve().parents[1] / "src" / "static"
+
+
+class Handler(http.server.BaseHTTPRequestHandler):
+    def do_GET(self):
+        if self.path == "/" or self.path.startswith("/?"):
+            body = (STATIC / "dashboard.html").read_bytes()
+            self._send(200, "text/html; charset=utf-8", body)
+            return
+        # Local overrides for endpoints not yet deployed: DEV_PROXY_LOCAL=/api/x=file.json,...
+        for pair in filter(None, os.environ.get("DEV_PROXY_LOCAL", "").split(",")):
+            prefix, file = pair.split("=", 1)
+            if self.path.split("?")[0] == prefix:
+                self._send(200, "application/json", Path(file).read_bytes())
+                return
+        if self.path.startswith("/static/"):
+            f = STATIC / self.path[len("/static/"):].split("?")[0]
+            if f.is_file():
+                ctype = "application/json" if f.suffix in (".json", ".geojson") else "application/octet-stream"
+                self._send(200, ctype, f.read_bytes())
+                return
+        try:
+            req = urllib.request.Request(
+                UPSTREAM + self.path,
+                headers={"User-Agent": "Mozilla/5.0 (dev_proxy; +https://buildfor.life)", "Accept": "application/json"},
+            )
+            with urllib.request.urlopen(req, timeout=60) as r:
+                self._send(r.status, r.headers.get("Content-Type", "application/json"), r.read())
+        except urllib.error.HTTPError as e:
+            self._send(e.code, "application/json", e.read())
+
+    def _send(self, code, ctype, body):
+        self.send_response(code)
+        self.send_header("Content-Type", ctype)
+        self.send_header("Content-Length", str(len(body)))
+        self.send_header("Cache-Control", "no-store")
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *a):
+        pass
+
+
+if __name__ == "__main__":
+    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8765
+    print(f"http://localhost:{port}/  (API -> {UPSTREAM})")
+    http.server.ThreadingHTTPServer(("127.0.0.1", port), Handler).serve_forever()

scripts/e2e_notify.py

@@ -0,0 +1,132 @@
+"""Drive the production notify path in-process: startup init -> seeded readings
+-> forecast cache -> _notify_transitions -> sqlite state -> real ntfy."""
+
+import asyncio
+import datetime
+import json
+import os
+import sys
+
+import requests
+
+os.environ.update(
+    DB_TYPE="sqlite",
+    WATER_DB_PATH=os.path.join(os.environ["LOCALAPPDATA"], "Temp", "smoke3.db"),
+    NTFY_SERVER="http://127.0.0.1:2586",
+    NTFY_TOKEN=os.environ.get("NTFY_TOKEN", ""),
+    NTFY_TOPIC_PREFIX="ping",
+)
+for f in ("smoke3.db",):
+    p = os.path.join(os.environ["LOCALAPPDATA"], "Temp", f)
+    if os.path.exists(p):
+        os.remove(p)
+
+from src import web_api  # noqa: E402
+from src.config import Config  # noqa: E402
+
+assert Config.NTFY_SERVER
+
+
+async def main():
+    # what the lifespan does at startup, minus the scheduler
+    from src import notify as notify_mod
+    from src.forecast_history import ForecastHistoryStore
+    from src.water_scraper_v3 import EnhancedWaterMonitorScraper
+
+    db_config = Config.get_database_config()
+    web_api.app_state["scraper"] = EnhancedWaterMonitorScraper(db_config)
+    store = ForecastHistoryStore(db_config["connection_string"], db_config["type"])
+    store.connect()
+    web_api.app_state["forecast_store"] = store
+    state = notify_mod.NotificationState(store.engine, store.db_type)
+    pub = notify_mod.NtfyPublisher(
+        Config.NTFY_SERVER, prefix=Config.NTFY_TOPIC_PREFIX, token=Config.NTFY_TOKEN
+    )
+    web_api.app_state["notify"] = (pub, state)
+
+    scraper = web_api.app_state["scraper"]
+    now = datetime.datetime.now().replace(minute=0, second=0, microsecond=0)
+
+    def seed(level_p1, level_p103, ts):
+        rows = [
+            {
+                "station_code": "P.1",
+                "station_id": 1,
+                "timestamp": ts,
+                "water_level": level_p1,
+                "discharge": 400.0,
+                "station_name_en": "Nawarat Bridge",
+                "station_name_th": "สะพานนวรัฐ",
+                "discharge_percent": 30.0,
+                "status": "active",
+            },
+            {
+                "station_code": "P.103",
+                "station_id": 2,
+                "timestamp": ts,
+                "water_level": level_p103,
+                "discharge": 300.0,
+                "station_name_en": "Ring Road 3",
+                "station_name_th": "วงแหวน 3",
+                "discharge_percent": 20.0,
+                "status": "active",
+            },
+        ]
+        scraper.db_adapter.save_measurements(rows)
+
+    def forecast(p):
+        with web_api.FORECAST_CACHE_LOCK:
+            web_api.FORECAST_CACHE["all"] = (
+                0,
+                [
+                    {
+                        "station_code": "P.1",
+                        "horizon_hours": 24,
+                        "p_warning": p,
+                        "predicted_max_level": 3.9,
+                        "source": "model",
+                    }
+                ],
+            )
+
+    def poll(topic):
+        out = []
+        for line in (
+            requests.get(f"{Config.NTFY_SERVER}/{topic}/json?poll=1", timeout=5)
+            .text.strip()
+            .splitlines()
+        ):
+            m = json.loads(line)
+            if m.get("event") == "message":
+                out.append(m.get("title") or m.get("message", "")[:40])
+        return out
+
+    # cycle 1: quiet
+    seed(1.6, 3.2, now - datetime.timedelta(hours=2))
+    forecast(0.02)
+    await web_api._notify_transitions()
+    # cycle 2: P.1 crosses warning, model outlook on
+    seed(3.75, 3.3, now - datetime.timedelta(hours=1))
+    forecast(0.7)
+    await web_api._notify_transitions()
+    # cycle 3: same state -> silence
+    seed(3.80, 3.3, now)
+    forecast(0.65)
+    await web_api._notify_transitions()
+
+    print("ping-p1-warning:", poll("ping-p1-warning"))
+    print("ping-warning:   ", poll("ping-warning"))
+    print("ping-p1-outlook:", poll("ping-p1-outlook"))
+    print("ping-p103-warning:", poll("ping-p103-warning"))
+    from sqlalchemy import text
+
+    with store.engine.connect() as c:
+        print(
+            "state table:",
+            c.execute(
+                text("SELECT key, state, value FROM notification_state ORDER BY key")
+            ).fetchall(),
+        )
+
+
+asyncio.run(main())

scripts/encode_password.py

@@ -1,57 +0,0 @@
-#!/usr/bin/env python3
-"""
-Password URL encoder for PostgreSQL connection strings
-"""
-
-import urllib.parse
-import sys
-
-def encode_password(password: str) -> str:
-    """URL encode a password for use in connection strings"""
-    return urllib.parse.quote(password, safe='')
-
-def build_connection_string(username: str, password: str, host: str, port: int, database: str) -> str:
-    """Build a properly encoded PostgreSQL connection string"""
-    encoded_password = encode_password(password)
-    return f"postgresql://{username}:{encoded_password}@{host}:{port}/{database}"
-
-def main():
-    print("PostgreSQL Password URL Encoder")
-    print("=" * 40)
-
-    if len(sys.argv) > 1:
-        # Password provided as argument
-        password = sys.argv[1]
-    else:
-        # Interactive mode
-        password = input("Enter your password: ")
-
-    encoded = encode_password(password)
-
-    print(f"\nOriginal password: {password}")
-    print(f"URL encoded: {encoded}")
-
-    # Optional: build full connection string
-    try:
-        build_full = input("\nBuild full connection string? (y/N): ").strip().lower() == 'y'
-    except (EOFError, KeyboardInterrupt):
-        print("\nDone!")
-        return
-
-    if build_full:
-        username = input("Username: ").strip()
-        host = input("Host: ").strip()
-        port = input("Port [5432]: ").strip() or "5432"
-        database = input("Database [water_monitoring]: ").strip() or "water_monitoring"
-
-        connection_string = build_connection_string(username, password, host, int(port), database)
-
-        print(f"\nComplete connection string:")
-        print(f"POSTGRES_CONNECTION_STRING={connection_string}")
-
-        print(f"\nAdd this to your .env file:")
-        print(f"DB_TYPE=postgresql")
-        print(f"POSTGRES_CONNECTION_STRING={connection_string}")
-
-if __name__ == "__main__":
-    main()

scripts/generate_badges.py

@@ -1,51 +0,0 @@
-#!/usr/bin/env python3
-"""
-Generate status badges for README.md
-"""
-
-import json
-import requests
-from datetime import datetime
-
-def generate_badge_url(label, message, color="brightgreen"):
-    """Generate a shields.io badge URL"""
-    return f"https://img.shields.io/badge/{label}-{message}-{color}"
-
-def generate_workflow_badge(repo_url, workflow_name, branch="main"):
-    """Generate workflow status badge"""
-    # For Gitea, you might need to adjust this based on your instance
-    badge_url = f"{repo_url}/actions/workflows/{workflow_name}/badge.svg?branch={branch}"
-    return badge_url
-
-def main():
-    """Generate badges for the project"""
-    repo_url = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor"
-    
-    badges = {
-        "CI/CD": generate_workflow_badge(repo_url, "ci.yml"),
-        "Security": generate_workflow_badge(repo_url, "security.yml"),
-        "Documentation": generate_workflow_badge(repo_url, "docs.yml"),
-        "Python": generate_badge_url("Python", "3.9%2B", "blue"),
-        "FastAPI": generate_badge_url("FastAPI", "0.104%2B", "green"),
-        "Docker": generate_badge_url("Docker", "Ready", "blue"),
-        "License": generate_badge_url("License", "MIT", "green"),
-        "Version": generate_badge_url("Version", "v3.1.3", "blue"),
-    }
-    
-    print("# Status Badges")
-    print()
-    print("Add these badges to your README.md:")
-    print()
-    
-    for name, url in badges.items():
-        print(f"[![{name}]({url})]({repo_url})")
-    
-    print()
-    print("# Markdown Format")
-    print()
-    
-    badge_line = " ".join([f"[![{name}]({url})]({repo_url})" for name, url in badges.items()])
-    print(badge_line)
-
-if __name__ == "__main__":
-    main()

scripts/init_git.bat

@@ -1,35 +0,0 @@
-@echo off
-REM Git initialization script for Northern Thailand Ping River Monitor
-
-echo 🏔️ Initializing Git repository for Northern Thailand Ping River Monitor
-
-REM Initialize git repository
-git init
-
-REM Add remote origin
-git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
-
-REM Add all files
-git add .
-
-REM Initial commit
-git commit -m "Initial commit: Northern Thailand Ping River Monitor v3.1.3
-
-Features:
-- Real-time water level monitoring for Ping River Basin
-- 16 monitoring stations from Chiang Dao to Nakhon Sawan
-- FastAPI web interface with station management
-- Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
-- Comprehensive monitoring and health checks
-- Docker deployment with Grafana integration
-- Production-ready architecture with CI/CD pipeline"
-
-echo ✅ Git repository initialized successfully!
-echo.
-echo Next steps:
-echo 1. Review and edit .env file with your configuration
-echo 2. Push to remote repository:
-echo    git push -u origin main
-echo.
-echo 3. Start the application:
-echo    python run.py --web-api

scripts/init_git.sh

@@ -1,89 +0,0 @@
-#!/bin/bash
-# Git initialization script for Northern Thailand Ping River Monitor
-
-echo "🏔️ Initializing Git repository for Northern Thailand Ping River Monitor"
-
-# Initialize git repository
-git init
-
-# Add remote origin
-git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
-
-# Create .gitignore if it doesn't exist
-if [ ! -f .gitignore ]; then
-    echo "Creating .gitignore file..."
-    cat > .gitignore << 'EOF'
-# Python
-__pycache__/
-*.py[cod]
-*.so
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-
-# Virtual environments
-.env
-.venv
-env/
-venv/
-ENV/
-
-# IDE
-.vscode/
-.idea/
-*.swp
-*.swo
-
-# Logs
-*.log
-logs/
-
-# Database files
-*.db
-*.sqlite
-*.sqlite3
-
-# OS
-.DS_Store
-Thumbs.db
-EOF
-fi
-
-# Add all files
-git add .
-
-# Initial commit
-git commit -m "Initial commit: Northern Thailand Ping River Monitor v3.1.3
-
-Features:
-- Real-time water level monitoring for Ping River Basin
-- 16 monitoring stations from Chiang Dao to Nakhon Sawan
-- FastAPI web interface with station management
-- Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
-- Comprehensive monitoring and health checks
-- Docker deployment with Grafana integration
-- Production-ready architecture with CI/CD pipeline"
-
-echo "✅ Git repository initialized successfully!"
-echo ""
-echo "Next steps:"
-echo "1. Review and edit .env file with your configuration"
-echo "2. Push to remote repository:"
-echo "   git push -u origin main"
-echo ""
-echo "3. Start the application:"
-echo "   make run-api"
-echo "   # or: python run.py --web-api"

scripts/install.sh

@@ -18,6 +18,7 @@ APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
 SERVICE_USER="${SERVICE_USER:-water-monitor}"
 SERVICE_GROUP="${SERVICE_GROUP:-${SERVICE_USER}}"
 SERVICE_NAME="water-monitor.service"
+RETRAIN_NAME="water-monitor-retrain"
 
 # Resolve the repo root (parent of this scripts/ directory).
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -72,11 +73,18 @@ if ! command -v uv >/dev/null 2>&1; then
 fi
 UV="$(command -v uv)"
 
-log "Creating virtualenv at ${APP_DIR}/venv"
+log "Syncing uv-managed virtualenv at ${APP_DIR}/.venv"
 cd "${APP_DIR}"
-# Named 'venv' (not uv's default .venv) to match the systemd unit's ExecStart.
-"${UV}" venv venv
-"${UV}" pip install --python venv/bin/python -r requirements.txt
+# ONE environment: uv sync owns .venv/ (from pyproject.toml + uv.lock, so the
+# ML extras such as scikit-learn/joblib are present) and both systemd units
+# run its interpreter directly. Never create a second env by another name --
+# a stale 'venv/' once coexisted here and broke manual retrains with
+# ModuleNotFoundError while the service itself ran fine.
+"${UV}" sync --python 3.11 --frozen
+if [ -d "${APP_DIR}/venv" ]; then
+    warn "Removing stale ${APP_DIR}/venv (superseded by .venv)"
+    rm -rf "${APP_DIR}/venv"
+fi
 
 # 4. Environment file ----------------------------------------------------------
 if [ ! -f "${APP_DIR}/.env" ]; then
@@ -100,11 +108,14 @@ if [ -f "${APP_DIR}/.env" ]; then
     chmod 0600 "${APP_DIR}/.env"
 fi
 
-# 6. Install and enable the systemd unit --------------------------------------
-log "Installing systemd unit"
+# 6. Install and enable the systemd units -------------------------------------
+log "Installing systemd units"
 install -m 0644 "${SCRIPT_DIR}/${SERVICE_NAME}" "/etc/systemd/system/${SERVICE_NAME}"
+install -m 0644 "${SCRIPT_DIR}/${RETRAIN_NAME}.service" "/etc/systemd/system/${RETRAIN_NAME}.service"
+install -m 0644 "${SCRIPT_DIR}/${RETRAIN_NAME}.timer" "/etc/systemd/system/${RETRAIN_NAME}.timer"
 systemctl daemon-reload
 systemctl enable "${SERVICE_NAME}"
+systemctl enable --now "${RETRAIN_NAME}.timer"
 
 log "Done."
 echo
@@ -112,3 +123,5 @@ echo "Next steps:"
 echo "  sudo systemctl start ${SERVICE_NAME}"
 echo "  systemctl status ${SERVICE_NAME}"
 echo "  sudo journalctl -u ${SERVICE_NAME} -f"
+echo "  systemctl list-timers ${RETRAIN_NAME}.timer   # monthly flood-model retrain"
+echo "  sudo systemctl start ${RETRAIN_NAME}.service   # retrain now"

scripts/install_ntfy.sh

@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+# Install ntfy (https://ntfy.sh) as the public notification server for the
+# Ping River Monitor. Run as root on the monitor VPS. Idempotent.
+#
+#   NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
+#
+# What it does:
+#   - installs the ntfy .deb from the official GitHub release (single Go
+#     binary, ~30 MB RSS, sqlite message cache)
+#   - writes /etc/ntfy/server.yml: listens on the Tailscale address only
+#     (the reverse proxy is another VPS on the tailnet; nothing is exposed
+#     on a public interface), anonymous READ on all topics, WRITE only with
+#     a token. Override with NTFY_LISTEN=host:port.
+#   - creates the `monitor` publishing user + token, writes NTFY_SERVER /
+#     NTFY_TOKEN into /opt/thailand-water-monitor/.env if not present
+#
+# Reverse proxy (on the Caddy VPS, over Tailscale):
+#     ntfy.buildfor.life {
+#         reverse_proxy <this host's tailscale ip>:2586
+#     }
+# Caddy passes websockets and keeps long-poll connections open by default;
+# subscribers hold one open. ntfy runs with behind-proxy: true so rate
+# limits key on X-Forwarded-For, not on the proxy's address.
+set -euo pipefail
+
+NTFY_DOMAIN="${NTFY_DOMAIN:?set NTFY_DOMAIN, e.g. ntfy.buildfor.life}"
+NTFY_VERSION="${NTFY_VERSION:-2.28.0}"
+MONITOR_DIR="${MONITOR_DIR:-/opt/thailand-water-monitor}"
+TS_IP="$(tailscale ip -4 2>/dev/null | head -1 || true)"
+LISTEN="${NTFY_LISTEN:-${TS_IP:-127.0.0.1}:2586}"
+echo "ntfy will listen on ${LISTEN}"
+
+if ! command -v ntfy >/dev/null || [[ "$(ntfy --version 2>/dev/null | awk '{print $3}')" != "$NTFY_VERSION" ]]; then
+    tmp=$(mktemp -d)
+    curl -fsSL -o "$tmp/ntfy.deb" \
+        "https://github.com/binwiederhier/ntfy/releases/download/v${NTFY_VERSION}/ntfy_${NTFY_VERSION}_linux_amd64.deb"
+    dpkg -i "$tmp/ntfy.deb"
+    rm -rf "$tmp"
+fi
+
+install -d -m 755 /var/cache/ntfy /var/lib/ntfy
+cat > /etc/ntfy/server.yml <<EOF
+# Ping River Monitor notification server. Managed by scripts/install_ntfy.sh.
+base-url: "https://${NTFY_DOMAIN}"
+listen-http: "${LISTEN}"
+behind-proxy: true
+
+# Messages are kept so a phone that was offline still gets the crossing.
+cache-file: "/var/cache/ntfy/cache.db"
+cache-duration: "72h"
+
+# Everyone may subscribe; only the monitor (token) may publish.
+auth-file: "/var/lib/ntfy/user.db"
+auth-default-access: "read-only"
+
+# The monitor publishes a handful of messages per flood; be strict with
+# everything else so the box cannot be used as a free relay.
+visitor-request-limit-burst: 30
+visitor-request-limit-replenish: "10s"
+visitor-subscription-limit: 60
+visitor-message-daily-limit: 200
+attachment-cache-dir: ""
+enable-signup: false
+enable-login: false
+enable-metrics: false
+EOF
+
+systemctl enable --now ntfy
+systemctl restart ntfy
+sleep 1
+curl -fsS "http://${LISTEN}/v1/health" >/dev/null && echo "ntfy up on ${LISTEN}"
+
+# Publishing identity for the monitor
+if ! ntfy user list 2>/dev/null | grep -q '^user monitor (role'; then
+    NTFY_PASSWORD="$(openssl rand -base64 24)" ntfy user add --role=user monitor
+fi
+ntfy access monitor 'ping-*' write-only >/dev/null
+# 'ping-*' read stays anonymous via auth-default-access
+
+token=$(ntfy token list monitor 2>/dev/null | awk '/^- tk_/{print $2; exit}')  # '- tk_xxx (label), ...'
+if [[ -z "$token" ]]; then
+    token=$(ntfy token add --label "water-monitor" monitor | grep -oE 'tk_[A-Za-z0-9]+' | head -1)  # 'token tk_xxx created for user monitor'
+fi
+
+env_file="${MONITOR_DIR}/.env"
+if [[ -f "$env_file" ]] && ! grep -q '^NTFY_SERVER=' "$env_file"; then
+    {
+        echo ""
+        echo "# ntfy public notifications (scripts/install_ntfy.sh)"
+        echo "NTFY_SERVER=https://${NTFY_DOMAIN}"
+        echo "NTFY_PUBLISH_URL=http://${LISTEN}"
+        echo "NTFY_TOPIC_PREFIX=ping"
+        echo "NTFY_TOKEN=${token}"
+    } >> "$env_file"
+    echo "wrote NTFY_* to ${env_file}; restart water-monitor to enable"
+else
+    echo "NTFY_TOKEN=${token}"
+fi
+
+echo
+echo "Subscribe test (anonymous read):  curl -s 'http://${LISTEN}/ping-status/json?poll=1'"
+echo "Publish test (needs token):       curl -s -H 'Authorization: Bearer ${token}' -d 'hello' http://${LISTEN}/ping-status"

scripts/migrate_geolocation.py

@@ -1,294 +0,0 @@
-#!/usr/bin/env python3
-"""
-Migration script to add geolocation columns to existing water monitoring database
-"""
-
-import os
-import sys
-import sqlite3
-import logging
-from typing import Dict, Any
-
-# Configure logging
-logging.basicConfig(
-    level=logging.INFO,
-    format='%(asctime)s - %(levelname)s - %(message)s'
-)
-
-def migrate_sqlite(db_path: str = 'water_monitoring.db') -> bool:
-    """Migrate SQLite database to add geolocation columns"""
-    try:
-        logging.info(f"Migrating SQLite database: {db_path}")
-        
-        # Connect to database
-        conn = sqlite3.connect(db_path)
-        cursor = conn.cursor()
-        
-        # Check if columns already exist
-        cursor.execute("PRAGMA table_info(stations)")
-        columns = [column[1] for column in cursor.fetchall()]
-        
-        logging.info(f"Current columns in stations table: {columns}")
-        
-        # Add columns if they don't exist
-        columns_added = []
-        
-        if 'latitude' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN latitude REAL")
-            columns_added.append('latitude')
-            logging.info("Added latitude column")
-        
-        if 'longitude' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN longitude REAL")
-            columns_added.append('longitude')
-            logging.info("Added longitude column")
-        
-        if 'geohash' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN geohash TEXT")
-            columns_added.append('geohash')
-            logging.info("Added geohash column")
-        
-        if columns_added:
-            # Update P.1 station with sample geolocation data
-            cursor.execute("""
-                UPDATE stations 
-                SET latitude = 15.6944, longitude = 100.2028, geohash = 'w5q6uuhvfcfp25'
-                WHERE station_code = 'P.1'
-            """)
-            
-            # Commit changes
-            conn.commit()
-            logging.info(f"Successfully added columns: {', '.join(columns_added)}")
-            logging.info("Updated P.1 station with sample geolocation data")
-        else:
-            logging.info("All geolocation columns already exist")
-        
-        # Verify the changes
-        cursor.execute("SELECT station_code, latitude, longitude, geohash FROM stations WHERE station_code = 'P.1'")
-        result = cursor.fetchone()
-        if result:
-            logging.info(f"P.1 station geolocation: {result}")
-        
-        conn.close()
-        return True
-        
-    except Exception as e:
-        logging.error(f"Error migrating SQLite database: {e}")
-        return False
-
-def migrate_postgresql(connection_string: str) -> bool:
-    """Migrate PostgreSQL database to add geolocation columns"""
-    try:
-        import psycopg2
-        from urllib.parse import urlparse
-        
-        logging.info("Migrating PostgreSQL database")
-        
-        # Parse connection string
-        parsed = urlparse(connection_string)
-        
-        # Connect to database
-        conn = psycopg2.connect(
-            host=parsed.hostname,
-            port=parsed.port or 5432,
-            database=parsed.path[1:],  # Remove leading slash
-            user=parsed.username,
-            password=parsed.password
-        )
-        cursor = conn.cursor()
-        
-        # Check if columns exist
-        cursor.execute("""
-            SELECT column_name 
-            FROM information_schema.columns 
-            WHERE table_name = 'stations'
-        """)
-        columns = [row[0] for row in cursor.fetchall()]
-        
-        logging.info(f"Current columns in stations table: {columns}")
-        
-        # Add columns if they don't exist
-        columns_added = []
-        
-        if 'latitude' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN latitude DECIMAL(10,8)")
-            columns_added.append('latitude')
-            logging.info("Added latitude column")
-        
-        if 'longitude' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN longitude DECIMAL(11,8)")
-            columns_added.append('longitude')
-            logging.info("Added longitude column")
-        
-        if 'geohash' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN geohash VARCHAR(20)")
-            columns_added.append('geohash')
-            logging.info("Added geohash column")
-        
-        if columns_added:
-            # Update P.1 station with sample geolocation data
-            cursor.execute("""
-                UPDATE stations 
-                SET latitude = 15.6944, longitude = 100.2028, geohash = 'w5q6uuhvfcfp25'
-                WHERE station_code = 'P.1'
-            """)
-            
-            # Commit changes
-            conn.commit()
-            logging.info(f"Successfully added columns: {', '.join(columns_added)}")
-            logging.info("Updated P.1 station with sample geolocation data")
-        else:
-            logging.info("All geolocation columns already exist")
-        
-        conn.close()
-        return True
-        
-    except ImportError:
-        logging.error("psycopg2 not installed. Run: pip install psycopg2-binary")
-        return False
-    except Exception as e:
-        logging.error(f"Error migrating PostgreSQL database: {e}")
-        return False
-
-def migrate_mysql(connection_string: str) -> bool:
-    """Migrate MySQL database to add geolocation columns"""
-    try:
-        import pymysql
-        from urllib.parse import urlparse
-        
-        logging.info("Migrating MySQL database")
-        
-        # Parse connection string
-        parsed = urlparse(connection_string)
-        
-        # Connect to database
-        conn = pymysql.connect(
-            host=parsed.hostname,
-            port=parsed.port or 3306,
-            database=parsed.path[1:],  # Remove leading slash
-            user=parsed.username,
-            password=parsed.password
-        )
-        cursor = conn.cursor()
-        
-        # Check if columns exist
-        cursor.execute("DESCRIBE stations")
-        columns = [row[0] for row in cursor.fetchall()]
-        
-        logging.info(f"Current columns in stations table: {columns}")
-        
-        # Add columns if they don't exist
-        columns_added = []
-        
-        if 'latitude' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN latitude DECIMAL(10,8)")
-            columns_added.append('latitude')
-            logging.info("Added latitude column")
-        
-        if 'longitude' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN longitude DECIMAL(11,8)")
-            columns_added.append('longitude')
-            logging.info("Added longitude column")
-        
-        if 'geohash' not in columns:
-            cursor.execute("ALTER TABLE stations ADD COLUMN geohash VARCHAR(20)")
-            columns_added.append('geohash')
-            logging.info("Added geohash column")
-        
-        if columns_added:
-            # Update P.1 station with sample geolocation data
-            cursor.execute("""
-                UPDATE stations 
-                SET latitude = 15.6944, longitude = 100.2028, geohash = 'w5q6uuhvfcfp25'
-                WHERE station_code = 'P.1'
-            """)
-            
-            # Commit changes
-            conn.commit()
-            logging.info(f"Successfully added columns: {', '.join(columns_added)}")
-            logging.info("Updated P.1 station with sample geolocation data")
-        else:
-            logging.info("All geolocation columns already exist")
-        
-        conn.close()
-        return True
-        
-    except ImportError:
-        logging.error("pymysql not installed. Run: pip install pymysql")
-        return False
-    except Exception as e:
-        logging.error(f"Error migrating MySQL database: {e}")
-        return False
-
-def load_config_from_env() -> Dict[str, Any]:
-    """Load database configuration from environment variables"""
-    db_type = os.getenv('DB_TYPE', 'sqlite').lower()
-    
-    if db_type == 'postgresql':
-        return {
-            'type': 'postgresql',
-            'connection_string': os.getenv('POSTGRES_CONNECTION_STRING',
-                'postgresql://postgres:password@localhost/water_monitoring')
-        }
-    elif db_type == 'mysql':
-        return {
-            'type': 'mysql',
-            'connection_string': os.getenv('MYSQL_CONNECTION_STRING', 
-                'mysql://root:password@localhost/water_monitoring')
-        }
-    elif db_type == 'victoriametrics':
-        logging.info("VictoriaMetrics doesn't require schema migration")
-        return {'type': 'victoriametrics'}
-    elif db_type == 'influxdb':
-        logging.info("InfluxDB doesn't require schema migration")
-        return {'type': 'influxdb'}
-    else:
-        # Default to SQLite
-        return {
-            'type': 'sqlite',
-            'db_path': os.getenv('SQLITE_DB_PATH', 'water_monitoring.db')
-        }
-
-def main():
-    """Main migration function"""
-    logging.info("Starting geolocation column migration...")
-    
-    # Load configuration
-    config = load_config_from_env()
-    db_type = config['type']
-    
-    logging.info(f"Detected database type: {db_type.upper()}")
-    
-    success = False
-    
-    if db_type == 'sqlite':
-        db_path = config.get('db_path', 'water_monitoring.db')
-        if not os.path.exists(db_path):
-            logging.error(f"Database file not found: {db_path}")
-            sys.exit(1)
-        success = migrate_sqlite(db_path)
-        
-    elif db_type == 'postgresql':
-        success = migrate_postgresql(config['connection_string'])
-        
-    elif db_type == 'mysql':
-        success = migrate_mysql(config['connection_string'])
-        
-    elif db_type in ['victoriametrics', 'influxdb']:
-        logging.info(f"{db_type.upper()} doesn't require schema migration")
-        success = True
-        
-    else:
-        logging.error(f"Unsupported database type: {db_type}")
-        sys.exit(1)
-    
-    if success:
-        logging.info("✅ Migration completed successfully!")
-        logging.info("You can now restart your water monitoring application")
-        logging.info("The system will automatically use the new geolocation columns")
-    else:
-        logging.error("❌ Migration failed!")
-        sys.exit(1)
-
-if __name__ == "__main__":
-    main()

scripts/retrain.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+#
+# Retrain the flood forecast models safely. Run by water-monitor-retrain.timer
+# (monthly) or by hand:  sudo systemctl start water-monitor-retrain.service
+#
+# Why a script rather than ExecStart=train_flood_model.py:
+#   * train.py writes each station's bundle straight into models/ over ~12 min,
+#     and the API's hourly precompute reloads bundles by mtime. Training into
+#     a staging dir and mv-ing (atomic on one filesystem) means the API never
+#     sees a half-written joblib file or a mixed old/new set.
+#   * A run that produced gauge-only (v2) bundles, or trained too few stations,
+#     must NOT replace the deployed models. train.py already aborts on a
+#     missing rain series; this script re-checks the written metrics anyway.
+#   * No API restart is needed: predict.py reloads changed bundles on the next
+#     precompute (every scrape cycle, hourly), so the new models are live
+#     within an hour. Restart manually if you want them live immediately.
+#
+# Exit codes: 0 ok, 2 training refused (see log), 3 verification failed.
+set -euo pipefail
+
+APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
+PYTHON="${PYTHON:-${APP_DIR}/.venv/bin/python}"
+MODELS_DIR="${APP_DIR}/models"
+STAGE_DIR="${MODELS_DIR}/.staging"
+# P.4A is NOT_TRAINABLE by design (17% fill); 15 of 16 is the normal outcome.
+MIN_TRAINED="${MIN_TRAINED:-14}"
+EXPECT_VERSION_PREFIX="${EXPECT_VERSION_PREFIX:-hgb-v3+}"
+
+log() { printf '%s retrain: %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }
+
+cd "${APP_DIR}"
+[ -x "${PYTHON}" ] || { log "no interpreter at ${PYTHON} (run uv sync)"; exit 3; }
+
+rm -rf "${STAGE_DIR}"
+mkdir -p "${STAGE_DIR}"
+log "training into ${STAGE_DIR} (python=${PYTHON}, OMP_NUM_THREADS=${OMP_NUM_THREADS:-unset})"
+
+# train_flood_model.py exits 2 on a missing rain series (RainUnavailableError)
+# instead of silently writing v2 bundles -- propagate that unchanged.
+set +e
+"${PYTHON}" scripts/train_flood_model.py --stations all --models-dir "${STAGE_DIR}" "$@"
+rc=$?
+set -e
+if [ "${rc}" -ne 0 ]; then
+    log "training failed (exit ${rc}); deployed models untouched"
+    rm -rf "${STAGE_DIR}"
+    exit "${rc}"
+fi
+
+# Verify before promoting. Reads metrics.json from the stage dir.
+VERSION="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
+import json, sys
+m = json.load(open(sys.argv[1]))
+print(m["model_version"])
+PY
+)"
+TRAINED="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
+import json, sys
+m = json.load(open(sys.argv[1]))
+print(sum(1 for s in m["stations"].values() if s.get("status") == "trained"))
+PY
+)"
+log "staged model_version=${VERSION} trained_stations=${TRAINED}"
+
+case "${VERSION}" in
+    "${EXPECT_VERSION_PREFIX}"*) ;;
+    *)
+        log "REFUSING to deploy: version '${VERSION}' does not start with '${EXPECT_VERSION_PREFIX}'"
+        rm -rf "${STAGE_DIR}"
+        exit 3
+        ;;
+esac
+if [ "${TRAINED}" -lt "${MIN_TRAINED}" ]; then
+    log "REFUSING to deploy: only ${TRAINED} stations trained (< ${MIN_TRAINED})"
+    rm -rf "${STAGE_DIR}"
+    exit 3
+fi
+
+# Promote: per-file rename is atomic; readers see either the old or the new
+# bundle, never a partial one. Keep one previous generation for rollback.
+mkdir -p "${MODELS_DIR}/.previous"
+for f in "${STAGE_DIR}"/flood_*.joblib "${STAGE_DIR}/metrics.json"; do
+    name="$(basename "${f}")"
+    if [ -f "${MODELS_DIR}/${name}" ]; then
+        mv -f "${MODELS_DIR}/${name}" "${MODELS_DIR}/.previous/${name}"
+    fi
+    mv -f "${f}" "${MODELS_DIR}/${name}"
+done
+rm -rf "${STAGE_DIR}"
+log "deployed ${VERSION} (${TRAINED} stations); previous generation in models/.previous. The API picks it up on its next hourly precompute."

scripts/summarize_eval.py

@@ -0,0 +1,62 @@
+"""Summarise rolling-origin harness output side by side.
+
+Usage:
+    uv run python scripts/summarize_eval.py models/eval_2026-09-12.json [more.json ...]
+
+Aggregates each (station, variant) across folds: mean MAE, mean flood-regime
+MAE, mean Brier, total false-alarm episodes, and every warning event with its
+first-alert lead and the 24 h-ahead peak error -- the operational numbers that
+decide whether a variant ships.
+"""
+
+import json
+import statistics
+import sys
+from collections import OrderedDict
+
+
+def summarize(paths):
+    for path in paths:
+        results = json.load(open(path, encoding="utf-8"))
+        print(f"\n##### {path}")
+        for station in results:
+            print(f"\n=== {station['station']} (warn {station['warn_thr']:.2f} m) ===")
+            agg = OrderedDict()
+            for fold in station["folds"]:
+                for name, m in fold["variants"].items():
+                    a = agg.setdefault(
+                        name, {"mae": [], "mae_hi": [], "brier": [], "fa": 0, "events": []}
+                    )
+                    a["mae"].append(m["mae"])
+                    if m.get("mae_above_2p5") is not None:
+                        a["mae_hi"].append(m["mae_above_2p5"])
+                    if m.get("brier_warn") is not None:
+                        a["brier"].append(m["brier_warn"])
+                    a["fa"] += m["false_alarm_episodes"]
+                    for e in m["events"]:
+                        err = (
+                            None
+                            if e["peak_pred_24h_before"] is None
+                            else e["peak_pred_24h_before"] - e["peak_level"]
+                        )
+                        a["events"].append((fold["year"], e["crossing"][:10], e["lead_h"], e["peak_level"], err))
+            print(f"{'variant':22} {'MAE':>6} {'MAE_hi':>7} {'Brier':>7} {'FA':>3}  events: year crossing lead_h peak(err24h)")
+            for name, a in agg.items():
+                ev = "  ".join(
+                    f"{y} {d} {'—' if l is None else format(l, '+.0f')}h {p:.2f}({'—' if err is None else format(err, '+.2f')})"
+                    for y, d, l, p, err in a["events"]
+                )
+                leads = [l for *_, l, _, _ in a["events"] if l is not None]
+                print(
+                    f"{name:22} {statistics.mean(a['mae']):6.3f} "
+                    f"{statistics.mean(a['mae_hi']) if a['mae_hi'] else float('nan'):7.3f} "
+                    f"{statistics.mean(a['brier']) if a['brier'] else float('nan'):7.4f} "
+                    f"{a['fa']:>3}  {ev}"
+                )
+                if leads:
+                    print(f"{'':22} lead: mean {statistics.mean(leads):+.1f} h, min {min(leads):+.0f} h, "
+                          f"missed {sum(1 for *_, l, _, _ in a['events'] if l is None)}/{len(a['events'])}")
+
+
+if __name__ == "__main__":
+    summarize(sys.argv[1:] or ["models/eval_variants.json"])

scripts/train_flood_model.py

@@ -11,7 +11,7 @@ import sys
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
 
-from src.ml.train import main
+from src.ml.train import cli
 
 if __name__ == "__main__":
-    main()
+    raise SystemExit(cli())

scripts/water-monitor-retrain.service

@@ -0,0 +1,39 @@
+[Unit]
+Description=Retrain the Ping River flood forecast models
+Documentation=https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/-/blob/master/docs/FLOOD_FORECASTING.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=water-monitor
+Group=water-monitor
+WorkingDirectory=/opt/thailand-water-monitor
+EnvironmentFile=/opt/thailand-water-monitor/.env
+# Same interpreter as water-monitor.service -- the uv-managed .venv.
+# scripts/retrain.sh trains into models/.staging, refuses to promote anything
+# that is not a rain-enabled (hgb-v3) set covering the expected stations, then
+# renames the bundles into place. The API reloads them on its next hourly
+# precompute; no restart, so a failed run leaves the old models serving.
+ExecStart=/bin/bash /opt/thailand-water-monitor/scripts/retrain.sh
+# HistGradientBoosting is CPU-bound; cap threads so training cannot starve
+# the API (docs/FLOOD_FORECASTING.md section 6 measured 4 as the sweet spot).
+Environment=OMP_NUM_THREADS=4
+Environment=PYTHONPATH=/opt/thailand-water-monitor
+Environment=PYTHONUNBUFFERED=1
+Nice=15
+IOSchedulingClass=idle
+# 15 stations at ~50 s each plus data load: 12 min observed on 2026-09-12.
+TimeoutStartSec=45min
+
+# Same sandbox as the API unit.
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ReadWritePaths=/opt/thailand-water-monitor
+CapabilityBoundingSet=
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=water-monitor-retrain

scripts/water-monitor-retrain.timer

@@ -0,0 +1,17 @@
+[Unit]
+Description=Monthly flood-model retrain (docs/FLOOD_FORECASTING.md section 7)
+
+[Timer]
+# Policy: at minimum once pre-monsoon (May-June), monthly through the season
+# (July-November), and after any major flood. A retrain costs ~12 min and RAM
+# peaks ~300 MB, so running it every month all year is cheaper than remembering
+# which months matter. 1st of the month, 03:30 server-local -- between the
+# hourly scrapes and outside Thai daytime traffic.
+OnCalendar=*-*-01 03:30:00
+# Catch up if the box was off at the scheduled time.
+Persistent=true
+RandomizedDelaySec=20min
+Unit=water-monitor-retrain.service
+
+[Install]
+WantedBy=timers.target

scripts/water-monitor.service

@@ -9,17 +9,19 @@ Type=simple
 User=water-monitor
 Group=water-monitor
 WorkingDirectory=/opt/thailand-water-monitor
-ExecStart=/opt/thailand-water-monitor/venv/bin/python src/water_scraper_v3.py
+# The uv-managed env (uv sync -> .venv). Same interpreter for water-monitor-retrain.service.
+ExecStart=/opt/thailand-water-monitor/.venv/bin/python run.py --web-api
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=60
 TimeoutStopSec=30
 
-# Environment variables
-Environment=DB_TYPE=victoriametrics
-Environment=VM_HOST=localhost
-Environment=VM_PORT=8428
+# DB_TYPE / POSTGRES_CONNECTION_STRING / MATRIX_* come from the .env file.
+EnvironmentFile=/opt/thailand-water-monitor/.env
 Environment=PYTHONPATH=/opt/thailand-water-monitor
+# Serving path is latency-bound; single-threaded BLAS is 2.6x faster per call
+# (docs/FLOOD_FORECASTING.md section 6). Training sets its own value.
+Environment=OMP_NUM_THREADS=1
 Environment=PYTHONUNBUFFERED=1
 
 # Security settings

setup.py.backup

@@ -1,106 +0,0 @@
-#!/usr/bin/env python3
-"""
-Setup script for Northern Thailand Ping River Monitor
-"""
-
-from setuptools import setup, find_packages
-import os
-
-# Read the README file
-with open("README.md", "r", encoding="utf-8") as fh:
-    long_description = fh.read()
-
-# Read requirements
-try:
-    with open("requirements.txt", "r", encoding="utf-8") as fh:
-        requirements = [line.strip() for line in fh if line.strip() and not line.startswith("#")]
-except FileNotFoundError:
-    # Fallback to minimal requirements if file not found
-    requirements = [
-        "requests>=2.31.0",
-        "schedule>=1.2.0",
-        "pandas>=2.1.0",
-        "fastapi>=0.104.0",
-        "uvicorn>=0.24.0",
-    ]
-
-# Extract core requirements (exclude dev dependencies)
-core_requirements = []
-for req in requirements:
-    if not any(dev_keyword in req.lower() for dev_keyword in ['pytest', 'black', 'flake8', 'mypy', 'sphinx']):
-        core_requirements.append(req)
-
-setup(
-    name="northern-thailand-ping-river-monitor",
-    version="3.1.3",
-    author="Ping River Monitor Team",
-    author_email="contact@example.com",
-    description="Real-time water level monitoring system for the Ping River Basin in Northern Thailand",
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    url="https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor",
-    project_urls={
-        "Bug Tracker": "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues",
-        "Documentation": "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/wiki",
-        "Source Code": "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor",
-    },
-    packages=find_packages(),
-    classifiers=[
-        "Development Status :: 4 - Beta",
-        "Intended Audience :: Science/Research",
-        "Intended Audience :: System Administrators",
-        "Topic :: Scientific/Engineering :: Hydrology",
-        "Topic :: System :: Monitoring",
-        "License :: OSI Approved :: MIT License",
-        "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.9",
-        "Programming Language :: Python :: 3.10",
-        "Programming Language :: Python :: 3.11",
-        "Programming Language :: Python :: 3.12",
-        "Operating System :: OS Independent",
-        "Environment :: Web Environment",
-        "Framework :: FastAPI",
-    ],
-    python_requires=">=3.9",
-    install_requires=core_requirements,
-    extras_require={
-        "dev": [
-            "pytest>=7.4.3",
-            "pytest-cov>=4.1.0",
-            "black>=23.11.0",
-            "flake8>=6.1.0",
-            "mypy>=1.7.1",
-            "pre-commit>=3.5.0",
-        ],
-        "docs": [
-            "sphinx>=7.2.6",
-            "sphinx-rtd-theme>=1.3.0",
-        ],
-        "all": [
-            "influxdb>=5.3.1",
-            "pymysql>=1.1.0",
-            "psycopg2-binary>=2.9.9",
-        ],
-    },
-    entry_points={
-        "console_scripts": [
-            "ping-river-monitor=src.main:main",
-            "ping-river-api=src.web_api:main",
-        ],
-    },
-    include_package_data=True,
-    package_data={
-        "src": ["*.py"],
-    },
-    keywords=[
-        "water monitoring",
-        "hydrology",
-        "thailand",
-        "ping river",
-        "environmental monitoring",
-        "time series",
-        "fastapi",
-        "real-time data",
-    ],
-    zip_safe=False,
-)

src/__init__.py

@@ -12,9 +12,13 @@ __description__ = "Northern Thailand Ping River Monitoring System"
 
 from .config import Config
 from .database_adapters import DatabaseAdapter, create_database_adapter
-from .exceptions import (APIConnectionError, ConfigurationError,
-                         DatabaseConnectionError, DataValidationError,
-                         WaterMonitorException)
+from .exceptions import (
+    APIConnectionError,
+    ConfigurationError,
+    DatabaseConnectionError,
+    DataValidationError,
+    WaterMonitorException,
+)
 from .models import DatabaseConfig, StationInfo, WaterMeasurement
 from .water_scraper_v3 import EnhancedWaterMonitorScraper
 

src/config.py

@@ -38,6 +38,17 @@ class Config:
     TARGET_URL = "https://hyd-app-db.rid.go.th/hydro1h.html"
     API_URL = "https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx"
     THAIWATER_API_KEY = os.getenv("THAIWATER_API_KEY")
+
+    # Public flood notifications (ntfy). Off unless NTFY_SERVER is set.
+    # NTFY_SERVER is what subscribers use (public https URL, shown on the
+    # dashboard). NTFY_PUBLISH_URL is where the monitor POSTs; defaults to
+    # NTFY_SERVER, set it to http://127.0.0.1:2586 when ntfy runs on the same
+    # host so publishing never depends on DNS/proxy/tunnel being up.
+    NTFY_SERVER = os.getenv("NTFY_SERVER", "").strip()
+    NTFY_PUBLISH_URL = os.getenv("NTFY_PUBLISH_URL", "").strip() or NTFY_SERVER
+    NTFY_TOPIC_PREFIX = os.getenv("NTFY_TOPIC_PREFIX", "ping").strip()
+    NTFY_TOKEN = os.getenv("NTFY_TOKEN", "").strip()  # publish token if ACL enabled
+    PUBLIC_URL = os.getenv("PUBLIC_URL", "https://water.buildfor.life/").strip()
     REQUEST_TIMEOUT = int(os.getenv("REQUEST_TIMEOUT", "30"))
     USER_AGENT = (
         "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "

src/database_adapters.py

@@ -139,12 +139,16 @@ class InfluxDBAdapter(DatabaseAdapter):
                     "time": measurement["timestamp"].isoformat(),
                     "fields": {
                         "water_level": float(measurement["water_level"]),
-                        "discharge": float(measurement["discharge"])
-                        if measurement.get("discharge") is not None
-                        else None,
-                        "discharge_percent": float(measurement["discharge_percent"])
-                        if measurement.get("discharge_percent")
-                        else None,
+                        "discharge": (
+                            float(measurement["discharge"])
+                            if measurement.get("discharge") is not None
+                            else None
+                        ),
+                        "discharge_percent": (
+                            float(measurement["discharge_percent"])
+                            if measurement.get("discharge_percent")
+                            else None
+                        ),
                     },
                 }
                 points.append(point)
@@ -551,13 +555,13 @@ class SQLAdapter(DatabaseAdapter):
                             "station_code": row[1],
                             "station_name_en": row[2],
                             "station_name_th": row[3],
-                            "water_level": float(row[4])
-                            if row[4] is not None
-                            else None,
+                            "water_level": (
+                                float(row[4]) if row[4] is not None else None
+                            ),
                             "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
-                            if row[6] is not None
-                            else None,
+                            "discharge_percent": (
+                                float(row[6]) if row[6] is not None else None
+                            ),
                             "status": row[7],
                         }
                     )
@@ -611,13 +615,13 @@ class SQLAdapter(DatabaseAdapter):
                             "station_code": row[1],
                             "station_name_en": row[2],
                             "station_name_th": row[3],
-                            "water_level": float(row[4])
-                            if row[4] is not None
-                            else None,
+                            "water_level": (
+                                float(row[4]) if row[4] is not None else None
+                            ),
                             "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
-                            if row[6] is not None
-                            else None,
+                            "discharge_percent": (
+                                float(row[6]) if row[6] is not None else None
+                            ),
                             "status": row[7],
                         }
                     )
@@ -666,13 +670,13 @@ class SQLAdapter(DatabaseAdapter):
                             "station_id": row[1],
                             "station_code": row[2] or f"Station_{row[1]}",
                             "station_name_th": row[3] or f"Station {row[1]}",
-                            "water_level": float(row[4])
-                            if row[4] is not None
-                            else None,
+                            "water_level": (
+                                float(row[4]) if row[4] is not None else None
+                            ),
                             "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
-                            if row[6] is not None
-                            else None,
+                            "discharge_percent": (
+                                float(row[6]) if row[6] is not None else None
+                            ),
                             "status": row[7],
                         }
                     )
@@ -767,9 +771,7 @@ class SQLAdapter(DatabaseAdapter):
             return hours_by_day
 
         except Exception as e:
-            logging.error(
-                f"Error querying {self.db_type.upper()} recorded hours: {e}"
-            )
+            logging.error(f"Error querying {self.db_type.upper()} recorded hours: {e}")
             return None
 
     def get_database_stats(self) -> Optional[Dict]:
@@ -814,9 +816,7 @@ class SQLAdapter(DatabaseAdapter):
             # the DISTINCT day-hour slots and coverage cannot exceed 100%
             first_slot = first_ts.replace(minute=0, second=0, microsecond=0)
             last_slot = last_ts.replace(minute=0, second=0, microsecond=0)
-            expected_hours = (
-                int((last_slot - first_slot).total_seconds() // 3600) + 1
-            )
+            expected_hours = int((last_slot - first_slot).total_seconds() // 3600) + 1
             recorded_hours = int(row[4])
             coverage_percent = round(100.0 * recorded_hours / expected_hours, 1)
 

src/health_check.py

@@ -125,9 +125,9 @@ class DatabaseHealthCheck(HealthCheck):
                 "message": "Database connection OK",
                 "details": {
                     "latest_data_count": len(latest_data),
-                    "latest_timestamp": str(latest_data[0].get("timestamp"))
-                    if latest_data
-                    else None,
+                    "latest_timestamp": (
+                        str(latest_data[0].get("timestamp")) if latest_data else None
+                    ),
                 },
             }
 

src/hii_backfill.py

@@ -17,9 +17,9 @@ import time
 from typing import Dict, List, Optional
 
 from .hii_collector import (
+    PING_BASIN_CODE,
     HiiClient,
     HiiStore,
-    PING_BASIN_CODE,
     _parse_datetime,
     _to_float,
 )
@@ -145,9 +145,7 @@ def backfill(
                 station_rows += store.save_waterlevel_history(sid, rows)
             except Exception as e:
                 totals["errors"] += 1
-                logger.warning(
-                    f"{label}: {chunk_start}..{chunk_end} failed: {e}"
-                )
+                logger.warning(f"{label}: {chunk_start}..{chunk_end} failed: {e}")
             time.sleep(sleep_seconds)
         totals["rows"] += station_rows
         logger.info(f"{label} (id {sid}): {station_rows} rows saved")

src/hii_collector.py

@@ -66,9 +66,7 @@ def rid_code_from_oldcode(oldcode: Optional[str]) -> Optional[str]:
     return match.group(1) if match else None
 
 
-def parse_rain_records(
-    payload: Dict, basin_code: int = PING_BASIN_CODE
-) -> List[Dict]:
+def parse_rain_records(payload: Dict, basin_code: int = PING_BASIN_CODE) -> List[Dict]:
     """Extract per-station rainfall rows from a rain_24h payload."""
     records = []
     for row in payload.get("data") or []:
@@ -186,9 +184,7 @@ class HiiStore:
     def __init__(self, connection_string: str, db_type: str):
         self.db_type = db_type.lower()
         if self.db_type not in ("sqlite", "postgresql", "mysql"):
-            raise ValueError(
-                f"HII collection requires a SQL database, got '{db_type}'"
-            )
+            raise ValueError(f"HII collection requires a SQL database, got '{db_type}'")
         self.connection_string = connection_string
         self.engine = None
 
@@ -400,9 +396,7 @@ class HiiStore:
         from sqlalchemy import text
 
         now = datetime.datetime.now()
-        station_sql = self._upsert(
-            station_table, ["id"], station_cols + ["updated_at"]
-        )
+        station_sql = self._upsert(station_table, ["id"], station_cols + ["updated_at"])
         measurement_sql = self._upsert(
             measurement_table, ["station_id", "timestamp"], measurement_cols
         )

src/ml/dam.py

@@ -61,9 +61,7 @@ def load_daily(
                 params["start"] = start
             engine = create_engine(resolved, pool_pre_ping=True)
             with engine.connect() as conn:
-                daily = pd.read_sql(
-                    text(query + " ORDER BY date"), conn, params=params
-                )
+                daily = pd.read_sql(text(query + " ORDER BY date"), conn, params=params)
             daily["date"] = pd.to_datetime(daily["date"])
             daily = daily.set_index("date")
             for col in DAM_COLUMNS:

src/ml/data.py

@@ -23,7 +23,9 @@ from .features import UPSTREAM_LEADS
 
 logger = logging.getLogger(__name__)
 
-DEFAULT_API_URL = "http://100.81.167.42:8000"
+# Public dashboard. Override with --api-url for a local instance; the
+# Tailscale address of the server is deliberately not the default here.
+DEFAULT_API_URL = "https://water.buildfor.life"
 # Anchored to the repo root so training/prediction work from any CWD; a relative
 # path here silently produced 0 rows when the CLI ran outside the repo root.
 CACHE_DIR = Path(__file__).resolve().parents[2] / "models" / "cache"
@@ -205,15 +207,13 @@ def fill_from_hii(
                 "timestamp": missing["timestamp"],
                 "station_code": code,
                 "water_level": missing["wl_msl"] - offset,
-                "discharge": missing["discharge"]
-                if code in _HII_EXACT_MIRRORS
-                else float("nan"),
+                "discharge": (
+                    missing["discharge"] if code in _HII_EXACT_MIRRORS else float("nan")
+                ),
             }
         )
         fills.append(fill)
-        logger.info(
-            f"HII gap-fill {code}: +{len(fill)} hours (offset {offset:.3f} m)"
-        )
+        logger.info(f"HII gap-fill {code}: +{len(fill)} hours (offset {offset:.3f} m)")
     if not fills:
         return df
     return _normalize_long(pd.concat([df] + fills, ignore_index=True))
@@ -282,6 +282,10 @@ def _read_cache(cache_dir: Path, stations: Optional[List[str]]) -> pd.DataFrame:
     frames = []
     for path in sorted(cache_dir.glob("*.csv.gz")):
         code = path.name[: -len(".csv.gz")]
+        # The dir is shared with rain.py / dam.py caches (rain_openmeteo,
+        # dam_<id>): only station files (P.<n>) are measurements.
+        if not code.startswith("P."):
+            continue
         if stations and code not in stations:
             continue
         with gzip.open(path, "rt", encoding="utf-8") as handle:

src/ml/evaluate.py

@@ -52,22 +52,40 @@ def _flood_weights(y_abs: pd.Series) -> np.ndarray:
     return 1.0 + 4.0 * np.clip((y_abs.to_numpy() - 2.5) / 1.2, 0.0, 1.0)
 
 
+# Experimental forward-48h rain sum, built in evaluate_station (not in
+# features.build_features) so the served feature set is untouched until the
+# harness says it helps. Serving could supply it: fetch_forecast() already
+# pulls forecast_days=2.
+EXTRA_RAIN_FEATURES = ("rain_fc48",)
+
+
 class Variant:
     """A trainable candidate producing (pred_abs, sigma_per_row) on test rows."""
 
-    def __init__(self, name: str, target: str, weighted: bool = False,
-                 quantile: bool = False, use_rain: bool = False,
-                 use_dam: bool = False):
+    def __init__(
+        self,
+        name: str,
+        target: str,
+        weighted: bool = False,
+        quantile: bool = False,
+        use_rain: bool = False,
+        use_dam: bool = False,
+        use_fc48: bool = False,
+        qsigma: bool = False,
+    ):
         self.name = name
         self.target = target  # 'abs' or 'rise'
         self.weighted = weighted
         self.quantile = quantile
         self.use_rain = use_rain
         self.use_dam = use_dam
+        self.use_fc48 = use_fc48
+        # Hybrid: L2 head for the point prediction (keeps the lead-time
+        # behaviour of the deployed model exactly, since p>=0.5 alerts are
+        # sigma-independent) and quantile heads ONLY for a per-row sigma.
+        self.qsigma = qsigma
 
-    def fit_predict(
-        self, X_tr, y_abs_tr, X_te
-    ) -> Tuple[np.ndarray, np.ndarray]:
+    def fit_predict(self, X_tr, y_abs_tr, X_te) -> Tuple[np.ndarray, np.ndarray]:
         if not self.use_rain:
             drop = [c for c in features.RAIN_FEATURES if c in X_tr.columns]
             X_tr = X_tr.drop(columns=drop)
@@ -84,6 +102,12 @@ class Variant:
             raise ValueError(
                 f"{self.name} requires the dam series (rid_reservoir_daily backfilled)"
             )
+        if not self.use_fc48:
+            drop = [c for c in EXTRA_RAIN_FEATURES if c in X_tr.columns]
+            X_tr = X_tr.drop(columns=drop)
+            X_te = X_te.drop(columns=drop)
+        elif "rain_fc48" not in X_tr.columns:
+            raise ValueError(f"{self.name} requires the rain series")
         level_tr = X_tr["level"]
         level_te = X_te["level"].to_numpy()
         y_tr = (y_abs_tr - level_tr) if self.target == "rise" else y_abs_tr
@@ -99,7 +123,13 @@ class Variant:
         else:
             reg = _make_regressor().fit(X_tr, y_tr, sample_weight=weights)
             pred = reg.predict(X_te)
-            sigma = np.full(len(X_te), FIXED_SIGMA)
+            if self.qsigma:
+                q50 = _quantile_regressor(0.5).fit(X_tr, y_tr, sample_weight=weights)
+                q90 = _quantile_regressor(0.9).fit(X_tr, y_tr, sample_weight=weights)
+                spread = np.maximum(q90.predict(X_te) - q50.predict(X_te), 0.0)
+                sigma = np.maximum(spread / 1.2816, 0.05)
+            else:
+                sigma = np.full(len(X_te), FIXED_SIGMA)
 
         pred_abs = pred + level_te if self.target == "rise" else pred
         pred_abs = np.maximum(pred_abs, level_te)  # peak >= current, as served
@@ -110,18 +140,44 @@ VARIANTS: Dict[str, Variant] = {
     "baseline_abs": Variant("baseline_abs", target="abs"),
     "rise": Variant("rise", target="rise"),
     "rise_weighted": Variant("rise_weighted", target="rise", weighted=True),
-    "rise_quantile": Variant("rise_quantile", target="rise", weighted=True,
-                             quantile=True),
+    "rise_quantile": Variant(
+        "rise_quantile", target="rise", weighted=True, quantile=True
+    ),
     "rise_rain": Variant("rise_rain", target="rise", use_rain=True),
-    "rise_rain_dam": Variant("rise_rain_dam", target="rise", use_rain=True,
-                             use_dam=True),
+    "rise_rain_dam": Variant(
+        "rise_rain_dam", target="rise", use_rain=True, use_dam=True
+    ),
     "rise_dam": Variant("rise_dam", target="rise", use_dam=True),
+    # 2026-09-12 experiments on top of the deployed rise_rain configuration:
+    # per-row sigma from quantile heads (the served sigma sits on the 0.15
+    # floor at every P.1 horizon, so stage probabilities are constant-
+    # calibrated), and a longer forecast-rain window for the 24 h horizon.
+    "rise_rain_quantile": Variant(
+        "rise_rain_quantile", target="rise", weighted=True, quantile=True, use_rain=True
+    ),
+    "rise_rain_quantile_uw": Variant(
+        "rise_rain_quantile_uw", target="rise", quantile=True, use_rain=True
+    ),
+    "rise_rain_fc48": Variant(
+        "rise_rain_fc48", target="rise", use_rain=True, use_fc48=True
+    ),
+    "rise_rain_qsigma": Variant(
+        "rise_rain_qsigma", target="rise", use_rain=True, qsigma=True
+    ),
 }
 
 # Dam variants are opt-in by name: they require dam columns that only exist
 # for features.DAM_STATIONS and only when the reservoir series loaded, and
-# the 2026-08-13 ablation concluded them a negative result.
-DEFAULT_VARIANTS = [k for k, v in VARIANTS.items() if not v.use_dam]
+# the 2026-08-13 ablation concluded them a negative result. The 2026-09-12
+# experiments are opt-in too (see their results in docs/FLOOD_FORECASTING.md).
+DEFAULT_VARIANTS = [
+    k
+    for k, v in VARIANTS.items()
+    if not v.use_dam
+    and not v.use_fc48
+    and not v.qsigma
+    and not (v.quantile and v.use_rain)
+]
 
 
 def _find_events(observed: pd.Series, thr: float) -> List[dict]:
@@ -167,12 +223,16 @@ def _first_alert_lead(
     start = crossing - pd.Timedelta(hours=72)
     if window_start_floor is not None and window_start_floor > start:
         start = window_start_floor
-    window = p.loc[start: crossing + pd.Timedelta(hours=24)]
+    window = p.loc[start : crossing + pd.Timedelta(hours=24)]
     if len(window) < 2:
         return None
-    alert = (window >= ALERT_P) & (window.shift(-1) >= ALERT_P) & (
-        (window.index.to_series().shift(-1) - window.index.to_series())
-        <= pd.Timedelta(hours=2)
+    alert = (
+        (window >= ALERT_P)
+        & (window.shift(-1) >= ALERT_P)
+        & (
+            (window.index.to_series().shift(-1) - window.index.to_series())
+            <= pd.Timedelta(hours=2)
+        )
     )
     hits = window.index[alert.fillna(False)]
     if len(hits) == 0:
@@ -180,9 +240,7 @@ def _first_alert_lead(
     return float((crossing - hits[0]).total_seconds() / 3600.0)
 
 
-def _false_alarm_episodes(
-    p: pd.Series, observed: pd.Series, thr: float
-) -> int:
+def _false_alarm_episodes(p: pd.Series, observed: pd.Series, thr: float) -> int:
     """Alert episodes with no observed >=thr within +/- FALSE_ALARM_GRACE_H."""
     alert_hours = p[p >= ALERT_P].index
     if len(alert_hours) == 0:
@@ -222,6 +280,12 @@ def evaluate_station(
     warn_thr, _ = features.get_thresholds(station)
     grid = features.make_hourly_grid(df_long)
     X_all = features.build_features(grid, station, rain=rain, dam=dam)
+    if rain is not None:
+        # forward sum over (t, t+48]; same construction as rain_fc24
+        r = rain.reindex(X_all.index)
+        X_all["rain_fc48"] = (
+            r.shift(-1).iloc[::-1].rolling(48, min_periods=1).sum().iloc[::-1]
+        )
     observed = grid.observed[(station, "water_level")]
 
     keep = X_all["obs_age_h"].notna()
@@ -247,7 +311,9 @@ def evaluate_station(
         tr = (X_all.index <= train_end) & y_abs.notna()
         te = (X_all.index >= test_lo) & (X_all.index <= test_hi)
         if tr.sum() < 5000 or te.sum() < 500:
-            logger.info(f"{station} {year}: skipped (train {tr.sum()}, test {te.sum()})")
+            logger.info(
+                f"{station} {year}: skipped (train {tr.sum()}, test {te.sum()})"
+            )
             continue
 
         X_tr, X_te = X_all.loc[tr], X_all.loc[te]
@@ -323,9 +389,7 @@ def evaluate_station(
                 )
             fold["variants"][name] = {
                 "mae": float(errors.mean()) if len(errors) else None,
-                "mae_above_2p5": (
-                    float(errors[high].mean()) if high.any() else None
-                ),
+                "mae_above_2p5": (float(errors[high].mean()) if high.any() else None),
                 "brier_warn": brier,
                 "events": event_rows,
                 "false_alarm_episodes": _false_alarm_episodes(
@@ -346,17 +410,20 @@ def summarize(results: Dict) -> str:
     lines.append(header)
     for fold in results["folds"]:
         for name, m in fold["variants"].items():
-            events = "  ".join(
-                f"[{e['crossing'][:10]}: "
-                f"{'—' if e['lead_h'] is None else format(e['lead_h'], '+.0f')}h"
-                + (
-                    f" | {e['peak_pred_24h_before'] - e['peak_level']:+.2f}"
-                    if e["peak_pred_24h_before"] is not None
-                    else ""
+            events = (
+                "  ".join(
+                    f"[{e['crossing'][:10]}: "
+                    f"{'—' if e['lead_h'] is None else format(e['lead_h'], '+.0f')}h"
+                    + (
+                        f" | {e['peak_pred_24h_before'] - e['peak_level']:+.2f}"
+                        if e["peak_pred_24h_before"] is not None
+                        else ""
+                    )
+                    + "]"
+                    for e in m["events"]
                 )
-                + "]"
-                for e in m["events"]
-            ) or "no events"
+                or "no events"
+            )
             lines.append(
                 f"{name:16} {fold['year']:>5} "
                 f"{m['mae'] if m['mae'] is not None else float('nan'):6.3f} "
@@ -373,19 +440,32 @@ def main(argv=None) -> int:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--stations", default="P.1")
     parser.add_argument("--db-url", default=None)
-    parser.add_argument("--variants", default=None,
-                        help="comma list; default all")
+    parser.add_argument("--variants", default=None, help="comma list; default all")
     parser.add_argument("--out", default="models/eval_variants.json")
-    parser.add_argument("--no-rain", action="store_true",
-                        help="skip loading the Open-Meteo rain series")
-    parser.add_argument("--no-dam", action="store_true",
-                        help="skip loading the Mae Ngat reservoir series")
+    parser.add_argument(
+        "--no-rain", action="store_true", help="skip loading the Open-Meteo rain series"
+    )
+    parser.add_argument(
+        "--no-dam",
+        action="store_true",
+        help="skip loading the Mae Ngat reservoir series",
+    )
+    parser.add_argument(
+        "--from-cache",
+        action="store_true",
+        help="offline: read models/cache/ only (no DB, no API, "
+        "no Open-Meteo refresh) -- reproducible reruns",
+    )
     args = parser.parse_args(argv)
 
     logging.basicConfig(
         level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
     )
-    df = data.load_measurements(db_url=args.db_url)
+    if args.from_cache:
+        df = data._read_cache(data.CACHE_DIR, None)
+        logger.info(f"measurements from cache: {len(df)} rows")
+    else:
+        df = data.load_measurements(db_url=args.db_url)
     if df.empty:
         logger.error("no measurement data")
         return 1
@@ -394,7 +474,9 @@ def main(argv=None) -> int:
     if not args.no_rain:
         from . import rain as rain_mod
 
-        rain_series = rain_mod.catchment_mean(rain_mod.load_history())
+        rain_series = rain_mod.catchment_mean(
+            rain_mod.load_history(refresh=not args.from_cache)
+        )
         if rain_series is None:
             logger.warning("rain history unavailable; rain features will be NaN")
         else:
@@ -404,7 +486,7 @@ def main(argv=None) -> int:
             )
 
     dam_frame = None
-    if not args.no_dam:
+    if not args.no_dam and not args.from_cache:
         from . import dam as dam_mod
 
         dam_frame = dam_mod.load_history(db_url=args.db_url)

src/ml/features.py

@@ -37,9 +37,17 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
     "P.4A": (3.40, 3.90),
     "P.5": (4.55, 4.95),
     "P.67": (2.45, 2.90),
-    "P.75": (2.75, 3.50),
+    # P.75: 2024 (the only year with a full flood record, 191% capacity peak)
+    # puts 75-85% at 3.45 m and 95-105% at 3.72 m; 2018/2022 agree within
+    # 0.15 m. The 2026-08 value (2.75) alerted on 15 quiet-season hours.
+    "P.75": (3.20, 3.65),
     "P.76": (5.35, 5.45),
-    "P.77": (2.85, 3.35),
+    # P.77: recalibrated 2026-09-12. The 2026-08 value (2.85) sat below the
+    # gauge's own dry-season baseline (2.6-2.7 m at 8-14% capacity), so the
+    # first ntfy cycle fired a "warning" at 22% capacity. Across 2018-2024,
+    # 75-85% capacity reads 3.35-4.57 m and 95-105% 4.27-5.08 m; 2024 (the
+    # best-sampled flood year) gives 4.57 / 5.08. Slightly conservative:
+    "P.77": (4.30, 4.90),
     "P.81": (5.15, 6.30),
     # P.82 never reached 100% capacity in the record (max level 3.78, max 96.4%);
     # danger sits just below the observed maximum so the head can actually train.

src/ml/hii_rain.py

@@ -0,0 +1,122 @@
+"""Catchment-mean hourly rain from the HII/ThaiWater gauge network.
+
+Independent of Open-Meteo (src/ml/rain.py): those are model-analysis values,
+these are what the gauges measured. The `hii_rainfall` table has been filled
+by the hourly collector since 2026-08-11 and there is NO archive behind it
+(the api-v3 rain_24h_graph endpoint ignores its date range, see
+docs/DATA_SOURCES.md 2.1), so this series cannot yet be a training feature:
+every training row before 2026-08 would be NaN and HistGradientBoosting
+would learn nothing from the column. It becomes a candidate once a full
+monsoon season of gauge rows exists in the rolling-origin harness's test
+span -- the 2027 fold (train through 2027-04-30, test Jun-Nov 2027) is the
+first that could show anything.
+
+Until then it serves two purposes:
+  * a live cross-check of the Open-Meteo catchment mean (/api/hii/rainfall
+    already exposes the raw gauges; this gives the comparable aggregate);
+  * accumulating the comparison so the eventual feature evaluation has a
+    documented bias/variance relationship between the two sources.
+"""
+
+import logging
+from typing import Optional, Sequence, Tuple
+
+import pandas as pd
+
+from .data import resolve_db_url
+
+logger = logging.getLogger(__name__)
+
+# Same footprint as rain.CATCHMENT_POINTS: the upper Ping above P.1. Gauges
+# inside this box are averaged; there are ~130 with recent data (DWR, FOP,
+# HII, RID, TMD), far denser than the five Open-Meteo points.
+CATCHMENT_BOX: Tuple[float, float, float, float] = (18.75, 19.60, 98.60, 99.30)
+# A gauge that reports the same rain_24h for many hours is stuck; drop hours
+# where fewer than this many gauges reported at all.
+MIN_GAUGES_PER_HOUR = 5
+
+
+def load_gauge_mean(
+    db_url: Optional[str] = None,
+    start: Optional[pd.Timestamp] = None,
+    end: Optional[pd.Timestamp] = None,
+    box: Sequence[float] = CATCHMENT_BOX,
+    engine=None,
+) -> Optional[pd.Series]:
+    """Hourly catchment-mean rain_1h (mm) across HII gauges in `box`.
+
+    Pass `engine` (the API's HII store engine) to reuse a pool; otherwise a
+    connection is resolved from db_url / config. Returns None if the DB is
+    unavailable or the table is empty. Hours with fewer than
+    MIN_GAUGES_PER_HOUR reporting gauges are NaN.
+    """
+    if engine is None:
+        resolved = resolve_db_url(db_url)
+        if not resolved:
+            return None
+    lat_lo, lat_hi, lon_lo, lon_hi = box
+    try:
+        from sqlalchemy import create_engine, text
+
+        query = (
+            "SELECT m.timestamp, COUNT(m.rain_1h) AS n, AVG(m.rain_1h) AS rain_1h "
+            "FROM hii_rainfall m JOIN hii_rain_stations s ON s.id = m.station_id "
+            "WHERE s.latitude BETWEEN :lat_lo AND :lat_hi "
+            "AND s.longitude BETWEEN :lon_lo AND :lon_hi "
+            "AND m.rain_1h IS NOT NULL"
+        )
+        params = {
+            "lat_lo": lat_lo,
+            "lat_hi": lat_hi,
+            "lon_lo": lon_lo,
+            "lon_hi": lon_hi,
+        }
+        if start is not None:
+            query += " AND m.timestamp >= :start"
+            params["start"] = pd.Timestamp(start).to_pydatetime()
+        if end is not None:
+            query += " AND m.timestamp <= :end"
+            params["end"] = pd.Timestamp(end).to_pydatetime()
+        query += " GROUP BY m.timestamp ORDER BY m.timestamp"
+        if engine is None:
+            engine = create_engine(resolved, pool_pre_ping=True)
+        with engine.connect() as conn:
+            frame = pd.read_sql(text(query), conn, params=params)
+    except Exception as error:
+        logger.warning(f"HII gauge rain load failed: {error}")
+        return None
+    if frame.empty:
+        return None
+    frame["timestamp"] = pd.to_datetime(frame["timestamp"]).dt.floor("h")
+    frame = frame.groupby("timestamp").agg(n=("n", "sum"), rain_1h=("rain_1h", "mean"))
+    series = pd.to_numeric(frame["rain_1h"], errors="coerce")
+    series[frame["n"] < MIN_GAUGES_PER_HOUR] = float("nan")
+    series.name = "hii_gauge_mean"
+    return series
+
+
+def compare_with_openmeteo(
+    gauge: pd.Series, openmeteo: pd.Series, window_h: int = 24
+) -> dict:
+    """Bias/correlation of Open-Meteo against the gauges over the overlap.
+
+    Both are summed over trailing `window_h` so single-hour timing offsets
+    (gauges report at :00, the model's hour is an interval) do not dominate.
+    """
+    joined = pd.concat({"gauge": gauge, "openmeteo": openmeteo}, axis=1).dropna()
+    if joined.empty:
+        return {"overlap_hours": 0}
+    g = joined["gauge"].rolling(window_h, min_periods=window_h).sum()
+    o = joined["openmeteo"].rolling(window_h, min_periods=window_h).sum()
+    both = pd.concat({"g": g, "o": o}, axis=1).dropna()
+    if both.empty:
+        return {"overlap_hours": int(len(joined))}
+    return {
+        "overlap_hours": int(len(joined)),
+        "window_h": window_h,
+        "gauge_mean_mm": float(both["g"].mean()),
+        "openmeteo_mean_mm": float(both["o"].mean()),
+        "bias_mm": float((both["o"] - both["g"]).mean()),
+        "mae_mm": float((both["o"] - both["g"]).abs().mean()),
+        "corr": float(both["g"].corr(both["o"])),
+    }

src/ml/predict.py

@@ -182,14 +182,18 @@ def _model_forecast(
         )
         p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
         if warn_head is not None:
-            p_warning = max(p_warning, float(warn_head.predict_proba(feature_row)[0][1]))
+            p_warning = max(
+                p_warning, float(warn_head.predict_proba(feature_row)[0][1])
+            )
 
         danger_head = (
             None if thresholds_stale else bundle["heads"].get(f"danger_{horizon_h}")
         )
         p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
         if danger_head is not None:
-            p_danger = max(p_danger, float(danger_head.predict_proba(feature_row)[0][1]))
+            p_danger = max(
+                p_danger, float(danger_head.predict_proba(feature_row)[0][1])
+            )
 
         p_warning = _clip_probability(p_warning)
         p_danger = min(_clip_probability(p_danger), p_warning)
@@ -400,6 +404,4 @@ def get_latest_forecasts(
         logger.warning("dam state unavailable; dam features will be NaN")
         dam = pd.DataFrame()
 
-    return get_forecasts(
-        readings_by_station, models_dir=models_dir, rain=rain, dam=dam
-    )
+    return get_forecasts(readings_by_station, models_dir=models_dir, rain=rain, dam=dam)

src/ml/rain.py

@@ -121,12 +121,8 @@ def load_history(
     cursor = fetch_from.date()
     try:
         while cursor <= end:
-            chunk_end = min(
-                datetime.date(cursor.year, 12, 31), end
-            )
-            chunks.append(
-                fetch_history(cursor.isoformat(), chunk_end.isoformat())
-            )
+            chunk_end = min(datetime.date(cursor.year, 12, 31), end)
+            chunks.append(fetch_history(cursor.isoformat(), chunk_end.isoformat()))
             cursor = datetime.date(cursor.year + 1, 1, 1)
     except Exception as error:
         logger.warning(f"Open-Meteo history fetch failed: {error}")
@@ -174,7 +170,7 @@ def backfill_db(engine, db_type: str, chunk_rows: int = 5000) -> int:
         return 0
     total = 0
     for start in range(0, len(history), chunk_rows):
-        part = history.iloc[start: start + chunk_rows]
+        part = history.iloc[start : start + chunk_rows]
         total += save_to_db(part, engine, db_type)
         logger.info(f"openmeteo_rain backfill: {total}/{len(history)} rows")
     return total
@@ -204,8 +200,7 @@ def save_to_db(df: pd.DataFrame, engine, db_type: str) -> int:
     cols = ["timestamp"] + point_cols + ["catchment_mean"]
     placeholders = ", ".join(f":{c}" for c in cols)
     updates = ", ".join(
-        f"{c} = "
-        + (f"VALUES({c})" if db_type == "mysql" else f"EXCLUDED.{c}")
+        f"{c} = " + (f"VALUES({c})" if db_type == "mysql" else f"EXCLUDED.{c}")
         for c in cols[1:]
     )
     if db_type == "mysql":

src/ml/skill.py

@@ -0,0 +1,170 @@
+"""Live forecast skill: what the deployed model said versus what the river did.
+
+Every hour the precompute stores the issued 24 h forecast (forecast_history);
+water_measurements holds what actually happened. Joining the two gives a
+verification that needs no retraining and answers the question the dashboard
+is asked most: "is the model getting better?" — per model version, on the
+hours that version was actually serving.
+
+Metrics per version and horizon:
+  n           verified forecasts (issued, and the horizon has since elapsed)
+  mae         |predicted_max - observed_max| over the horizon window, metres
+  bias        mean(predicted - observed): >0 over-predicts the peak
+  persistence MAE of the trivial "peak = current level" forecast on the
+              same rows; a model is only useful if it beats this
+  skill       1 - mae/persistence  (0 = no better than persistence, 1 = perfect)
+  above_2m    same MAE restricted to rows where the observed peak >= 2 m,
+              i.e. the flood-relevant regime
+
+Only the P.1 gauge is verified by default: it is the one the city threshold
+is keyed to, and one station keeps the query cheap enough to run on request.
+"""
+
+import datetime
+import logging
+from typing import Dict, List, Optional
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_STATION = "P.1"
+DEFAULT_HORIZON = 24
+MIN_VERIFIED = 24  # fewer than a day of verified hours is not a number
+
+
+def _sql_for(db_type: str) -> str:
+    """Join each issued forecast to the observed max over (as_of, as_of + h]."""
+    if db_type == "postgresql":
+        window_end = "f.as_of + (f.horizon_hours || ' hours')::interval"
+    elif db_type == "mysql":
+        window_end = "DATE_ADD(f.as_of, INTERVAL f.horizon_hours HOUR)"
+    else:  # sqlite
+        window_end = "datetime(f.as_of, '+' || f.horizon_hours || ' hours')"
+    return f"""
+        SELECT f.as_of, f.model_version, f.predicted_max_level, f.current_level,
+               (SELECT MAX(m.water_level) FROM water_measurements m
+                 JOIN stations s ON s.id = m.station_id
+                WHERE s.station_code = f.station_code
+                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_max,
+               (SELECT COUNT(m.water_level) FROM water_measurements m
+                 JOIN stations s ON s.id = m.station_id
+                WHERE s.station_code = f.station_code
+                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_n
+        FROM forecast_history f
+        WHERE f.station_code = :code AND f.horizon_hours = :horizon
+          AND f.source = 'model' AND f.predicted_max_level IS NOT NULL
+          AND f.as_of <= :verifiable_before
+        ORDER BY f.as_of
+    """
+
+
+def compute_skill(
+    engine,
+    db_type: str,
+    station_code: str = DEFAULT_STATION,
+    horizon_hours: int = DEFAULT_HORIZON,
+    now: Optional[datetime.datetime] = None,
+) -> Dict:
+    """Per-model-version verification of issued forecasts against observations."""
+    from sqlalchemy import text
+
+    now = now or datetime.datetime.now()
+    verifiable_before = now - datetime.timedelta(hours=horizon_hours)
+    with engine.connect() as conn:
+        rows = [
+            dict(r._mapping)
+            for r in conn.execute(
+                text(_sql_for(db_type)),
+                {
+                    "code": station_code,
+                    "horizon": horizon_hours,
+                    "verifiable_before": verifiable_before,
+                },
+            )
+        ]
+
+    def _ts(value):
+        # sqlite hands back strings; postgres/mysql give datetimes
+        if isinstance(value, datetime.datetime):
+            return value
+        return datetime.datetime.fromisoformat(str(value).replace(" ", "T"))
+
+    by_version: Dict[str, List[dict]] = {}
+    for r in rows:
+        r["as_of"] = _ts(r["as_of"])
+        # need most of the window observed, or the "max" is not the peak
+        if r["observed_max"] is None or (r["observed_n"] or 0) < horizon_hours * 0.75:
+            continue
+        by_version.setdefault(r["model_version"] or "unknown", []).append(r)
+
+    versions = []
+    for version, vrows in by_version.items():
+        pred = [float(r["predicted_max_level"]) for r in vrows]
+        obs = [float(r["observed_max"]) for r in vrows]
+        cur = [
+            float(r["current_level"]) if r["current_level"] is not None else None
+            for r in vrows
+        ]
+        err = [p - o for p, o in zip(pred, obs)]
+        mae = sum(abs(e) for e in err) / len(err)
+        bias = sum(err) / len(err)
+        pers_rows = [(c, o) for c, o in zip(cur, obs) if c is not None]
+        persistence = (
+            sum(abs(c - o) for c, o in pers_rows) / len(pers_rows)
+            if pers_rows
+            else None
+        )
+        high = [(p, o) for p, o in zip(pred, obs) if o >= 2.0]
+        versions.append(
+            {
+                "model_version": version,
+                "first_issued": min(r["as_of"] for r in vrows).isoformat(),
+                "last_issued": max(r["as_of"] for r in vrows).isoformat(),
+                "n": len(vrows),
+                "mae_m": round(mae, 3),
+                "bias_m": round(bias, 3),
+                "persistence_mae_m": (
+                    None if persistence is None else round(persistence, 3)
+                ),
+                "skill": (
+                    None if not persistence else round(1.0 - mae / persistence, 3)
+                ),
+                "above_2m_n": len(high),
+                "above_2m_mae_m": (
+                    round(sum(abs(p - o) for p, o in high) / len(high), 3)
+                    if high
+                    else None
+                ),
+                "enough_data": len(vrows) >= MIN_VERIFIED,
+            }
+        )
+    versions.sort(key=lambda v: v["first_issued"])
+
+    # Headline: current version vs the previous one that had enough data
+    current = versions[-1] if versions else None
+    previous = (
+        next((v for v in reversed(versions[:-1]) if v["enough_data"]), None)
+        if versions
+        else None
+    )
+    trend = None
+    if current and previous and current["enough_data"]:
+        trend = {
+            "previous_version": previous["model_version"],
+            "mae_delta_m": round(current["mae_m"] - previous["mae_m"], 3),
+            "skill_delta": (
+                None
+                if current["skill"] is None or previous["skill"] is None
+                else round(current["skill"] - previous["skill"], 3)
+            ),
+            "better": current["mae_m"] < previous["mae_m"],
+        }
+
+    return {
+        "station_code": station_code,
+        "horizon_hours": horizon_hours,
+        "verified_until": verifiable_before.isoformat(),
+        "min_verified": MIN_VERIFIED,
+        "versions": versions,
+        "current": current,
+        "trend": trend,
+    }

src/ml/train.py

@@ -45,6 +45,15 @@ MIN_SIGMA = 0.15
 MIN_ROWS_TO_TRAIN = 200
 MIN_ROWS_FOR_HEAD = 50
 
+
+class RainUnavailableError(RuntimeError):
+    """Raised when a rain-enabled training run cannot obtain the rain series.
+
+    Training would otherwise fall through to gauge-only (v2) bundles and
+    overwrite the deployed v3 artifacts without anyone noticing.
+    """
+
+
 HGB_PARAMS = {
     "max_iter": 300,
     "learning_rate": 0.06,
@@ -311,9 +320,9 @@ def train_station(
                     skipped_heads,
                 )
             else:
-                skipped_heads[
-                    head_key
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                skipped_heads[head_key] = (
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
             heads[head_key] = clf
 
             if not skip_eval:
@@ -408,9 +417,9 @@ def train_station(
                 if clf is not None:
                     skipped_heads.pop(head_key, None)
             else:
-                skipped_heads[
-                    head_key
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                skipped_heads[head_key] = (
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
                 final_heads[head_key] = None
 
     # v4 = + Mae Ngat dam features; v3 = rise + rain; v2 = rise target only
@@ -456,8 +465,13 @@ def train_all(
     models_dir = Path(models_dir)
     models_dir.mkdir(parents=True, exist_ok=True)
 
-    # Catchment rain (Open-Meteo archive, 2021+). Optional: without it the
-    # models train as v2 (no rain columns) and still serve correctly.
+    # Catchment rain (Open-Meteo archive, 2021+). A rain-less run produces v2
+    # bundles that serve fine but have measurably less flood lead (the 2024
+    # record flood: 13 h early with rain vs 18 h late without). The 2026-09-01
+    # server retrain hit exactly that -- the archive fetch failed on a checkout
+    # with no models/cache/ and the run quietly wrote v2 over v3. So the
+    # downgrade is now an error unless the caller opts out with use_rain=False
+    # (the --no-rain flag), which is the only way to get v2 deliberately.
     rain_series = None
     if use_rain:
         try:
@@ -465,7 +479,19 @@ def train_all(
 
             rain_series = rain_mod.catchment_mean(rain_mod.load_history())
         except Exception as error:
-            logger.warning(f"rain history unavailable, training without it: {error}")
+            raise RainUnavailableError(
+                f"rain history unavailable ({error}); refusing to silently "
+                "downgrade to v2 bundles -- fix Open-Meteo access or restore "
+                "models/cache/rain_openmeteo.csv.gz, or pass --no-rain to "
+                "train gauge-only bundles on purpose"
+            ) from error
+        if rain_series is None:
+            raise RainUnavailableError(
+                "rain history unavailable (Open-Meteo archive unreachable and "
+                "no models/cache/rain_openmeteo.csv.gz); refusing to silently "
+                "downgrade to v2 bundles -- fix access, restore the cache file, "
+                "or pass --no-rain to train gauge-only bundles on purpose"
+            )
     if rain_series is not None:
         logger.info(
             f"rain series: {rain_series.index.min()} .. {rain_series.index.max()}"
@@ -493,9 +519,7 @@ def train_all(
                 "training v3-style bundles WITHOUT dam features"
             )
     if dam_frame is not None:
-        logger.info(
-            f"dam series: {dam_frame.index.min()} .. {dam_frame.index.max()}"
-        )
+        logger.info(f"dam series: {dam_frame.index.min()} .. {dam_frame.index.max()}")
 
     # Run-level version: v4 only if some requested station actually receives
     # dam columns (they are gated to DAM_STATIONS; per-bundle versions are
@@ -584,7 +608,9 @@ def main(argv: Optional[List[str]] = None) -> None:
     parser.add_argument(
         "--no-rain",
         action="store_true",
-        help="train without the Open-Meteo rain features (v2-style bundles)",
+        help="DELIBERATELY train without the Open-Meteo rain features "
+        "(v2-style bundles). Without this flag a missing rain series aborts "
+        "the run instead of quietly downgrading the deployed model",
     )
     parser.add_argument(
         "--dam",
@@ -627,9 +653,21 @@ def main(argv: Optional[List[str]] = None) -> None:
         1 for s in metrics_payload["stations"].values() if s["status"] == "trained"
     )
     logger.info(
-        f"Done: {trained}/{len(stations)} stations trained. metrics.json written to {args.models_dir}"
+        f"Done: {trained}/{len(stations)} stations trained "
+        f"({metrics_payload['model_version']}). "
+        f"metrics.json written to {args.models_dir}"
     )
 
 
+def cli() -> int:
+    """Console entry: RainUnavailableError becomes a one-line error, exit 2."""
+    try:
+        main()
+    except RainUnavailableError as error:
+        logger.error(str(error))
+        return 2
+    return 0
+
+
 if __name__ == "__main__":
-    main()
+    raise SystemExit(cli())

src/notify.py

@@ -0,0 +1,454 @@
+"""Public flood notifications over ntfy.
+
+Runs once per collection cycle inside the API process (leader only), right
+after the forecast precompute, so it sees the same readings and forecasts the
+dashboard shows. Publishes to a self-hosted ntfy server; anyone subscribes to
+a topic from the free app or a browser, no account needed.
+
+Topics (all under one configurable prefix, default "ping"):
+
+  {prefix}-{station}-warning     observed level crossed the station's warning threshold
+  {prefix}-{station}-danger      observed level crossed the danger threshold
+  {prefix}-warning               any station crossed warning (basin-wide digest)
+  {prefix}-danger                any station crossed danger
+  {prefix}-p1-outlook            model early warning for Chiang Mai city: P.1's 24 h
+                                 warning probability crossed the alert level (opt-in;
+                                 the forecast is experimental and says so)
+  {prefix}-status                feed/monitor health: data stale, recovered
+
+Each notification is a TRANSITION, not a state: crossing UP into a level sends
+one message; dropping back below (with hysteresis) sends an all-clear. While
+the river sits above a threshold nothing is repeated, so a subscriber in a
+flood gets a handful of messages, not one an hour. The per-topic state is
+persisted (notification_state table) so a restart never re-sends.
+
+Everything is fail-safe: ntfy unreachable, table missing, malformed
+reading -> a logged warning, never an exception into the collection loop.
+"""
+
+import datetime
+import logging
+from dataclasses import dataclass
+from typing import Dict, Iterable, List, Optional
+
+import requests
+
+from .ml import features
+
+logger = logging.getLogger(__name__)
+
+# Hysteresis: an all-clear needs the level this far BELOW the threshold, so a
+# river bobbing around 3.70 m does not toggle warning/clear every hour.
+CLEAR_MARGIN_M = 0.10
+# Capacity guard. The level thresholds in features.THRESHOLDS were calibrated
+# from RID's discharge_percent (% of channel capacity); if RID re-rates a
+# gauge or moves its datum, the level crosses while capacity says the channel
+# is nearly empty (P.77, 2026-09: 3.0 m "warning" at 22 %). A crossing is
+# only announced when the reported capacity agrees that the river is high.
+# P.1 is exempt: its stages come from the municipal inundation map, not from
+# capacity. Readings without a capacity figure fall back to level only.
+CAPACITY_GUARD_MIN_PCT = 60.0
+CAPACITY_GUARD_EXEMPT = {"P.1"}
+# Outlook alert fires when p_warning(24h) rises through ON, clears below OFF.
+OUTLOOK_ON = 0.50
+OUTLOOK_OFF = 0.25
+# Below this the outlook is not announced at all (avoid "5 % chance" noise).
+OUTLOOK_HORIZON = 24
+
+STATION_NAMES: Dict[str, str] = {
+    "P.1": "Nawarat Bridge, Chiang Mai city",
+    "P.103": "Ring Road Bridge 3, Chiang Mai",
+    "P.67": "Ban Tae (Mae Taeng)",
+    "P.21": "Ban Rim Tai (Mae Rim)",
+    "P.75": "Ban Chai Lat",
+    "P.92": "Ban Muang Aut",
+    "P.20": "Ban Chiang Dao",
+    "P.4A": "Ban Mae Taeng",
+    "P.5": "Tha Nang Bridge (downstream)",
+    "P.81": "Ban Pong (downstream)",
+    "P.82": "Ban Sob Win",
+    "P.84": "Ban Panton",
+    "P.87": "Ban Pa Sang",
+    "P.77": "Ban Sop Mae Sapuat",
+    "P.85": "Ban Lai Kaew",
+    "P.76": "Ban Mae I Hai",
+}
+
+
+def _slug(code: str) -> str:
+    return code.lower().replace(".", "")
+
+
+@dataclass
+class Notification:
+    topic: str
+    title: str
+    message: str
+    priority: int = 3  # ntfy: 1 min .. 5 max
+    tags: Optional[List[str]] = None
+    click: Optional[str] = None
+
+
+class NtfyPublisher:
+    def __init__(
+        self,
+        server: str,
+        prefix: str = "ping",
+        token: Optional[str] = None,
+        dashboard_url: str = "https://water.buildfor.life/",
+        timeout: int = 10,
+    ):
+        self.server = server.rstrip("/")
+        self.prefix = prefix
+        self.token = token
+        self.dashboard_url = dashboard_url
+        self.timeout = timeout
+
+    def topic(self, *parts: str) -> str:
+        return "-".join([self.prefix, *parts])
+
+    def publish(self, n: Notification) -> bool:
+        headers = {
+            "Title": n.title,
+            "Priority": str(n.priority),
+            "Click": n.click or self.dashboard_url,
+            "Actions": f"view, Open dashboard, {n.click or self.dashboard_url}",
+        }
+        if n.tags:
+            headers["Tags"] = ",".join(n.tags)
+        if self.token:
+            headers["Authorization"] = f"Bearer {self.token}"
+        try:
+            r = requests.post(
+                f"{self.server}/{n.topic}",
+                data=n.message.encode("utf-8"),
+                headers=headers,
+                timeout=self.timeout,
+            )
+            if r.status_code >= 300:
+                logger.warning(f"ntfy {n.topic}: HTTP {r.status_code} {r.text[:120]}")
+                return False
+            return True
+        except Exception as error:
+            logger.warning(f"ntfy {n.topic}: {error}")
+            return False
+
+
+class NotificationState:
+    """Per-key last-sent state, in the monitor's own SQL database."""
+
+    def __init__(self, engine, db_type: str):
+        self.engine = engine
+        self.db_type = db_type
+        self._ensure()
+
+    def _ensure(self) -> None:
+        from sqlalchemy import text
+
+        ddl = (
+            "CREATE TABLE IF NOT EXISTS notification_state ("
+            "key VARCHAR(64) PRIMARY KEY, state VARCHAR(16) NOT NULL, "
+            "value NUMERIC(8,3), updated_at TIMESTAMP NOT NULL)"
+        )
+        try:
+            with self.engine.begin() as conn:
+                conn.execute(text(ddl))
+        except Exception as error:
+            # Postgres: two sessions racing CREATE TABLE IF NOT EXISTS can
+            # both pass the existence check; the loser fails with a unique
+            # violation on pg_type. The table exists either way; verify.
+            with self.engine.connect() as conn:
+                conn.execute(text("SELECT 1 FROM notification_state WHERE 1=0"))
+            logger.debug(f"notification_state DDL raced, table present: {error}")
+
+    def get(self, key: str) -> Optional[str]:
+        from sqlalchemy import text
+
+        with self.engine.connect() as conn:
+            row = conn.execute(
+                text("SELECT state FROM notification_state WHERE key = :k"), {"k": key}
+            ).fetchone()
+        return row[0] if row else None
+
+    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
+        from sqlalchemy import text
+
+        now = datetime.datetime.now()
+        with self.engine.begin() as conn:
+            if self.db_type == "mysql":
+                sql = (
+                    "INSERT INTO notification_state (key, state, value, updated_at) "
+                    "VALUES (:k, :s, :v, :t) ON DUPLICATE KEY UPDATE "
+                    "state = VALUES(state), value = VALUES(value), updated_at = VALUES(updated_at)"
+                )
+            else:
+                sql = (
+                    "INSERT INTO notification_state (key, state, value, updated_at) "
+                    "VALUES (:k, :s, :v, :t) ON CONFLICT (key) DO UPDATE SET "
+                    "state = EXCLUDED.state, value = EXCLUDED.value, updated_at = EXCLUDED.updated_at"
+                )
+            conn.execute(text(sql), {"k": key, "s": state, "v": value, "t": now})
+
+
+class InMemoryState(NotificationState):
+    """For tests and when no SQL engine is available (loses state on restart)."""
+
+    def __init__(self):  # noqa: D107 - intentionally skips the SQL parent
+        self._d: Dict[str, str] = {}
+
+    def get(self, key: str) -> Optional[str]:
+        return self._d.get(key)
+
+    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
+        self._d[key] = state
+
+
+def _level_state(level: float, warn: float, danger: float, prev: Optional[str]) -> str:
+    """'clear' | 'warning' | 'danger', with hysteresis on the way down."""
+    if level >= danger:
+        return "danger"
+    if level >= warn:
+        # from danger: stay 'danger' until below danger - margin
+        if prev == "danger" and level >= danger - CLEAR_MARGIN_M:
+            return "danger"
+        return "warning"
+    if prev in ("warning", "danger") and level >= warn - CLEAR_MARGIN_M:
+        return "warning"
+    return "clear"
+
+
+def evaluate(
+    readings: Iterable[dict],
+    forecasts: Iterable[dict],
+    state: NotificationState,
+    publisher: NtfyPublisher,
+    stale_after_h: float = 3.0,
+    now: Optional[datetime.datetime] = None,
+) -> List[Notification]:
+    """Compare current readings/forecasts with last-sent state; publish transitions.
+
+    readings: rows with station_code, water_level, timestamp (latest per station)
+    forecasts: /forecast rows (station_code, horizon_hours, p_warning, predicted_max_level)
+    Returns the notifications that were published (for logs/tests).
+    """
+    now = now or datetime.datetime.now()
+    sent: List[Notification] = []
+
+    def emit(n: Notification) -> bool:
+        ok = publisher.publish(n)
+        if ok:
+            sent.append(n)
+        return ok
+
+    # ---- observed levels, per station, plus basin-wide fan-out
+    basin_changes: Dict[str, List[str]] = {"warning": [], "danger": [], "clear": []}
+    latest_ts: Optional[datetime.datetime] = None
+    for r in readings:
+        code = r.get("station_code")
+        level = r.get("water_level")
+        if not code or level is None:
+            continue
+        try:
+            level = float(level)
+        except (TypeError, ValueError):
+            continue
+        ts = r.get("timestamp")
+        if isinstance(ts, str):
+            try:
+                ts = datetime.datetime.fromisoformat(ts)
+            except ValueError:
+                ts = None
+        if isinstance(ts, datetime.datetime) and (latest_ts is None or ts > latest_ts):
+            latest_ts = ts
+        warn, danger = features.get_thresholds(code)
+        key = f"level:{code}"
+        prev = state.get(key) or "clear"
+        cur = _level_state(level, warn, danger, prev)
+        pct = r.get("discharge_percent")
+        if (
+            cur != "clear"
+            and prev == "clear"
+            and code not in CAPACITY_GUARD_EXEMPT
+            and pct is not None
+        ):
+            try:
+                if float(pct) < CAPACITY_GUARD_MIN_PCT:
+                    logger.info(
+                        f"{code}: level {level:.2f} m >= {warn:.2f} but only "
+                        f"{float(pct):.0f}% capacity; threshold looks stale, not alerting"
+                    )
+                    continue
+            except (TypeError, ValueError):
+                pass
+        if cur == prev:
+            continue
+        name = STATION_NAMES.get(code, code)
+        slug = _slug(code)
+        when = (
+            ts.strftime("%d %b %H:%M") if isinstance(ts, datetime.datetime) else "now"
+        )
+        if cur == "danger":
+            ok = emit(
+                Notification(
+                    publisher.topic(slug, "danger"),
+                    f"DANGER level at {code}",
+                    f"{name}: {level:.2f} m at {when}, above the danger level of {danger:.2f} m.",
+                    priority=5,
+                    tags=["rotating_light", code],
+                )
+            )
+            basin_changes["danger"].append(f"{code} {level:.2f} m")
+        elif cur == "warning":
+            if prev == "danger":
+                ok = emit(
+                    Notification(
+                        publisher.topic(slug, "danger"),
+                        f"{code} back below danger level",
+                        f"{name}: {level:.2f} m at {when}; still above the warning level of {warn:.2f} m.",
+                        priority=3,
+                        tags=["arrow_down", code],
+                    )
+                )
+                basin_changes["clear"].append(f"{code} below danger ({level:.2f} m)")
+            else:
+                ok = emit(
+                    Notification(
+                        publisher.topic(slug, "warning"),
+                        f"Warning level at {code}",
+                        f"{name}: {level:.2f} m at {when}, above the warning level of {warn:.2f} m.",
+                        priority=4,
+                        tags=["warning", code],
+                    )
+                )
+                basin_changes["warning"].append(f"{code} {level:.2f} m")
+        else:  # clear
+            ok = emit(
+                Notification(
+                    publisher.topic(slug, "warning"),
+                    f"{code} back to normal",
+                    f"{name}: {level:.2f} m at {when}, below the warning level of {warn:.2f} m.",
+                    priority=2,
+                    tags=["white_check_mark", code],
+                )
+            )
+            basin_changes["clear"].append(f"{code} normal ({level:.2f} m)")
+        # Only remember the transition once it was actually delivered: if ntfy
+        # was down, the next cycle retries instead of silently swallowing a
+        # flood crossing.
+        if ok:
+            state.set(key, cur, level)
+
+    if basin_changes["danger"]:
+        emit(
+            Notification(
+                publisher.topic("danger"),
+                "Ping River: danger level reached",
+                "; ".join(basin_changes["danger"]),
+                priority=5,
+                tags=["rotating_light"],
+            )
+        )
+    if basin_changes["warning"]:
+        emit(
+            Notification(
+                publisher.topic("warning"),
+                "Ping River: warning level reached",
+                "; ".join(basin_changes["warning"]),
+                priority=4,
+                tags=["warning"],
+            )
+        )
+    if basin_changes["clear"]:
+        emit(
+            Notification(
+                publisher.topic("warning"),
+                "Ping River: levels falling",
+                "; ".join(basin_changes["clear"]),
+                priority=2,
+                tags=["white_check_mark"],
+            )
+        )
+
+    # ---- model outlook for the city gauge (opt-in topic, experimental)
+    p1 = next(
+        (
+            f
+            for f in forecasts
+            if f.get("station_code") == "P.1"
+            and f.get("horizon_hours") == OUTLOOK_HORIZON
+            and f.get("source") == "model"
+        ),
+        None,
+    )
+    if p1 and p1.get("p_warning") is not None:
+        p = float(p1["p_warning"])
+        key = "outlook:P.1"
+        prev = state.get(key) or "off"
+        cur = (
+            "on" if (p >= OUTLOOK_ON or (prev == "on" and p >= OUTLOOK_OFF)) else "off"
+        )
+        if cur != prev:
+            peak = p1.get("predicted_max_level")
+            warn, _ = features.get_thresholds("P.1")
+            if cur == "on":
+                ok = emit(
+                    Notification(
+                        publisher.topic("p1-outlook"),
+                        "Early warning: Chiang Mai flood risk rising",
+                        f"The forecast model gives a {p * 100:.0f}% chance that Nawarat Bridge (P.1) "
+                        f"reaches {warn:.2f} m within 24 h"
+                        + (
+                            f" (expected peak {float(peak):.2f} m)"
+                            if peak is not None
+                            else ""
+                        )
+                        + ". Experimental model output, not an official warning; "
+                        "follow ThaiWater/TMD for official alerts.",
+                        priority=4,
+                        tags=["crystal_ball"],
+                    )
+                )
+            else:
+                ok = emit(
+                    Notification(
+                        publisher.topic("p1-outlook"),
+                        "Chiang Mai flood risk easing",
+                        f"The model's 24 h probability of reaching {warn:.2f} m at P.1 has dropped to {p * 100:.0f}%.",
+                        priority=2,
+                        tags=["crystal_ball"],
+                    )
+                )
+            if ok:
+                state.set(key, cur, p)
+
+    # ---- feed health
+    if latest_ts is not None:
+        age_h = (now - latest_ts).total_seconds() / 3600.0
+        key = "feed"
+        prev = state.get(key) or "ok"
+        cur = "stale" if age_h >= stale_after_h else "ok"
+        if cur != prev:
+            if cur == "stale":
+                ok = emit(
+                    Notification(
+                        publisher.topic("status"),
+                        "Ping River monitor: gauge feed stale",
+                        f"No new readings for {age_h:.0f} h (last {latest_ts:%d %b %H:%M}). "
+                        "Levels and forecasts on the dashboard are not current.",
+                        priority=3,
+                        tags=["hourglass"],
+                    )
+                )
+            else:
+                ok = emit(
+                    Notification(
+                        publisher.topic("status"),
+                        "Ping River monitor: feed recovered",
+                        f"Readings are current again (latest {latest_ts:%d %b %H:%M}).",
+                        priority=2,
+                        tags=["white_check_mark"],
+                    )
+                )
+            if ok:
+                state.set(key, cur, age_h)
+    return sent

src/postgres_history.py

@@ -51,8 +51,7 @@ class PostgresHistory:
         if start >= end:
             raise ValueError("start must be before end")
 
-        query = text(
-            """
+        query = text("""
             SELECT m.timestamp, s.station_code, m.water_level,
                    m.discharge, m.discharge_percent
             FROM water_measurements m
@@ -62,8 +61,7 @@ class PostgresHistory:
               AND m.timestamp <= :end_time
             ORDER BY m.timestamp ASC
             LIMIT :limit
-            """
-        )
+            """)
         with self.engine.connect() as connection:
             rows = connection.execute(
                 query,
@@ -91,9 +89,9 @@ class PostgresHistory:
                         "station_code": station_code,
                         "water_level": water_level,
                         "discharge": discharge,
-                        "discharge_percent": float(row[4])
-                        if row[4] is not None
-                        else None,
+                        "discharge_percent": (
+                            float(row[4]) if row[4] is not None else None
+                        ),
                     }
                 )
             return result

src/rate_limiter.py

@@ -173,8 +173,10 @@ class RequestTracker:
                 "failed_requests": self.failed_requests,
                 "success_rate": self.successful_requests / self.total_requests,
                 "average_response_time": self.total_response_time / self.total_requests,
-                "last_request_time": self.last_request_time.isoformat()
-                if self.last_request_time
-                else None,
+                "last_request_time": (
+                    self.last_request_time.isoformat()
+                    if self.last_request_time
+                    else None
+                ),
                 "error_breakdown": dict(self.error_count_by_type),
             }

src/rid_reservoir.py

@@ -323,9 +323,11 @@ class RidReservoirStore:
             if not preserve_cols:
                 return f"INSERT OR REPLACE INTO {table} ({col_list}) VALUES ({params})"
             updates = ", ".join(
-                f"{c} = COALESCE(excluded.{c}, {table}.{c})"
-                if c in preserve_cols
-                else f"{c} = excluded.{c}"
+                (
+                    f"{c} = COALESCE(excluded.{c}, {table}.{c})"
+                    if c in preserve_cols
+                    else f"{c} = excluded.{c}"
+                )
                 for c in value_cols
             )
             return (
@@ -334,9 +336,11 @@ class RidReservoirStore:
             )
         if self.db_type == "postgresql":
             updates = ", ".join(
-                f"{c} = COALESCE(EXCLUDED.{c}, {table}.{c})"
-                if c in preserve_cols
-                else f"{c} = EXCLUDED.{c}"
+                (
+                    f"{c} = COALESCE(EXCLUDED.{c}, {table}.{c})"
+                    if c in preserve_cols
+                    else f"{c} = EXCLUDED.{c}"
+                )
                 for c in value_cols
             )
             return (
@@ -344,9 +348,11 @@ class RidReservoirStore:
                 f"ON CONFLICT ({conflict}) DO UPDATE SET {updates}"
             )
         updates = ", ".join(
-            f"{c} = COALESCE(VALUES({c}), {c})"
-            if c in preserve_cols
-            else f"{c} = VALUES({c})"
+            (
+                f"{c} = COALESCE(VALUES({c}), {c})"
+                if c in preserve_cols
+                else f"{c} = VALUES({c})"
+            )
             for c in value_cols
         )
         return (
@@ -402,9 +408,7 @@ class RidReservoirStore:
             measure_row = {
                 c: _bounded(record.get(c), _MEASURE_BOUNDS[c]) for c in measure_cols
             }
-            measure_row.update(
-                {"dam_id": record["dam_id"], "date": record["date"]}
-            )
+            measure_row.update({"dam_id": record["dam_id"], "date": record["date"]})
             measurements.append(measure_row)
         try:
             with self.engine.begin() as conn:
@@ -495,9 +499,7 @@ def backfill(
     if not store.engine and not store.connect():
         logger.error("backfill aborted: database connection failed")
         return 0
-    span = [
-        start + datetime.timedelta(days=i) for i in range((end - start).days + 1)
-    ]
+    span = [start + datetime.timedelta(days=i) for i in range((end - start).days + 1)]
     present = store.present_dates(start, end)
     targets = [d for d in span if d not in present]
     logger.info(

src/static/dashboard.html

@@ -33,6 +33,57 @@
             --red: #cc4b37;
             --border: #dce7e3;
             --shadow: 0 16px 40px rgba(23, 57, 67, .10);
+            --surface: #ffffff;          /* buttons, inputs, overlays */
+            --surface-2: #f1f7f5;        /* hover rows, popup metrics */
+            --surface-3: #f7fbfa;        /* outlook box */
+            --overlay: rgba(255,255,255,.93);
+            --overlay-border: rgba(207,224,218,.9);
+            --map-bg: #dcebea;
+            --glow: rgba(56, 180, 213, .12);
+            --map-filter: none;
+            --chart-grid: rgba(19,43,53,.08);
+            --mint-ink: #146644;
+            --mint-border: #b6dfc9;
+            --ok-bg: #e2f3ea; --ok-ink: #0c5138; --ok-border: #bfe3d2;
+            --watch-bg: #fdf1dc; --watch-ink: #6b4a05; --watch-border: #f0d9a8;
+            --danger-bg: #fbe3de; --danger-ink: #7c1d10; --danger-border: #f2c0b6;
+            --demo-bg: #fdeeda; --demo-ink: #9a6200;
+            color-scheme: light;
+        }
+        /* Dark theme: same hues, inverted lightness. Flow/rain/marker colours
+           on the map are data encodings and stay identical in both themes. */
+        [data-theme="dark"] {
+            --ink: #e6eef1;
+            --muted: #97a9b0;
+            --paper: #0f1a1f;
+            --card: #16252c;
+            --river: #3fb3d8;
+            --river-light: #5ccbe8;
+            --mint: #143b2c;
+            --green: #3fb07f;
+            --amber: #e6a83a;
+            --red: #e46a55;
+            --border: #27393f;
+            --shadow: 0 16px 40px rgba(0, 0, 0, .45);
+            --surface: #1c2c33;
+            --surface-2: #213238;
+            --surface-3: #1a2a30;
+            --overlay: rgba(22,37,44,.92);
+            --overlay-border: rgba(64,86,94,.9);
+            --map-bg: #1a262b;
+            --glow: rgba(63, 179, 216, .10);
+            /* OSM tiles are light; invert + rotate keeps roads/labels legible
+               while the water/river overlays (drawn in SVG, unfiltered) keep
+               their real colours. */
+            --map-filter: invert(1) hue-rotate(180deg) brightness(.92) contrast(.9) saturate(.75);
+            --chart-grid: rgba(230,238,241,.10);
+            --mint-ink: #7fd7ac;
+            --mint-border: #2a5c45;
+            --ok-bg: #143b2c; --ok-ink: #9fe3c3; --ok-border: #2a6b4d;
+            --watch-bg: #3d2f0d; --watch-ink: #f3d489; --watch-border: #6b4f14;
+            --danger-bg: #421c15; --danger-ink: #ffb3a6; --danger-border: #7a2f22;
+            --demo-bg: #3d2f0d; --demo-ink: #f3d489;
+            color-scheme: dark;
         }
         * { box-sizing: border-box; }
         html, body { margin: 0; min-height: 100%; }
@@ -43,7 +94,7 @@
             color: var(--ink);
             overflow-x: hidden;
             background:
-                radial-gradient(circle at 8% 0%, rgba(56, 180, 213, .12), transparent 25rem),
+                radial-gradient(circle at 8% 0%, var(--glow), transparent 25rem),
                 var(--paper);
             /* Thai families first so Thai text uses a proper Thai face where one
                is installed (Android/iOS/Windows all ship one); Latin falls
@@ -60,7 +111,32 @@
         html[lang="th"] body { line-break: normal; overflow-wrap: anywhere; }
         html[lang="th"] .stat-value, html[lang="th"] .flow-value { overflow-wrap: normal; }
         .lang-toggle { padding: 9px 12px; font-size: .78rem; font-weight: 800; white-space: nowrap; }
-        .lang-toggle[data-active-lang="th"] { background: var(--mint); border-color: #b6dfc9; color: #146644; }
+        .lang-toggle[data-active-lang="th"] { background: var(--mint); border-color: var(--mint-border); color: var(--mint-ink); }
+        .theme-toggle { padding: 9px 11px; font-size: .95rem; line-height: 1; }
+        .alerts-panel { margin-bottom: 14px; padding: 18px 20px; border-radius: 16px; background: var(--card); border: 1px solid var(--border); box-shadow: var(--shadow); }
+        .alerts-head { display: flex; justify-content: space-between; align-items: center; gap: 12px; }
+        .alerts-head h2 { margin: 0; font-size: 1.15rem; }
+        .alerts-close { background: transparent; border: 0; color: var(--muted); font-size: 1.1rem; cursor: pointer; padding: 4px 8px; }
+        .alerts-intro { color: var(--muted); font-size: .92rem; line-height: 1.5; margin: 8px 0 12px; }
+        .alerts-server { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; font-size: .9rem; margin-bottom: 12px; }
+        .alerts-server code { background: var(--surface); border: 1px solid var(--border); padding: 4px 8px; border-radius: 8px; font-size: .9rem; }
+        .alerts-server button { padding: 4px 10px; font-size: .8rem; }
+        .alerts-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 10px; }
+        .alerts-topic { border: 1px solid var(--border); border-radius: 12px; padding: 10px 12px; background: var(--surface); display: flex; flex-direction: column; gap: 4px; }
+        .alerts-topic .name { font-weight: 600; font-size: .95rem; }
+        .alerts-topic .desc { color: var(--muted); font-size: .82rem; line-height: 1.4; }
+        .alerts-topic .row { display: flex; align-items: center; gap: 8px; margin-top: 4px; flex-wrap: wrap; }
+        .alerts-topic code { font-size: .82rem; background: var(--card); border: 1px solid var(--border); padding: 2px 6px; border-radius: 6px; }
+        .alerts-topic a { font-size: .82rem; }
+        .alerts-topic.danger { border-color: rgba(220, 38, 38, .45); }
+        .alerts-topic.outlook { border-style: dashed; }
+        .alerts-foot { color: var(--muted); font-size: .82rem; margin: 12px 0 0; line-height: 1.6; }
+        .alerts-disclaimer { display: block; margin-top: 4px; }
+        .leaflet-popup-content-wrapper, .leaflet-popup-tip { background: var(--card); color: var(--ink); }
+        .leaflet-container a.leaflet-popup-close-button { color: var(--muted); }
+        .leaflet-bar a, .leaflet-control-attribution { background: var(--surface); color: var(--ink); border-color: var(--border); }
+        .leaflet-control-attribution a { color: var(--river); }
+        [data-theme="dark"] .leaflet-control-attribution { background: var(--overlay); }
         .shell { max-width: 1500px; margin: 0 auto; padding: 24px; }
         header {
             display: flex; align-items: center; justify-content: space-between; gap: 20px;
@@ -78,16 +154,16 @@
         .header-actions { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; }
         .live-pill {
             display: flex; gap: 8px; align-items: center; padding: 9px 13px; border-radius: 999px;
-            background: var(--mint); color: #146644; font-weight: 750; font-size: .8rem;
+            background: var(--mint); color: var(--mint-ink); font-weight: 750; font-size: .8rem;
         }
         .live-dot { width: 8px; height: 8px; border-radius: 50%; background: #22a66c; box-shadow: 0 0 0 5px rgba(34,166,108,.12); }
-        .live-pill.demo { background: #fdeeda; color: #9a6200; }
+        .live-pill.demo { background: var(--demo-bg); color: var(--demo-ink); }
         .live-pill.demo .live-dot { background: #e6a23c; box-shadow: 0 0 0 5px rgba(230,162,60,.15); }
         button {
-            border: 1px solid var(--border); border-radius: 11px; background: white; color: var(--ink);
+            border: 1px solid var(--border); border-radius: 11px; background: var(--surface); color: var(--ink);
             padding: 10px 14px; cursor: pointer; font-weight: 700; box-shadow: 0 3px 10px rgba(22,52,62,.05);
         }
-        button:hover { border-color: #a9c4bb; transform: translateY(-1px); }
+        button:hover { border-color: var(--river-light); transform: translateY(-1px); }
         button:disabled { opacity: .55; cursor: wait; transform: none; }
         .stats { display: grid; grid-template-columns: repeat(4, minmax(0,1fr)); gap: 14px; margin-bottom: 14px; }
         .stat {
@@ -97,16 +173,31 @@
         .stat-label { color: var(--muted); text-transform: uppercase; letter-spacing: .09em; font-size: .68rem; font-weight: 800; }
         .stat-value { margin-top: 9px; font-size: 1.65rem; font-weight: 800; letter-spacing: -.04em; white-space: nowrap; }
         .stat-note { color: var(--muted); margin-top: 3px; font-size: .77rem; }
+        .stat.stale { border-color: var(--red); background: rgba(204,75,55,.08); }
+        .skill-panel { border: 1px solid var(--border); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: var(--surface-3); }
+        .skill-head { display: flex; justify-content: space-between; align-items: baseline; gap: 12px; flex-wrap: wrap; }
+        .skill-headline { margin: 8px 0 10px; font-weight: 700; font-size: .9rem; }
+        .skill-headline.better { color: var(--green); }
+        .skill-headline.worse { color: var(--amber); }
+        .skill-table-wrap { overflow-x: auto; }
+        .skill-table { border-collapse: collapse; font-size: .76rem; width: 100%; min-width: 560px; }
+        .skill-table th { text-align: left; color: var(--muted); font-weight: 700; font-size: .66rem; text-transform: uppercase; letter-spacing: .06em; padding: 4px 8px; border-bottom: 1px solid var(--border); }
+        .skill-table td { padding: 5px 8px; border-bottom: 1px solid var(--border); white-space: nowrap; }
+        .skill-table tr.current td { font-weight: 700; }
+        .skill-table td.num { text-align: right; font-variant-numeric: tabular-nums; }
+        .skill-table td.dim { color: var(--muted); }
+        .stat.stale .stat-value, .stat.stale .stat-note { color: var(--red); }
         .workspace { display: grid; grid-template-columns: minmax(0, 1fr) 330px; gap: 14px; min-height: 640px; }
         .map-card, .side-card { background: var(--card); border: 1px solid var(--border); border-radius: 19px; box-shadow: var(--shadow); overflow: hidden; }
         .map-card { position: relative; }
-        #station-map { height: 640px; width: 100%; background: #dcebea; }
+        #station-map { height: 640px; width: 100%; background: var(--map-bg); }
+        .leaflet-tile-pane { filter: var(--map-filter); }
         .map-overlay {
             position: absolute; z-index: 500; top: 16px; left: 52px; right: 16px;
             display: flex; justify-content: space-between; align-items: flex-start; pointer-events: none;
         }
         .map-heading, .legend {
-            background: rgba(255,255,255,.93); backdrop-filter: blur(9px); border: 1px solid rgba(207,224,218,.9);
+            background: var(--overlay); backdrop-filter: blur(9px); border: 1px solid var(--overlay-border);
             border-radius: 13px; padding: 11px 13px; box-shadow: 0 7px 20px rgba(22,58,68,.12);
             pointer-events: auto; /* .map-overlay disables events; re-enable for the legend's rain toggle */
         }
@@ -128,7 +219,7 @@
             width: 100%; display: grid; grid-template-columns: 40px minmax(0,1fr) auto; gap: 10px; align-items: center;
             padding: 11px; border: 0; border-radius: 12px; box-shadow: none; text-align: left; background: transparent;
         }
-        .station-row:hover { background: #f1f7f5; transform: none; }
+        .station-row:hover { background: var(--surface-2); transform: none; }
         .station-code { width: 40px; height: 40px; display: grid; place-items: center; border-radius: 11px; color: white; font-size: .68rem; font-weight: 850; }
         .station-code.long { font-size: .55rem; word-break: break-all; line-height: 1.15; padding: 2px; text-align: center; }
         .station-name { overflow: hidden; }
@@ -137,9 +228,9 @@
         .station-name span { color: var(--muted); font-size: .68rem; margin-top: 3px; }
         .flow-value { text-align: right; font-size: .82rem; font-weight: 800; }
         .flow-value span { display: block; color: var(--muted); font-size: .61rem; font-weight: 650; margin-top: 2px; }
-        .loading-panel, .error-panel { position: absolute; z-index: 600; inset: 0; display: grid; place-items: center; background: rgba(243,247,245,.88); }
-        .loading-card { background: white; padding: 18px 22px; border-radius: 14px; box-shadow: var(--shadow); font-weight: 750; }
-        .error-panel { display: none; color: #8d2f22; text-align: center; padding: 25px; }
+        .loading-panel, .error-panel { position: absolute; z-index: 600; inset: 0; display: grid; place-items: center; background: var(--overlay); }
+        .loading-card { background: var(--card); padding: 18px 22px; border-radius: 14px; box-shadow: var(--shadow); font-weight: 750; }
+        .error-panel { display: none; color: var(--red); text-align: center; padding: 25px; }
         .marker-wrap { background: none; border: 0; }
         .flow-marker {
             --marker-color: #087da5; --marker-size: 26px;
@@ -179,7 +270,7 @@
         .risk-chips { display: flex; gap: 6px; }
         .risk-chip { flex: 1; text-align: center; border-radius: 8px; padding: 5px 4px; font-size: .64rem; font-weight: 800; color: white; }
         .risk-chip span { display: block; font-weight: 650; font-size: .58rem; opacity: .85; }
-        .p1-outlook { border: 1px solid var(--border); border-left: 4px solid var(--river); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: #f7fbfa; }
+        .p1-outlook { border: 1px solid var(--border); border-left: 4px solid var(--river); border-radius: 12px; padding: 12px 14px; margin-top: 14px; background: var(--surface-3); }
         .p1-outlook-head { display: flex; justify-content: space-between; align-items: center; gap: 10px; flex-wrap: wrap; }
         .p1-outlook-head strong { font-size: .88rem; }
         .p1-peak { color: var(--muted); font-size: .76rem; }
@@ -193,16 +284,16 @@
         .popup h3 { margin: 4px 0 2px; font-size: 1rem; }
         .popup-th { color: var(--muted); font-size: .74rem; }
         .popup-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 8px; margin-top: 12px; }
-        .popup-metric { background: #f1f7f5; padding: 8px; border-radius: 9px; }
+        .popup-metric { background: var(--surface-2); padding: 8px; border-radius: 9px; }
         .popup-metric span { display: block; color: var(--muted); font-size: .62rem; }
         .popup-metric strong { display: block; margin-top: 2px; font-size: .86rem; }
         .popup-time { margin-top: 9px; color: var(--muted); font-size: .64rem; }
         /* Verdict banner: light tinted backgrounds with dark ink (contrast-safe) */
-        #flood-verdict.ok { background: #e2f3ea; color: #0c5138; border-color: #bfe3d2; }
-        #flood-verdict.watch { background: #fdf1dc; color: #6b4a05; border-color: #f0d9a8; }
-        #flood-verdict.danger { background: #fbe3de; color: #7c1d10; border-color: #f2c0b6; }
+        #flood-verdict.ok { background: var(--ok-bg); color: var(--ok-ink); border-color: var(--ok-border); }
+        #flood-verdict.watch { background: var(--watch-bg); color: var(--watch-ink); border-color: var(--watch-border); }
+        #flood-verdict.danger { background: var(--danger-bg); color: var(--danger-ink); border-color: var(--danger-border); }
         #legend-toggle {
-            display: none; pointer-events: auto; background: rgba(255,255,255,.93); border: 1px solid rgba(207,224,218,.9);
+            display: none; pointer-events: auto; background: var(--overlay); border: 1px solid var(--overlay-border);
             border-radius: 11px; padding: 8px 12px; font-size: .72rem; font-weight: 800; color: var(--ink);
             box-shadow: 0 7px 20px rgba(22,58,68,.12);
         }
@@ -275,10 +366,34 @@
             <div class="live-pill" id="live-pill"><span class="live-dot"></span> <span id="live-pill-text" data-i18n="pill.live">LIVE DATA</span></div>
             <button id="refresh-button" type="button" data-i18n="action.refresh">↻ Refresh</button>
             <button id="lang-toggle" class="lang-toggle" type="button" data-active-lang="en" aria-label="Switch to Thai">ไทย</button>
+            <button id="theme-toggle" class="theme-toggle" type="button" data-i18n-aria="theme.toggle" aria-label="Switch to dark mode" title="Switch to dark mode">🌙</button>
+            <button id="alerts-button" type="button" data-i18n="alerts.button" style="display:none">🔔 Get alerts</button>
             <button id="replay-2024" type="button" data-i18n="replay.start">▶ Replay Oct 2024 flood</button>
         </div>
     </header>
 
+    <section id="alerts-panel" class="alerts-panel" style="display:none" aria-labelledby="alerts-title">
+        <div class="alerts-head">
+            <h2 id="alerts-title" data-i18n="alerts.title">Flood alerts on your phone</h2>
+            <button type="button" class="alerts-close" id="alerts-close" data-i18n-aria="alerts.close" aria-label="Close">✕</button>
+        </div>
+        <p class="alerts-intro" data-i18n="alerts.intro">Free push notifications when a gauge crosses its warning or danger level, and an all-clear when it drops back. No account: install the ntfy app (iOS / Android / any browser), add the server, subscribe to the topics you want. You get a message only when something changes: a few per flood, none in a quiet season.</p>
+        <div class="alerts-server">
+            <span data-i18n="alerts.server">Server</span>
+            <code id="alerts-server-url"></code>
+            <button type="button" id="alerts-copy" data-i18n="alerts.copy">Copy</button>
+        </div>
+        <div class="alerts-grid" id="alerts-topics"></div>
+        <p class="alerts-foot">
+            <span data-i18n="alerts.apps">Apps:</span>
+            <a href="https://apps.apple.com/us/app/ntfy/id1625396347" target="_blank" rel="noopener">iOS</a> ·
+            <a href="https://play.google.com/store/apps/details?id=io.heckel.ntfy" target="_blank" rel="noopener">Android</a> ·
+            <a href="https://f-droid.org/en/packages/io.heckel.ntfy/" target="_blank" rel="noopener">F-Droid</a> ·
+            <a id="alerts-web-link" href="#" target="_blank" rel="noopener" data-i18n="alerts.web">Web (no install)</a>
+            <span class="alerts-disclaimer" data-i18n="alerts.disclaimer">Unofficial community service, best effort. For official warnings follow ThaiWater / TMD / your district office.</span>
+        </p>
+    </section>
+
     <section id="flood-verdict" role="status" aria-live="polite" style="display:none;margin-bottom:14px;padding:15px 18px;border-radius:16px;border:1px solid;display:none">
         <div style="display:flex;gap:12px;align-items:baseline;flex-wrap:wrap">
             <strong id="verdict-icon" style="font-size:1.2rem"></strong>
@@ -321,7 +436,7 @@
                     <div class="legend-row"><i class="swatch" style="background:#2a1668;border-radius:50%"></i> <span data-i18n="legend.rain.extreme">Extreme &gt; 150</span></div>
                     <div class="legend-title" style="margin-top:10px" data-i18n="legend.other">Other markers</div>
                     <div class="legend-row"><i class="swatch" style="background:#7b8f94"></i> <span data-i18n="legend.other.nodata">Gauge · no recent data</span></div>
-                    <div class="legend-row"><span style="font-weight:800;color:#0f6844">+</span> <span data-i18n="legend.other.sensor">Water-level sensor · colour = % of bank height</span></div>
+                    <div class="legend-row"><span style="font-weight:800;color:var(--green)">+</span> <span data-i18n="legend.other.sensor">Water-level sensor · colour = % of bank height</span></div>
                 </div>
                 </div>
             </div>
@@ -332,7 +447,7 @@
 
         <aside class="side-card">
             <div class="side-head"><h2 data-i18n="side.title">Current station flow</h2><p data-i18n="side.subtitle">Select a station to locate it and load its history</p>
-                <input type="search" id="station-search" data-i18n-placeholder="search.placeholder" data-i18n-aria="search.aria" placeholder="Search stations · code, name, river…" aria-label="Search stations" style="margin-top:10px;width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:white;font-size:.8rem">
+                <input type="search" id="station-search" data-i18n-placeholder="search.placeholder" data-i18n-aria="search.aria" placeholder="Search stations · code, name, river…" aria-label="Search stations" style="margin-top:10px;width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink);font-size:.8rem">
             </div>
             <div class="station-list" id="river-flow" aria-live="polite"></div>
             <div class="side-head" id="sensors-head" style="cursor:pointer" role="button" tabindex="0" aria-expanded="false" aria-controls="thaiwater-sensors" data-i18n-title="sensors.tip" title="Show / hide the ThaiWater/HII station list"><h2><span data-i18n="sensors.title">Additional basin stations</span> <span id="sensors-arrow" style="color:var(--muted);font-size:.8rem">▸</span></h2><p id="thaiwater-count" data-i18n="sensors.loading">Loading ThaiWater/HII stations…</p></div>
@@ -356,6 +471,16 @@
             <div class="p1-peak" style="margin-top:7px" data-i18n="outlook.explainer">Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.</div>
         </div>
         <button type="button" class="zones-button" id="forecast-expand" style="display:none;margin-top:12px">Show all station forecasts ▾</button>
+        <div class="skill-panel" id="skill-panel" style="display:none">
+            <div class="skill-head">
+                <strong data-i18n="skill.title">Is the model getting better?</strong>
+                <span class="subtitle" id="skill-sub"></span>
+            </div>
+            <div class="skill-headline" id="skill-headline"></div>
+            <p class="subtitle skill-caveat" id="skill-caveat" style="margin:-4px 0 10px"></p>
+            <div class="skill-table-wrap"><table class="skill-table" id="skill-table"></table></div>
+            <p class="subtitle" style="margin:8px 0 0" data-i18n="skill.explain">Every hour the deployed model's 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.</p>
+        </div>
         <div class="forecast-grid" id="forecast-grid" style="display:none"></div>
     </section>
 
@@ -363,10 +488,10 @@
         <div style="display:flex;justify-content:space-between;align-items:center;gap:14px;flex-wrap:wrap">
             <div><h2 id="history-title" style="margin:0;font-size:1rem" data-i18n="history.title">Station history</h2><p id="history-status" class="subtitle" data-i18n="history.status">Select a station to load the last 7 days</p></div>
             <div class="history-controls" style="display:flex;gap:8px;align-items:center;flex-wrap:wrap">
-                <select id="history-range" style="padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:white"><option value="24" data-i18n="range.24h">Last 24 hours</option><option value="168" selected data-i18n="range.7d">Last 7 days</option><option value="720" data-i18n="range.30d">Last 30 days</option><option value="2160" data-i18n="range.90d">Last 90 days</option><option value="876000" data-i18n="range.all">All time</option><option value="custom" hidden data-i18n="range.custom">Custom range</option></select>
-                <input type="date" id="history-start" data-i18n-title="range.from" title="From date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:white">
+                <select id="history-range" style="padding:9px 12px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink)"><option value="24" data-i18n="range.24h">Last 24 hours</option><option value="168" selected data-i18n="range.7d">Last 7 days</option><option value="720" data-i18n="range.30d">Last 30 days</option><option value="2160" data-i18n="range.90d">Last 90 days</option><option value="876000" data-i18n="range.all">All time</option><option value="custom" hidden data-i18n="range.custom">Custom range</option></select>
+                <input type="date" id="history-start" data-i18n-title="range.from" title="From date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink)">
                 <span style="color:var(--muted)">–</span>
-                <input type="date" id="history-end" data-i18n-title="range.to" title="To date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:white">
+                <input type="date" id="history-end" data-i18n-title="range.to" title="To date" style="padding:8px 10px;border:1px solid var(--border);border-radius:10px;background:var(--surface);color:var(--ink)">
             </div>
         </div>
         <div style="height:260px;margin-top:14px;overflow:hidden;position:relative"><canvas id="history-chart" data-i18n-aria="aria.chart" aria-label="Historical water level and discharge chart" style="display:block"></canvas><div id="history-placeholder" data-i18n="history.placeholder" style="position:absolute;inset:0;display:grid;place-items:center;color:var(--muted);font-size:.85rem;text-align:center;padding:20px">Click any station on the map or in the list to see its history</div></div>
@@ -395,7 +520,7 @@
 <script>
 (function () {
     'use strict';
-    const state = { map: null, layers: [], markers: new Map(), hasFit: false, historyChart: null, selectedStation: null, historyRequestId: 0, p1Stages: null, p1Now: null, rainLayer: null, useDates: false, forecastExpanded: false };
+    const state = { map: null, layers: [], markers: new Map(), hasFit: false, historyChart: null, selectedStation: null, historyRequestId: 0, p1Stages: null, p1Now: null, p1NowAt: null, rainLayer: null, useDates: false, forecastExpanded: false };
     const $ = (id) => document.getElementById(id);
 
     // ---- Localisation -----------------------------------------------------
@@ -407,6 +532,8 @@
             'app.title': 'Ping River Live Monitor',
             'app.subtitle': 'Current water level and discharge across Northern Thailand',
             'pill.live': 'LIVE DATA',
+            'theme.toggle': 'Toggle dark mode',
+            'pill.stale': '⚠ STALE FEED',
             'pill.replay': '⏪ 2024 REPLAY',
             'pill.sim': '⚠ SIMULATION',
             'action.refresh': '↻ Refresh',
@@ -439,7 +566,9 @@
             'stat.stress.tip': 'How full the river channel is at the busiest gauge — 100% means water reaches the top of the bank',
             'stat.updated': 'Last updated',
             'stat.updated.note': 'Loading latest readings',
-            'stat.updated.ago': (date, mins) => `${date} · ${mins} min ago`,
+            'stat.updated.ago': (date, mins) => `${date} · ${mins} min ago (ICT)`,
+            'stat.updated.agoh': (date, hours) => `${date} · ${hours} h ago (ICT)`,
+            'stat.updated.stale': (date, hours) => `⚠ Feed stale · last reading ${date}, ${hours} h ago`,
             'stat.updated.none': 'No timestamp available',
             'map.title': 'Station flow map',
             'map.subtitle': 'River width, colour & dash speed follow live discharge',
@@ -491,6 +620,51 @@
             'forecast.chip.peak': (lvl) => ` · peak ~${lvl} m`,
             'forecast.chip.heuristic': ' · heuristic fallback',
             'forecast.expand': (n) => `Show all ${n} station forecasts ▾`,
+            'skill.title': 'Is the model getting better?',
+            'skill.sub': (n, since) => `${n} verified 24 h forecasts for P.1 since ${since}`,
+            'skill.explain': 'Every hour the deployed model\'s 24 h peak forecast for P.1 is stored; once those 24 hours have passed it is compared with what the river actually did. "Skill" is how much better the model was than assuming the level stays where it is (0 = no better, 1 = perfect). Versions retrained on more data appear as new rows, so improvement, or its absence, is visible here rather than claimed.',
+            'skill.better': (v, prev, d) => `Current model ${v} is more accurate than ${prev}: peak error ${d} cm lower on the hours it has served.`,
+            'skill.worse': (v, prev, d) => `Current model ${v} has a higher peak error than ${prev} so far (+${d} cm).`,
+            'skill.caveat.quiet': 'All verified hours so far were below 2 m: this measures quiet-river accuracy only. The model is built and judged for flood onset (lead time before 3.70 m), which no quiet week can test — see the backtests in the documentation.',
+            'skill.caveat.regime': 'Versions served different weeks; the ≥ 2 m column compares them on the hours that matter.',
+            'alerts.button': '🔔 Get alerts',
+            'alerts.title': 'Flood alerts on your phone',
+            'alerts.intro': 'Free push notifications when a gauge crosses its warning or danger level, and an all-clear when it drops back. No account: install the ntfy app (iOS / Android / any browser), add the server, subscribe to the topics you want. You get a message only when something changes: a few per flood, none in a quiet season.',
+            'alerts.server': 'Server',
+            'alerts.copy': 'Copy',
+            'alerts.copied': 'Copied',
+            'alerts.close': 'Close',
+            'alerts.apps': 'Apps:',
+            'alerts.web': 'Web (no install)',
+            'alerts.disclaimer': 'Unofficial community service, best effort. For official warnings follow ThaiWater / TMD / your district office.',
+            'alerts.subscribe': 'Subscribe in app',
+            'alerts.t.warning': 'Any gauge: warning level',
+            'alerts.t.warning.d': 'One message when any Ping River gauge crosses its warning level, and when levels fall back. The one to pick if unsure.',
+            'alerts.t.danger': 'Any gauge: danger level',
+            'alerts.t.danger.d': 'Only the serious crossings, basin-wide. Highest priority: rings through Do Not Disturb on most phones.',
+            'alerts.t.p1.warning': 'Chiang Mai city (P.1) warning',
+            'alerts.t.p1.warning.d': 'Nawarat Bridge crosses 3.70 m (stage 1: low-lying riverside areas), and the all-clear.',
+            'alerts.t.p1.danger': 'Chiang Mai city (P.1) danger',
+            'alerts.t.p1.danger.d': 'Nawarat Bridge crosses 4.20 m (stage 5: inner city districts).',
+            'alerts.t.p103.warning': 'Ring Road 3 (P.103) warning',
+            'alerts.t.p103.warning.d': 'Downstream city gauge crosses 5.95 m.',
+            'alerts.t.outlook': 'Early warning (model forecast)',
+            'alerts.t.outlook.d': 'Experimental: the forecast model gives a ≥ 50 % chance that P.1 reaches its warning level within 24 h. Up to ~13 h earlier than the gauge, but it can be wrong.',
+            'alerts.t.status': 'Monitor status',
+            'alerts.t.status.d': 'Gauge feed stale / recovered. For people who rely on the dashboard.',
+            'skill.single': (v) => `Only ${v} has enough verified hours yet; the next retrain adds a row to compare.`,
+            'skill.young': (v, n, min) => `${v} has ${n} verified hours; a comparison needs ${min}.`,
+            'skill.none': 'No verified forecasts yet — the first appear 24 h after a model starts serving.',
+            'skill.col.version': 'Model',
+            'skill.col.period': 'Served',
+            'skill.col.n': 'Hours',
+            'skill.col.mae': 'Peak error',
+            'skill.col.bias': 'Bias',
+            'skill.col.pers': 'Persistence',
+            'skill.col.skill': 'Skill',
+            'skill.col.high': '≥ 2 m error',
+            'skill.cm': (v) => `${v} cm`,
+            'skill.na': '—',
             'forecast.collapse': 'Hide station forecasts ▴',
             'outlook.title': 'Chiang Mai city flood outlook · P.1 Nawarat Bridge',
             'outlook.explainer': 'Chance the river reaches each official inundation stage within 24 h — city flooding begins at stage 1 (3.70 m); each stage floods additional districts.',
@@ -579,6 +753,8 @@
             'app.title': 'ติดตามระดับน้ำปิงแบบเรียลไทม์',
             'app.subtitle': 'ระดับน้ำและอัตราการไหลปัจจุบันทั่วภาคเหนือของประเทศไทย',
             'pill.live': 'ข้อมูลสด',
+            'theme.toggle': 'สลับโหมดมืด/สว่าง',
+            'pill.stale': '⚠ ข้อมูลไม่อัปเดต',
             'pill.replay': '⏪ ย้อนเหตุการณ์ 2567',
             'pill.sim': '⚠ การจำลอง',
             'action.refresh': '↻ รีเฟรช',
@@ -611,7 +787,9 @@
             'stat.stress.tip': 'ระดับความเต็มของลำน้ำที่สถานีที่มีน้ำมากที่สุด — 100% หมายถึงน้ำถึงระดับตลิ่ง',
             'stat.updated': 'อัปเดตล่าสุด',
             'stat.updated.note': 'กำลังโหลดข้อมูลล่าสุด',
-            'stat.updated.ago': (date, mins) => `${date} · ${mins} นาทีที่แล้ว`,
+            'stat.updated.ago': (date, mins) => `${date} · ${mins} นาทีที่แล้ว (เวลาไทย)`,
+            'stat.updated.agoh': (date, hours) => `${date} · ${hours} ชั่วโมงที่แล้ว (เวลาไทย)`,
+            'stat.updated.stale': (date, hours) => `⚠ ข้อมูลไม่อัปเดต · ค่าล่าสุด ${date}, ${hours} ชั่วโมงที่แล้ว`,
             'stat.updated.none': 'ไม่มีข้อมูลเวลา',
             'map.title': 'แผนที่การไหลของน้ำ',
             'map.subtitle': 'ความกว้าง สี และความเร็วเส้นประของแม่น้ำแสดงอัตราการไหลจริง',
@@ -663,6 +841,51 @@
             'forecast.chip.peak': (lvl) => ` · ระดับสูงสุดประมาณ ${lvl} ม.`,
             'forecast.chip.heuristic': ' · ใช้การประมาณอย่างง่าย',
             'forecast.expand': (n) => `แสดงพยากรณ์ทั้ง ${n} สถานี ▾`,
+            'skill.title': 'โมเดลแม่นยำขึ้นหรือไม่?',
+            'skill.sub': (n, since) => `พยากรณ์ 24 ชม. ของ P.1 ที่ตรวจสอบแล้ว ${n} ครั้ง ตั้งแต่ ${since}`,
+            'skill.explain': 'ทุกชั่วโมงระบบบันทึกค่าพยากรณ์ระดับน้ำสูงสุดใน 24 ชม. ของ P.1 ไว้ เมื่อครบ 24 ชม. จึงนำมาเทียบกับระดับน้ำจริง "ทักษะ" คือโมเดลดีกว่าการสมมติว่าระดับน้ำคงที่มากเพียงใด (0 = ไม่ดีกว่า, 1 = สมบูรณ์แบบ) โมเดลที่ฝึกใหม่ด้วยข้อมูลมากขึ้นจะปรากฏเป็นแถวใหม่ จึงเห็นได้ว่าดีขึ้นจริงหรือไม่',
+            'skill.better': (v, prev, d) => `โมเดลปัจจุบัน ${v} แม่นยำกว่า ${prev}: ค่าคลาดเคลื่อนต่ำกว่า ${d} ซม. ในช่วงที่ให้บริการ`,
+            'skill.worse': (v, prev, d) => `โมเดลปัจจุบัน ${v} มีค่าคลาดเคลื่อนสูงกว่า ${prev} (+${d} ซม.)`,
+            'skill.caveat.quiet': 'ชั่วโมงที่ตรวจสอบทั้งหมดอยู่ต่ำกว่า 2 ม.: วัดได้เพียงความแม่นยำช่วงน้ำปกติ โมเดลถูกสร้างและประเมินสำหรับช่วงน้ำเริ่มท่วม (เวลาเตือนล่วงหน้าก่อน 3.70 ม.) ซึ่งสัปดาห์ปกติทดสอบไม่ได้ — ดูผลทดสอบย้อนหลังในเอกสาร',
+            'skill.caveat.regime': 'แต่ละเวอร์ชันให้บริการคนละช่วงเวลา คอลัมน์ ≥ 2 ม. เปรียบเทียบเฉพาะชั่วโมงที่สำคัญ',
+            'alerts.button': '🔔 รับการแจ้งเตือน',
+            'alerts.title': 'แจ้งเตือนน้ำท่วมบนมือถือของคุณ',
+            'alerts.intro': 'การแจ้งเตือนฟรีเมื่อระดับน้ำที่สถานีใดข้ามระดับเฝ้าระวังหรือระดับอันตราย และแจ้งเมื่อกลับสู่ปกติ ไม่ต้องสมัครสมาชิก: ติดตั้งแอป ntfy (iOS / Android / เบราว์เซอร์) เพิ่มเซิร์ฟเวอร์ แล้วเลือกหัวข้อที่ต้องการ คุณจะได้รับข้อความเฉพาะเมื่อมีการเปลี่ยนแปลง: ไม่กี่ข้อความต่อเหตุการณ์น้ำท่วม และไม่มีเลยในช่วงปกติ',
+            'alerts.server': 'เซิร์ฟเวอร์',
+            'alerts.copy': 'คัดลอก',
+            'alerts.copied': 'คัดลอกแล้ว',
+            'alerts.close': 'ปิด',
+            'alerts.apps': 'แอป:',
+            'alerts.web': 'เว็บ (ไม่ต้องติดตั้ง)',
+            'alerts.disclaimer': 'บริการชุมชนอย่างไม่เป็นทางการ พยายามอย่างดีที่สุด สำหรับคำเตือนอย่างเป็นทางการโปรดติดตาม ThaiWater / กรมอุตุนิยมวิทยา / สำนักงานอำเภอของคุณ',
+            'alerts.subscribe': 'สมัครในแอป',
+            'alerts.t.warning': 'สถานีใดก็ได้: ระดับเฝ้าระวัง',
+            'alerts.t.warning.d': 'หนึ่งข้อความเมื่อสถานีใดในแม่น้ำปิงข้ามระดับเฝ้าระวัง และเมื่อระดับน้ำลดลง หากไม่แน่ใจให้เลือกอันนี้',
+            'alerts.t.danger': 'สถานีใดก็ได้: ระดับอันตราย',
+            'alerts.t.danger.d': 'เฉพาะการข้ามระดับที่ร้ายแรง ทั้งลุ่มน้ำ ความสำคัญสูงสุด: ดังผ่านโหมดห้ามรบกวนในโทรศัพท์ส่วนใหญ่',
+            'alerts.t.p1.warning': 'เมืองเชียงใหม่ (P.1) ระดับเฝ้าระวัง',
+            'alerts.t.p1.warning.d': 'สะพานนวรัฐข้าม 3.70 ม. (ระยะที่ 1: พื้นที่ริมน้ำที่ต่ำ) และแจ้งเมื่อกลับสู่ปกติ',
+            'alerts.t.p1.danger': 'เมืองเชียงใหม่ (P.1) ระดับอันตราย',
+            'alerts.t.p1.danger.d': 'สะพานนวรัฐข้าม 4.20 ม. (ระยะที่ 5: ย่านใจกลางเมือง)',
+            'alerts.t.p103.warning': 'ถนนวงแหวน 3 (P.103) ระดับเฝ้าระวัง',
+            'alerts.t.p103.warning.d': 'สถานีท้ายเมืองข้าม 5.95 ม.',
+            'alerts.t.outlook': 'เตือนล่วงหน้า (แบบจำลองพยากรณ์)',
+            'alerts.t.outlook.d': 'ทดลอง: แบบจำลองพยากรณ์ให้โอกาส ≥ 50% ที่ P.1 จะถึงระดับเฝ้าระวังภายใน 24 ชม. เร็วกว่าสถานีวัดได้ถึง ~13 ชม. แต่อาจผิดพลาดได้',
+            'alerts.t.status': 'สถานะระบบ',
+            'alerts.t.status.d': 'ข้อมูลสถานีล่าช้า / กลับมาปกติ สำหรับผู้ที่พึ่งพาแดชบอร์ด',
+            'skill.single': (v) => `มีเพียง ${v} ที่มีข้อมูลตรวจสอบเพียงพอ การฝึกครั้งถัดไปจะเพิ่มแถวให้เปรียบเทียบ`,
+            'skill.young': (v, n, min) => `${v} มีข้อมูลตรวจสอบ ${n} ชั่วโมง ต้องการอย่างน้อย ${min} เพื่อเปรียบเทียบ`,
+            'skill.none': 'ยังไม่มีพยากรณ์ที่ตรวจสอบได้ — จะเริ่มมี 24 ชม. หลังโมเดลเริ่มทำงาน',
+            'skill.col.version': 'โมเดล',
+            'skill.col.period': 'ช่วงเวลา',
+            'skill.col.n': 'ชั่วโมง',
+            'skill.col.mae': 'คลาดเคลื่อน',
+            'skill.col.bias': 'อคติ',
+            'skill.col.pers': 'ระดับคงที่',
+            'skill.col.skill': 'ทักษะ',
+            'skill.col.high': 'คลาดเคลื่อน ≥ 2 ม.',
+            'skill.cm': (v) => `${v} ซม.`,
+            'skill.na': '—',
             'forecast.collapse': 'ซ่อนพยากรณ์รายสถานี ▴',
             'outlook.title': 'แนวโน้มน้ำท่วมเมืองเชียงใหม่ · P.1 สะพานนวรัฐ',
             'outlook.explainer': 'โอกาสที่ระดับน้ำจะถึงแต่ละระดับการท่วมตามประกาศทางการภายใน 24 ชม. — น้ำเริ่มท่วมเมืองที่ระดับ 1 (3.70 ม.) และแต่ละระดับจะท่วมพื้นที่เพิ่มขึ้น',
@@ -774,6 +997,60 @@
         return typeof value === 'function' ? value(...args) : value;
     }
 
+    // ---- public push notifications (ntfy) -------------------------------------
+    let ALERTS_CFG = null;
+    const ALERT_TOPICS = [
+        { key: 'warning',     topic: 'warning',     cls: '' },
+        { key: 'danger',      topic: 'danger',      cls: 'danger' },
+        { key: 'p1.warning',  topic: 'p1-warning',  cls: '' },
+        { key: 'p1.danger',   topic: 'p1-danger',   cls: 'danger' },
+        { key: 'p103.warning', topic: 'p103-warning', cls: '' },
+        { key: 'outlook',     topic: 'p1-outlook',  cls: 'outlook' },
+        { key: 'status',      topic: 'status',      cls: '' },
+    ];
+    async function loadAlertsConfig() {
+        try {
+            const r = await fetch('/api/notifications');
+            if (!r.ok) return;
+            const cfg = await r.json();
+            if (!cfg.enabled || !cfg.server) return;
+            ALERTS_CFG = cfg;
+            $('alerts-button').style.display = '';
+            renderAlertsPanel();
+        } catch (e) { /* no notifications configured */ }
+    }
+    function renderAlertsPanel() {
+        if (!ALERTS_CFG) return;
+        const server = ALERTS_CFG.server.replace(/\/$/, '');
+        const host = server.replace(/^https?:\/\//, '');
+        $('alerts-server-url').textContent = host;
+        $('alerts-web-link').href = server + '/' + ALERTS_CFG.prefix + '-warning';
+        $('alerts-topics').innerHTML = ALERT_TOPICS.map(tp => {
+            const full = ALERTS_CFG.prefix + '-' + tp.topic;
+            const url = server + '/' + full;
+            return `<div class="alerts-topic ${tp.cls}">
+                <div class="name">${esc(t('alerts.t.' + tp.key))}</div>
+                <div class="desc">${esc(t('alerts.t.' + tp.key + '.d'))}</div>
+                <div class="row"><code>${esc(full)}</code> <a href="ntfy://${esc(host)}/${esc(full)}">${esc(t('alerts.subscribe'))}</a> · <a href="${esc(url)}" target="_blank" rel="noopener">web</a></div>
+            </div>`;
+        }).join('');
+    }
+    function esc(x) { return String(x).replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c])); }
+    $('alerts-button').addEventListener('click', () => {
+        const p = $('alerts-panel');
+        const open = p.style.display === 'none';
+        p.style.display = open ? '' : 'none';
+        if (open) { renderAlertsPanel(); p.scrollIntoView({ behavior: 'smooth', block: 'start' }); }
+    });
+    $('alerts-close').addEventListener('click', () => { $('alerts-panel').style.display = 'none'; });
+    $('alerts-copy').addEventListener('click', async () => {
+        try {
+            await navigator.clipboard.writeText(ALERTS_CFG ? ALERTS_CFG.server : '');
+            $('alerts-copy').textContent = t('alerts.copied');
+            setTimeout(() => { $('alerts-copy').textContent = t('alerts.copy'); }, 1500);
+        } catch (e) { /* clipboard blocked */ }
+    });
+
     function applyTranslations() {
         document.documentElement.lang = state.lang;
         document.querySelectorAll('[data-i18n]').forEach((el) => {
@@ -801,7 +1078,34 @@
         }
     }
 
+    // Theme: explicit choice persists; otherwise follow the OS and track its changes.
+    const THEME_KEY = 'ping-river-theme';
+    const osDark = window.matchMedia('(prefers-color-scheme: dark)');
+    function currentTheme() { return document.documentElement.dataset.theme === 'dark' ? 'dark' : 'light'; }
+    function applyTheme(theme, persist) {
+        document.documentElement.dataset.theme = theme;
+        const button = $('theme-toggle');
+        if (button) {
+            button.textContent = theme === 'dark' ? '☀️' : '🌙';
+        }
+        if (persist) { try { localStorage.setItem(THEME_KEY, theme); } catch (e) { /* private mode */ } }
+        // Chart.js reads colours at construction: rebuild the open chart
+        if (state.historyChart && state.selectedStation) loadHistory(state.selectedStation);
+    }
+    (() => {
+        let saved = null;
+        try { saved = localStorage.getItem(THEME_KEY); } catch (e) { /* private mode */ }
+        applyTheme(saved === 'dark' || saved === 'light' ? saved : (osDark.matches ? 'dark' : 'light'), false);
+        osDark.addEventListener('change', (e) => {
+            let pinned = null;
+            try { pinned = localStorage.getItem(THEME_KEY); } catch (err) { /* ignore */ }
+            if (!pinned) applyTheme(e.matches ? 'dark' : 'light', false);
+        });
+    })();
+    function cssVar(name) { return getComputedStyle(document.documentElement).getPropertyValue(name).trim(); }
+
     function setLang(lang) {
+        setTimeout(renderAlertsPanel, 0);
         state.lang = lang;
         try { localStorage.setItem(LANG_KEY, lang); } catch (e) { /* private mode */ }
         applyTranslations();
@@ -841,6 +1145,25 @@
     // what the replay label ("ต.ค. 2567") already says — pinning Gregorian here
     // put two different year systems on the same screen.
     function loc() { return state.lang === 'th' ? 'th-TH' : 'en-GB'; }
+    // Every timestamp the API emits is Asia/Bangkok wall-clock WITHOUT an
+    // offset ("2026-09-12T02:00:00"). new Date() on such a string uses the
+    // browser's own zone, so a viewer in Europe read a 02:00 ICT reading as
+    // five hours in the future and saw "0 min ago" forever. Pin the offset
+    // here and always format with timeZone: TZ so the site shows river time
+    // no matter where it is opened.
+    const TZ = 'Asia/Bangkok';
+    const STALE_AFTER_MIN = 180;
+    function parseTs(value) {
+        if (value == null || value === '') return null;
+        if (value instanceof Date) return value;
+        const text = String(value).trim();
+        const naive = /^\d{4}-\d\d-\d\d[T ]\d\d:\d\d(:\d\d(\.\d+)?)?$/.test(text);
+        const date = new Date(naive ? text.replace(' ', 'T') + '+07:00' : text);
+        return Number.isNaN(date.getTime()) ? null : date;
+    }
+    // Bucket keys in river-local time (a Bangkok calendar day, not a UTC one)
+    const dayKey = (value) => parseTs(value).toLocaleDateString('en-CA', { timeZone: TZ });
+    const hourKey = (value) => `${dayKey(value)}-${parseTs(value).toLocaleTimeString('en-GB', { timeZone: TZ, hour: '2-digit' })}`;
     // Metre abbreviation: "ม." reads as a unit in Thai, "m" mid-sentence does not.
     function metres(value, digits = 2) {
         return `${Number(value).toFixed(digits)} ${t('unit.m')}`;
@@ -885,7 +1208,7 @@
             const mm = r.rain_24h == null ? null : Number(r.rain_24h);
             const bin = rainBin(mm);
             const name = (state.lang === 'th' ? r.name_th || r.name_en : r.name_en || r.name_th) || r.oldcode || `Station ${r.station_id}`;
-            const time = r.timestamp ? new Date(r.timestamp).toLocaleString(loc(), { dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
+            const time = r.timestamp ? parseTs(r.timestamp).toLocaleString(loc(), { timeZone: TZ, dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
             L.circleMarker([r.latitude, r.longitude], {
                 radius: bin.radius, color: '#ffffff', weight: 1.5,
                 fillColor: bin.color, fillOpacity: bin.opacity ?? .85
@@ -912,7 +1235,7 @@
         const latest = new Map();
         measurements.forEach((item) => {
             const prior = latest.get(item.station_code);
-            if (!prior || new Date(item.timestamp) > new Date(prior.timestamp)) latest.set(item.station_code, item);
+            if (!prior || parseTs(item.timestamp) > parseTs(prior.timestamp)) latest.set(item.station_code, item);
         });
         return latest;
     }
@@ -945,7 +1268,7 @@
     function buildPopup(station, measurement) {
         const flow = measurement ? measurement.discharge : null;
         const level = measurement ? measurement.water_level : null;
-        const time = measurement ? new Date(measurement.timestamp).toLocaleString(loc(), { dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
+        const time = measurement ? parseTs(measurement.timestamp).toLocaleString(loc(), { timeZone: TZ, dateStyle: 'medium', timeStyle: 'short' }) : t('popup.noreading');
         // Station names are bilingual in the data: lead with the reader's language
         const primary = state.lang === 'th' ? station.thai_name : station.english_name;
         const secondary = state.lang === 'th' ? station.english_name : station.thai_name;
@@ -1125,8 +1448,7 @@
             const downsample = (data) => {
                 const buckets = {};
                 data.forEach((row) => {
-                    const date = new Date(row.timestamp);
-                    const key = `${date.getUTCFullYear()}-${date.getUTCMonth()}-${date.getUTCDate()}`;
+                    const key = dayKey(row.timestamp);
                     if (!buckets[key]) buckets[key] = { ts: row.timestamp, discharge: [], level: [] };
                     const b = buckets[key];
                     if (row.discharge != null) b.discharge.push(row.discharge);
@@ -1147,12 +1469,7 @@
                 const fr = await fetch(`/api/forecast/history/${encodeURIComponent(stationCode)}?${query}&horizon=24`);
                 const forecastRows = fr.ok ? await fr.json() : [];
                 if (forecastRows.length) {
-                    const keyOf = (value) => {
-                        const d = new Date(value);
-                        return rows.length > 2000
-                            ? `${d.getUTCFullYear()}-${d.getUTCMonth()}-${d.getUTCDate()}`
-                            : `${d.getUTCFullYear()}-${d.getUTCMonth()}-${d.getUTCDate()}-${d.getUTCHours()}`;
-                    };
+                    const keyOf = (value) => rows.length > 2000 ? dayKey(value) : hourKey(value);
                     const byKey = new Map();
                     forecastRows.forEach((r) => {
                         if (r.predicted_max_level == null) return;
@@ -1177,7 +1494,7 @@
             state.historyChart = new Chart($('history-chart'), {
                 type: 'line',
                 data: {
-                    labels: sampled.map((row) => new Date(row.timestamp).toLocaleString(loc(), { timeZone: 'Asia/Bangkok', month: 'short', day: 'numeric', year: '2-digit', hour: '2-digit' })),
+                    labels: sampled.map((row) => parseTs(row.timestamp).toLocaleString(loc(), { timeZone: TZ, month: 'short', day: 'numeric', year: '2-digit', hour: '2-digit' })),
                     datasets: [
                         { label: t('chart.discharge'), data: sampled.map((row) => row.discharge), borderColor: '#087da5', backgroundColor: 'rgba(8,125,165,.12)', yAxisID: 'flow', pointRadius: 0, tension: .25 },
                         { label: t('chart.level'), data: sampled.map((row) => row.water_level), borderColor: '#d99018', yAxisID: 'level', pointRadius: 0, tension: .25 },
@@ -1187,13 +1504,14 @@
                 options: {
                     responsive: true, maintainAspectRatio: false, animation: { duration: 0 },
                     interaction: { mode: 'index', intersect: false },
+                    color: cssVar('--ink'),
                     scales: {
-                        flow: { type: 'linear', position: 'left' },
-                        level: { type: 'linear', position: 'right', grid: { drawOnChartArea: false } },
-                        x: { ticks: { maxTicksLimit: 12 } }
+                        flow: { type: 'linear', position: 'left', grid: { color: cssVar('--chart-grid') }, ticks: { color: cssVar('--muted') } },
+                        level: { type: 'linear', position: 'right', grid: { drawOnChartArea: false }, ticks: { color: cssVar('--muted') } },
+                        x: { ticks: { maxTicksLimit: 12, color: cssVar('--muted') }, grid: { color: cssVar('--chart-grid') } }
                     },
                     plugins: {
-                        legend: { display: true },
+                        legend: { display: true, labels: { color: cssVar('--ink') } },
                         floodBands: { enabled: true }
                     }
                 },
@@ -1328,6 +1646,30 @@
         });
     }
 
+    // Single source of truth for "the river right now".
+    //
+    // Two independent feeds carry a P.1 level: the latest measurement, and
+    // `current_level` on the forecast rows, which is whatever the model saw at
+    // its `as_of` (forecasts are precomputed hourly, so it lags). Both used to
+    // write state.p1Now unconditionally and the page showed two different
+    // "now" values at once — observed 1.66 m in the verdict banner against
+    // 1.52 m in the outlook directly below it. Harmless at low water, but at
+    // flood stage two contradictory river levels destroy trust in the warning.
+    //
+    // The freshest reading wins; `force` is for the replay and demo hooks,
+    // which deliberately pin a level that is not the live one.
+    function setP1Level(value, at, opts) {
+        const options = opts || {};
+        if (value == null || Number.isNaN(Number(value))) return;
+        const stamp = at ? (parseTs(at)?.getTime() ?? NaN) : null;
+        const known = Number.isFinite(stamp) ? stamp : null;
+        if (!options.force && known != null && state.p1NowAt != null && known < state.p1NowAt) {
+            return; // an older snapshot must not overwrite a newer one
+        }
+        state.p1Now = Number(value);
+        if (known != null) state.p1NowAt = known;
+    }
+
     // At-a-glance verdict from P.1 level + 24 h forecast. Level thresholds are
     // the official Chiang Mai inundation stages (city flooding starts 3.70 m).
     function updateFloodVerdict() {
@@ -1359,7 +1701,7 @@
 
     function renderSummary(stations, readings) {
         const current = stations.map((s) => readings.get(s.station_code)).filter(Boolean);
-        const timestamps = current.map((m) => new Date(m.timestamp)).filter((date) => !Number.isNaN(date.getTime()));
+        const timestamps = current.map((m) => parseTs(m.timestamp)).filter(Boolean);
         const latest = timestamps.length ? new Date(Math.max(...timestamps.map((date) => date.getTime()))) : null;
         $('station-count').textContent = `${current.length} / ${stations.length}`;
         if (state.hiiReportingCount != null) {
@@ -1368,7 +1710,7 @@
         // Flow at the basin anchor (summing sequential mainstem gauges would
         // double-count the same water, so no basin-wide total is shown).
         const p1 = readings.get('P.1');
-        if (p1?.water_level != null) state.p1Now = Number(p1.water_level);
+        setP1Level(p1?.water_level, p1?.timestamp);
         updateFloodVerdict();
         const p1Flow = p1?.discharge == null ? null : Number(p1.discharge);
         $('total-flow').textContent = p1Flow == null ? '—' : p1Flow.toLocaleString(loc(), { maximumFractionDigits: 1 });
@@ -1379,12 +1721,24 @@
         const worst = stressed.length ? stressed.reduce((max, item) => item.percent > max.percent ? item : max) : null;
         $('peak-flow').textContent = worst ? `${worst.percent.toFixed(0)}%` : '—';
         $('peak-station').textContent = worst ? t('stat.stress.capacity', worst.code) : t('stat.stress.nodata');
-        $('last-updated').textContent = latest ? latest.toLocaleTimeString(loc(), { hour: '2-digit', minute: '2-digit' }) : '—';
+        $('last-updated').textContent = latest ? latest.toLocaleTimeString(loc(), { timeZone: TZ, hour: '2-digit', minute: '2-digit' }) : '—';
+        const tile = $('last-updated').closest('.stat');
         if (latest) {
             const minutes = Math.max(0, Math.round((Date.now() - latest.getTime()) / 60000));
-            $('data-age').textContent = t('stat.updated.ago',
-                latest.toLocaleDateString(loc(), { day: 'numeric', month: 'short' }), minutes);
-        } else $('data-age').textContent = t('stat.updated.none');
+            const day = latest.toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' });
+            // RID publishes hourly and the scrape runs hourly, so anything past
+            // ~3 h means the feed or the collector has stopped: say so loudly
+            // instead of letting "6000 min ago" pass as a number.
+            const stale = minutes >= STALE_AFTER_MIN;
+            $('data-age').textContent = stale
+                ? t('stat.updated.stale', day, Math.round(minutes / 60))
+                : minutes >= 120 ? t('stat.updated.agoh', day, Math.round(minutes / 60))
+                : t('stat.updated.ago', day, minutes);
+            tile.classList.toggle('stale', stale);
+            if (!state.replayTimer && (state.liveMode === 'live' || state.liveMode === 'stale')) {
+                setLiveIndicator(stale ? 'stale' : 'live', stale ? 'pill.stale' : 'pill.live');
+            }
+        } else { $('data-age').textContent = t('stat.updated.none'); tile.classList.remove('stale'); }
     }
 
     async function loadDashboard() {
@@ -1581,6 +1935,8 @@
     function endReplay() {
         if (state.replayTimer) window.clearInterval(state.replayTimer);
         state.replayTimer = null;
+        state.p1Now = null;      // drop the replayed level so live data re-arbitrates
+        state.p1NowAt = null;
         $('replay-2024').textContent = t('replay.start');
         $('replay-clock').style.display = 'none';
         setLiveIndicator('live');
@@ -1626,15 +1982,16 @@
                 $('peak-station').textContent = peakFlow ? t('replay.stress', peakFlow.code) : t('replay.nodischarge');
                 restyleRiver(readings);
                 const p1Level = data.stations['P.1'] ? data.stations['P.1'].level[frame] : null;
-                if (p1Level != null) state.p1Now = Number(p1Level);
+                // force: replay frames are 2024 timestamps, older than anything live
+                if (p1Level != null) setP1Level(p1Level, null, { force: true });
                 restyleFloodZones();
-                const ts = new Date(data.timestamps[frame]);
+                const ts = parseTs(data.timestamps[frame]);
                 // minute included: a lone "17" reads as a year in Thai output
-                const when = ts.toLocaleString(loc(), { day: 'numeric', month: 'short', hour: '2-digit', minute: '2-digit' });
+                const when = ts.toLocaleString(loc(), { timeZone: TZ, day: 'numeric', month: 'short', hour: '2-digit', minute: '2-digit' });
                 const p1Text = state.p1Now == null ? '—' : metres(state.p1Now);
                 const basinFlow = Math.round(totalFlow).toLocaleString(loc());
                 $('p1-peak').textContent = t('replay.peak', when, p1Text, basinFlow);
-                $('replay-clock-time').textContent = ts.toLocaleString(loc(), { day: 'numeric', month: 'short', year: 'numeric', hour: '2-digit', minute: '2-digit' });
+                $('replay-clock-time').textContent = ts.toLocaleString(loc(), { timeZone: TZ, day: 'numeric', month: 'short', year: 'numeric', hour: '2-digit', minute: '2-digit' });
                 $('replay-clock-sub').textContent = t('replay.clock.sub', p1Text, basinFlow);
                 // model track: what the forecast system (trained pre-flood) said at this moment
                 const model = data.model || {};
@@ -1736,31 +2093,38 @@
         if (!p1rows.length) { card.style.display = 'none'; return; }
         const row = p1rows.reduce((best, r) => r.horizon_hours > best.horizon_hours ? r : best);
         state.p1Stages = row.stages;
-        state.p1Now = row.current_level == null ? null : Number(row.current_level);
+        // Offered, not imposed: only used when it is fresher than the latest
+        // measurement (see setP1Level).
+        setP1Level(row.current_level, row.as_of);
         // Demo hooks: ?demo_level=4.4 pins a simulated P.1 level; ?demo_rise=1 animates
         // the water from the real level up to the 2024 record (5.30 m).
         const params = new URLSearchParams(location.search);
         const demoLevel = Number(params.get('demo_level'));
         const simulated = Number.isFinite(demoLevel) && demoLevel > 0;
-        if (simulated) state.p1Now = demoLevel;
+        if (simulated) setP1Level(demoLevel, null, { force: true });
         if ((simulated || params.has('demo_rise')) && !state.replayTimer) setLiveIndicator('sim', 'pill.sim');
         if (params.has('demo_rise') && !state.demoRiseTimer) {
             let level = state.p1Now ?? 2.2;
             state.demoRiseTimer = window.setInterval(() => {
                 level = Math.min(level + 0.05, 5.30);
-                state.p1Now = level;
+                setP1Level(level, null, { force: true });
                 $('p1-peak').textContent = t('outlook.sim.rising', level.toFixed(2), Number(row.current_level).toFixed(2));
                 restyleFloodZones();
                 if (level >= 5.30) window.clearInterval(state.demoRiseTimer);
             }, 700);
         }
         restyleFloodZones();
+        // Show the SAME level the verdict banner shows, and never let the
+        // predicted peak read below it (the peak was computed from the
+        // forecast's own, possibly older, current_level).
+        const shownNow = state.p1Now != null ? state.p1Now : Number(row.current_level);
+        const shownPeak = Math.max(Number(row.predicted_max_level), shownNow);
         $('p1-peak').textContent = simulated
             ? t('outlook.sim', demoLevel.toFixed(2), Number(row.current_level).toFixed(2))
-            : t('outlook.now', Number(row.current_level).toFixed(2), row.horizon_hours,
-                Number(row.predicted_max_level).toFixed(2));
+            : t('outlook.now', shownNow.toFixed(2), row.horizon_hours, shownPeak.toFixed(2));
         renderStageChips(row.stages, row.horizon_hours);
         card.style.display = 'block';
+        updateFloodVerdict(); // banner reflects the arbitrated level, not a stale one
     }
 
     async function loadForecasts() {
@@ -1816,9 +2180,9 @@
                     `<div class="risk-chips">${chips}</div>`;
                 grid.appendChild(cardEl);
             });
-            const asOf = rows[0].as_of ? new Date(rows[0].as_of).toLocaleString(loc(), { dateStyle: 'medium', timeStyle: 'short' }) : null;
+            const asOf = rows[0].as_of ? parseTs(rows[0].as_of).toLocaleString(loc(), { timeZone: TZ, dateStyle: 'medium', timeStyle: 'short' }) : null;
             const modelRow = rows.find((r) => r.source === 'model');
-            const trainedAt = modelRow?.trained_at ? new Date(modelRow.trained_at).toLocaleDateString(loc(), { day: 'numeric', month: 'short' }) : null;
+            const trainedAt = modelRow?.trained_at ? parseTs(modelRow.trained_at).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' }) : null;
             const modelInfo = modelRow?.model_version
                 ? t('forecast.status.model', modelRow.model_version, trainedAt ? t('forecast.status.trained', trainedAt) : '')
                 : '';
@@ -1834,11 +2198,60 @@
                 ? t('forecast.collapse')
                 : t('forecast.expand', stations.length);
             card.style.display = 'block';
+            loadSkill(); // non-blocking; panel stays hidden until there is verified data
+            loadAlertsConfig(); // shows the "Get alerts" button only when ntfy is configured
         } catch (error) {
             card.style.display = 'none';
         }
     }
 
+    async function loadSkill() {
+        const panel = $('skill-panel');
+        try {
+            const response = await fetch('/api/forecast/skill?station_code=P.1&horizon=24');
+            if (!response.ok) throw new Error(`HTTP ${response.status}`);
+            const data = await response.json();
+            const versions = (data.versions || []).filter((v) => v.n > 0);
+            if (!versions.length) { panel.style.display = 'none'; return; }
+            const cm = (m) => m == null ? t('skill.na') : t('skill.cm', (m * 100).toFixed(1));
+            const fmtDay = (v) => parseTs(v).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short' });
+            const total = versions.reduce((a, v) => a + v.n, 0);
+            $('skill-sub').textContent = t('skill.sub', total.toLocaleString(loc()), fmtDay(versions[0].first_issued));
+            const head = $('skill-headline');
+            head.className = 'skill-headline';
+            const cur = data.current;
+            if (data.trend && cur) {
+                const delta = Math.abs(data.trend.mae_delta_m * 100).toFixed(1);
+                head.textContent = data.trend.better
+                    ? t('skill.better', cur.model_version, data.trend.previous_version, delta)
+                    : t('skill.worse', cur.model_version, data.trend.previous_version, delta);
+                head.classList.add(data.trend.better ? 'better' : 'worse');
+            } else if (cur && cur.enough_data) {
+                head.textContent = t('skill.single', cur.model_version);
+            } else if (cur) {
+                head.textContent = t('skill.young', cur.model_version, cur.n, data.min_verified);
+            } else head.textContent = t('skill.none');
+            const anyHigh = versions.some((v) => v.above_2m_n > 0);
+            const compared = versions.filter((v) => v.enough_data).length > 1;
+            $('skill-caveat').textContent = !anyHigh ? t('skill.caveat.quiet') : compared ? t('skill.caveat.regime') : '';
+            const cols = ['version', 'period', 'n', 'mae', 'bias', 'pers', 'skill', 'high'];
+            const rows = versions.map((v) => `<tr class="${v === cur ? 'current' : ''}${v.enough_data ? '' : ' young'}">`
+                + `<td>${escapeHtml(v.model_version)}</td>`
+                + `<td class="dim">${fmtDay(v.first_issued)} – ${fmtDay(v.last_issued)}</td>`
+                + `<td class="num">${v.n.toLocaleString(loc())}</td>`
+                + `<td class="num">${cm(v.mae_m)}</td>`
+                + `<td class="num">${v.bias_m == null ? t('skill.na') : (v.bias_m >= 0 ? '+' : '') + (v.bias_m * 100).toFixed(1)}</td>`
+                + `<td class="num dim">${cm(v.persistence_mae_m)}</td>`
+                + `<td class="num">${v.skill == null ? t('skill.na') : v.skill.toFixed(2)}</td>`
+                + `<td class="num">${v.above_2m_n ? `${cm(v.above_2m_mae_m)} <span class="dim">(${v.above_2m_n})</span>` : t('skill.na')}</td>`
+                + '</tr>').join('');
+            $('skill-table').innerHTML = `<thead><tr>${cols.map((c) => `<th>${escapeHtml(t('skill.col.' + c))}</th>`).join('')}</tr></thead><tbody>${rows}</tbody>`;
+            panel.style.display = 'block';
+        } catch (error) {
+            panel.style.display = 'none';
+        }
+    }
+
     async function loadDbStats() {
         const strip = $('db-stats');
         try {
@@ -1854,7 +2267,7 @@
         const strip = $('db-stats');
         try {
             state.lastStats = stats;
-            const fmtDate = (value) => new Date(value).toLocaleDateString(loc(), { day: 'numeric', month: 'short', year: 'numeric' });
+            const fmtDate = (value) => parseTs(value).toLocaleDateString(loc(), { timeZone: TZ, day: 'numeric', month: 'short', year: 'numeric' });
             state.dbFirstDate = String(stats.first_timestamp).slice(0, 10); // feeds the All-time date range
             $('db-total').textContent = Number(stats.total_measurements).toLocaleString(loc());
             if (stats.rid_measurements != null) {
@@ -1890,6 +2303,7 @@
             : t('forecast.expand', count);
     });
     $('lang-toggle').addEventListener('click', () => setLang(state.lang === 'th' ? 'en' : 'th'));
+    $('theme-toggle').addEventListener('click', () => applyTheme(currentTheme() === 'dark' ? 'light' : 'dark', true));
     $('station-search').addEventListener('input', applyStationSearch);
     $('sensors-head').addEventListener('click', () => setSensorsOpen(!state.sensorsOpen));
     $('sensors-head').addEventListener('keydown', (event) => {

src/water_scraper_v3.py

@@ -632,9 +632,7 @@ class EnhancedWaterMonitorScraper:
                 if data:
                     if self.save_to_database(data):
                         filled_count += len(data)
-                        logger.info(
-                            f"Filled {len(data)} measurements for {fetch_date}"
-                        )
+                        logger.info(f"Filled {len(data)} measurements for {fetch_date}")
                     else:
                         logger.warning(f"Failed to save data for {fetch_date}")
                 else:

src/web_api.py

@@ -210,7 +210,9 @@ async def lifespan(app: FastAPI):
     app_state["leader_lock"] = _acquire_collection_leadership(
         Config.COLLECTION_LEADER_PORT
     )
+    app_state["notify"] = None
     if app_state["leader_lock"]:
+        app_state["notify"] = _init_notifications()
         app_state["scraping_task"] = asyncio.create_task(background_scraping_task())
         logger.info("This worker is the background-collection leader")
     else:
@@ -296,6 +298,73 @@ async def _persist_rain():
         logger.warning(f"rain persistence failed: {e}")
 
 
+def _init_notifications():
+    """Publisher + persisted state for ntfy, or None if off/unavailable.
+
+    Called only by the collection leader: it is the one process that
+    publishes, so the notification_state DDL runs exactly once per host.
+    """
+    if not Config.NTFY_SERVER:
+        return None
+    try:
+        from . import notify as notify_mod
+
+        store = app_state.get("forecast_store")
+        if store and not store.engine:
+            store.connect()
+        state = (
+            notify_mod.NotificationState(store.engine, store.db_type)
+            if store and store.engine
+            else notify_mod.InMemoryState()
+        )
+        if isinstance(state, notify_mod.InMemoryState):
+            logger.warning(
+                "ntfy: no SQL store; notification state is in-memory "
+                "(a restart may re-send the current level)"
+            )
+        publisher = notify_mod.NtfyPublisher(
+            Config.NTFY_PUBLISH_URL,
+            prefix=Config.NTFY_TOPIC_PREFIX,
+            token=Config.NTFY_TOKEN or None,
+            dashboard_url=Config.PUBLIC_URL,
+        )
+        logger.info(
+            f"ntfy notifications: publish to {Config.NTFY_PUBLISH_URL}, "
+            f"subscribers use {Config.NTFY_SERVER}, topics {Config.NTFY_TOPIC_PREFIX}-*"
+        )
+        return publisher, state
+    except Exception as e:
+        logger.error(f"ntfy init failed (notifications off): {e}")
+        return None
+
+
+async def _notify_transitions():
+    """Publish flood/outlook/feed transitions to ntfy (leader only, fail-safe)."""
+    cfg = app_state.get("notify")
+    if not cfg:
+        return
+    publisher, state = cfg
+    try:
+        from . import notify as notify_mod
+
+        scraper = app_state["scraper"]
+        readings = await asyncio.to_thread(
+            scraper.db_adapter.get_latest_measurements, 200
+        )
+        with FORECAST_CACHE_LOCK:
+            cached = FORECAST_CACHE.get("all")
+        forecasts = cached[1] if cached else []
+        sent = await asyncio.to_thread(
+            notify_mod.evaluate, readings, forecasts, state, publisher
+        )
+        if sent:
+            logger.info(
+                "ntfy: published " + ", ".join(f"{n.topic}: {n.title}" for n in sent)
+            )
+    except Exception as e:
+        logger.warning(f"ntfy notify cycle failed: {e}")
+
+
 async def _precompute_forecasts():
     """Refresh the forecast cache and persist the issued forecasts (leader only)."""
     try:
@@ -374,9 +443,7 @@ async def background_scraping_task():
                             hii_counts = await asyncio.get_event_loop().run_in_executor(
                                 None, hii_collector.run_cycle
                             )
-                            set_gauge(
-                                "hii_rainfall_rows_saved", hii_counts["rainfall"]
-                            )
+                            set_gauge("hii_rainfall_rows_saved", hii_counts["rainfall"])
                             set_gauge(
                                 "hii_waterlevel_rows_saved", hii_counts["waterlevel"]
                             )
@@ -404,6 +471,10 @@ async def background_scraping_task():
                     # evaluation.
                     await _precompute_forecasts()
 
+                    # Push notifications for threshold crossings (uses the
+                    # forecasts just computed; no-op unless NTFY_SERVER set).
+                    await _notify_transitions()
+
                 app_state["is_scraping"] = False
 
             # Calculate next run time
@@ -521,7 +592,9 @@ _STATIC_DIR = os.path.dirname(_DASHBOARD_HTML_PATH)
 
 @app.get("/robots.txt", include_in_schema=False)
 async def robots_txt():
-    return FileResponse(os.path.join(_STATIC_DIR, "robots.txt"), media_type="text/plain")
+    return FileResponse(
+        os.path.join(_STATIC_DIR, "robots.txt"), media_type="text/plain"
+    )
 
 
 @app.get("/llms.txt", include_in_schema=False)
@@ -864,7 +937,7 @@ def _hii_rows(sql: str, params: Dict[str, Any]) -> List[Dict[str, Any]]:
 # flood, slightly stale readings with a visible timestamp beat an error page.
 HII_CACHE: Dict[str, Any] = {}
 HII_CACHE_LOCK = Lock()
-_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock()}
+_HII_COMPUTE_LOCKS = {"rain": Lock(), "waterlevel": Lock(), "skill": Lock()}
 LATEST_CACHE: Dict[str, Any] = {}
 LATEST_CACHE_LOCK = Lock()
 _LATEST_COMPUTE_LOCK = Lock()
@@ -1001,6 +1074,91 @@ async def get_hii_rainfall_latest(
     return rows
 
 
+@app.get("/api/hii/rainfall/catchment")
+async def get_hii_rainfall_catchment(
+    response: Response, days: int = Query(14, ge=1, le=60)
+):
+    """Upper-Ping catchment-mean hourly rain: HII gauges vs the Open-Meteo
+    series the flood model actually uses, plus their agreement over the window.
+
+    Evidence-gathering endpoint (docs/FLOOD_FORECASTING.md, HII gauge rain):
+    the gauge table only exists since 2026-08 so it cannot be a training
+    feature yet; this makes the two sources' relationship observable meanwhile.
+    """
+    increment_counter("api_requests", labels={"endpoint": "hii_rain_catchment"})
+    start = datetime.now() - timedelta(days=days)
+
+    def compute():
+        import pandas as pd
+
+        from .ml import hii_rain
+
+        engine = _hii_engine()
+        if engine is None:
+            return {
+                "box": hii_rain.CATCHMENT_BOX,
+                "gauge": [],
+                "openmeteo": [],
+                "comparison_24h_sums": {"overlap_hours": 0},
+            }
+        gauge = hii_rain.load_gauge_mean(start=pd.Timestamp(start), engine=engine)
+        openmeteo = None
+        try:
+            from sqlalchemy import text
+
+            with engine.connect() as conn:
+                frame = pd.read_sql(
+                    text(
+                        "SELECT timestamp, catchment_mean FROM openmeteo_rain "
+                        "WHERE timestamp >= :start ORDER BY timestamp"
+                    ),
+                    conn,
+                    params={"start": start},
+                )
+            if not frame.empty:
+                frame["timestamp"] = pd.to_datetime(frame["timestamp"])
+                openmeteo = pd.to_numeric(
+                    frame.set_index("timestamp")["catchment_mean"], errors="coerce"
+                )
+        except Exception as error:  # openmeteo_rain may not exist yet
+            logger.warning(f"openmeteo_rain read failed: {error}")
+
+        def series_rows(s):
+            if s is None:
+                return []
+            return [
+                {
+                    "timestamp": ts.isoformat(),
+                    "rain_mm": None if pd.isna(v) else round(float(v), 2),
+                }
+                for ts, v in s.items()
+            ]
+
+        comparison = (
+            hii_rain.compare_with_openmeteo(gauge, openmeteo)
+            if gauge is not None and openmeteo is not None
+            else {"overlap_hours": 0}
+        )
+        return {
+            "box": hii_rain.CATCHMENT_BOX,
+            "gauge": series_rows(gauge),
+            "openmeteo": series_rows(openmeteo),
+            "comparison_24h_sums": comparison,
+        }
+
+    payload, stale = await _cached_swr(
+        HII_CACHE,
+        HII_CACHE_LOCK,
+        _HII_COMPUTE_LOCKS["rain"],
+        f"rain_catchment:{days}",
+        Config.HII_CACHE_TTL_SECONDS,
+        compute,
+    )
+    if stale:
+        response.headers["X-Data-Stale"] = "true"
+    return payload
+
+
 @app.get("/api/hii/waterlevel/latest")
 async def get_hii_waterlevel_latest(
     response: Response, hours: int = Query(26, ge=1, le=168)
@@ -1043,9 +1201,7 @@ async def get_postgres_history(
             return cached[1]
     try:
         db_config = Config.get_database_config()
-        end_time = (
-            datetime.combine(end, datetime.max.time()) if end else datetime.now()
-        )
+        end_time = datetime.combine(end, datetime.max.time()) if end else datetime.now()
         start_time = (
             datetime.combine(start, datetime.min.time())
             if start
@@ -1138,17 +1294,85 @@ async def get_forecast_history(
     store = app_state.get("forecast_store")
     if not store:
         return []
-    end_dt = (
-        datetime.combine(end, datetime.max.time()) if end else datetime.now()
-    )
+    end_dt = datetime.combine(end, datetime.max.time()) if end else datetime.now()
     start_dt = (
         datetime.combine(start, datetime.min.time())
         if start
         else end_dt - timedelta(hours=hours)
     )
-    return await asyncio.to_thread(
-        store.fetch, station_code, start_dt, end_dt, horizon
+    return await asyncio.to_thread(store.fetch, station_code, start_dt, end_dt, horizon)
+
+
+@app.get("/api/notifications")
+async def get_notifications_config():
+    """Public ntfy settings so the dashboard can offer subscribe links."""
+    server = Config.NTFY_SERVER
+    if not server:
+        return {"enabled": False}
+    prefix = Config.NTFY_TOPIC_PREFIX
+    return {
+        "enabled": True,
+        "server": server,
+        "prefix": prefix,
+        "topics": {
+            "warning": f"{prefix}-warning",
+            "danger": f"{prefix}-danger",
+            "p1_outlook": f"{prefix}-p1-outlook",
+            "status": f"{prefix}-status",
+            "station_pattern": f"{prefix}-<station>-warning | {prefix}-<station>-danger (station code lowercase, no dot: p1, p103)",
+        },
+        "semantics": "transitions only: one message on crossing up, one all-clear on the way down (0.10 m hysteresis)",
+    }
+
+
+@app.get("/api/forecast/skill")
+async def get_forecast_skill(
+    response: Response,
+    station_code: str = Query("P.1"),
+    horizon: int = Query(24, ge=1, le=48),
+):
+    """Is the model getting better? Issued forecasts verified against what the
+    river then did, per model version, with a persistence baseline.
+
+    Read from forecast_history (what each deployed version predicted, hourly)
+    joined to water_measurements; no retraining involved. Cached like the HII
+    feeds because the join is a few hundred correlated subqueries.
+    """
+    increment_counter("api_requests", labels={"endpoint": "forecast_skill"})
+    store = app_state.get("forecast_store")
+    if not store:
+        return {
+            "station_code": station_code,
+            "horizon_hours": horizon,
+            "versions": [],
+            "current": None,
+            "trend": None,
+        }
+
+    def compute():
+        from .ml import skill
+
+        if not store.engine and not store.connect():
+            return {
+                "station_code": station_code,
+                "horizon_hours": horizon,
+                "versions": [],
+                "current": None,
+                "trend": None,
+            }
+        return skill.compute_skill(store.engine, store.db_type, station_code, horizon)
+
+    payload, stale = await _cached_swr(
+        HII_CACHE,
+        HII_CACHE_LOCK,
+        _HII_COMPUTE_LOCKS["skill"],
+        f"skill:{station_code}:{horizon}",
+        max(Config.HII_CACHE_TTL_SECONDS, 900),
+        compute,
     )
+    if stale:
+        response.headers["X-Data-Stale"] = "true"
+    return payload
 
 
 @app.get("/measurements/latest", response_model=List[MeasurementResponse])
@@ -1234,9 +1458,7 @@ async def get_database_stats():
         from sqlalchemy import text
 
         with engine.connect() as conn:
-            return conn.execute(
-                text(
-                    """
+            return conn.execute(text("""
                     SELECT (SELECT COUNT(*) FROM hii_rainfall) AS rain_n,
                            (SELECT COUNT(*) FROM hii_waterlevel) AS wl_n,
                            (SELECT COUNT(*) FROM hii_rain_stations) AS rain_s,
@@ -1245,9 +1467,7 @@ async def get_database_stats():
                            (SELECT MAX(timestamp) FROM hii_rainfall) AS rain_hi,
                            (SELECT MIN(timestamp) FROM hii_waterlevel) AS wl_lo,
                            (SELECT MAX(timestamp) FROM hii_waterlevel) AS wl_hi
-                    """
-                )
-            ).one()
+                    """)).one()
 
     def compute():
         # Heavy: full-table counts and coverage over ~1.7M rows. Runs at most

tests/test_dashboard.py

@@ -175,3 +175,25 @@ def test_dashboard_default_language_respects_browser_order():
 
     assert "langs.some" not in html  # the old any-English-wins test
     assert "if (code.startsWith('th')) return 'th';" in html
+
+
+def test_dashboard_shows_one_current_river_level():
+    """The verdict banner and the P.1 outlook must not disagree about "now".
+
+    Production served 1.66 m in the banner and 1.52 m in the outlook at the
+    same moment: the banner used the latest measurement, the outlook used the
+    forecast payload's current_level from an older as_of.
+    """
+    html = DASHBOARD_PATH.read_text(encoding="utf-8")
+
+    # One arbitrated writer, freshest-wins
+    assert "function setP1Level(" in html
+    assert "state.p1NowAt" in html
+
+    # Neither feed may assign the level directly any more
+    assert "state.p1Now = Number(p1.water_level)" not in html
+    assert "state.p1Now = row.current_level" not in html
+
+    # The outlook renders the arbitrated level, and never a peak below it
+    assert "const shownNow = state.p1Now != null" in html
+    assert "Math.max(Number(row.predicted_max_level), shownNow)" in html

tests/test_flood_forecast.py

@@ -233,6 +233,75 @@ def test_heuristic_fallback(tmp_path):
         assert row["trained_at"] is None
 
 
+def _p1_synth(n: int = 300, seed: int = 11) -> pd.DataFrame:
+    upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
+    return make_synth(n, ["P.1"] + upstream, seed=seed, pulses={"P.1": [(100, 20, 2.0)]})
+
+
+def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
+    """use_rain=True with no rain series must abort, not write v2 bundles.
+
+    Regression for the 2026-09-01 server retrain that overwrote v3 with v2
+    because the Open-Meteo archive fetch failed on a cache-less checkout.
+    """
+    from src.ml import rain as rain_mod
+
+    df = _p1_synth()
+    overrides = {"max_iter": 10}
+
+    # Case 1: the loader returns None (archive unreachable, no cache file)
+    monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: None)
+    with pytest.raises(train.RainUnavailableError, match="--no-rain"):
+        train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
+    assert not (tmp_path / "flood_P.1.joblib").exists()
+    assert not (tmp_path / "metrics.json").exists()
+
+    # Case 2: the loader raises (network / parse error)
+    def boom(*a, **k):
+        raise ConnectionError("simulated Open-Meteo outage")
+
+    monkeypatch.setattr(rain_mod, "load_history", boom)
+    with pytest.raises(train.RainUnavailableError, match="simulated Open-Meteo outage"):
+        train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
+    assert not (tmp_path / "flood_P.1.joblib").exists()
+
+    # Explicit opt-out still produces v2 bundles as before
+    metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=False, use_dam=False)
+    assert metrics["model_version"].startswith("hgb-v2+")
+    assert (tmp_path / "flood_P.1.joblib").exists()
+
+
+def test_train_with_rain_series_yields_v3(tmp_path, monkeypatch):
+    from src.ml import rain as rain_mod
+
+    df = _p1_synth()
+    idx = pd.date_range(df["timestamp"].min(), df["timestamp"].max(), freq="h")
+    fake_rain = pd.DataFrame({"a": np.linspace(0, 1, len(idx)), "b": 0.5}, index=idx)
+    monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: fake_rain)
+
+    metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 10}, use_rain=True, use_dam=False)
+    assert metrics["model_version"].startswith("hgb-v3+")
+    bundle = joblib.load(tmp_path / "flood_P.1.joblib")
+    assert set(features.RAIN_FEATURES) <= set(bundle["feature_names"])
+
+
+def test_cli_exit_code_on_rain_failure(tmp_path, monkeypatch, caplog):
+    """The console entry turns the guard into a one-line error and exit 2."""
+    from src.ml import rain as rain_mod
+
+    df = _p1_synth()
+    monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: None)
+    monkeypatch.setattr(train, "load_measurements", lambda *a, **k: df)
+    monkeypatch.setattr(train, "resolve_db_url", lambda *a, **k: None)
+    monkeypatch.setattr(
+        "sys.argv",
+        ["train", "--stations", "P.1", "--models-dir", str(tmp_path), "--skip-eval"],
+    )
+    assert train.cli() == 2
+    assert "refusing to silently downgrade" in caplog.text
+    assert not (tmp_path / "metrics.json").exists()
+
+
 def test_feature_name_stability(tmp_path):
     upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
     data_stations = ["P.1"] + upstream

tests/test_forecast_skill.py

@@ -0,0 +1,93 @@
+"""Forecast skill verification: issued forecasts vs observed peaks (sqlite)."""
+
+import datetime
+
+import pytest
+from sqlalchemy import create_engine, text
+
+from src.ml import skill
+
+
+@pytest.fixture
+def engine(tmp_path):
+    eng = create_engine(f"sqlite:///{tmp_path / 'skill.db'}")
+    with eng.begin() as c:
+        c.execute(text("CREATE TABLE stations (id INTEGER PRIMARY KEY, station_code TEXT)"))
+        c.execute(text("INSERT INTO stations VALUES (1, 'P.1')"))
+        c.execute(
+            text(
+                "CREATE TABLE water_measurements (timestamp DATETIME, station_id INTEGER, water_level REAL)"
+            )
+        )
+        c.execute(
+            text(
+                "CREATE TABLE forecast_history (as_of TIMESTAMP, station_code TEXT, horizon_hours INTEGER, "
+                "predicted_max_level REAL, p_warning REAL, p_danger REAL, current_level REAL, "
+                "model_version TEXT, source TEXT)"
+            )
+        )
+    return eng
+
+
+def _fill(engine, start, hours, level_fn, forecasts):
+    """hours of hourly observations from `start`, plus (as_of_offset_h, version, pred) rows."""
+    with engine.begin() as c:
+        for h in range(hours):
+            ts = start + datetime.timedelta(hours=h)
+            c.execute(
+                text("INSERT INTO water_measurements VALUES (:t, 1, :l)"),
+                {"t": ts, "l": level_fn(h)},
+            )
+        for off, version, pred in forecasts:
+            ts = start + datetime.timedelta(hours=off)
+            c.execute(
+                text(
+                    "INSERT INTO forecast_history VALUES (:t, 'P.1', 24, :p, 0, 0, :cur, :v, 'model')"
+                ),
+                {"t": ts, "p": pred, "cur": level_fn(off), "v": version},
+            )
+
+
+def test_skill_per_version_and_trend(engine):
+    start = datetime.datetime(2026, 8, 1)
+    # river: flat 1.5 m, with a bump to 2.4 m around hour 100
+    level = lambda h: 2.4 if 96 <= h <= 104 else 1.5
+    forecasts = []
+    # old version: always predicts 1.5 (persistence-like, misses the bump)
+    for off in range(0, 60):
+        forecasts.append((off, "hgb-v2+aaaaaaa", 1.5))
+    # new version: predicts 1.5 normally and 2.3 ahead of the bump
+    for off in range(60, 200):
+        pred = 2.3 if 72 <= off <= 104 else 1.5
+        forecasts.append((off, "hgb-v3+bbbbbbb", pred))
+    _fill(engine, start, 260, level, forecasts)
+
+    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=300))
+    assert [v["model_version"] for v in out["versions"]] == ["hgb-v2+aaaaaaa", "hgb-v3+bbbbbbb"]
+    old, new = out["versions"]
+    assert old["n"] == 60 and old["enough_data"]
+    assert new["n"] == 140 and new["enough_data"]
+    # the old version issued only on flat hours: perfect there, no bump rows
+    assert old["mae_m"] == 0.0 and old["above_2m_n"] == 0
+    # the new version saw the bump: nonzero MAE but positive skill vs persistence
+    assert new["above_2m_n"] > 0
+    assert new["skill"] is not None and new["skill"] > 0
+    assert out["current"]["model_version"] == "hgb-v3+bbbbbbb"
+    assert out["trend"]["previous_version"] == "hgb-v2+aaaaaaa"
+    assert out["trend"]["better"] is False  # honest: old had an easier period
+
+
+def test_skill_requires_full_window(engine):
+    start = datetime.datetime(2026, 8, 1)
+    # forecasts issued at the very end have no observed window yet
+    _fill(engine, start, 30, lambda h: 1.5, [(o, "hgb-v3+ccccccc", 1.5) for o in range(0, 30)])
+    out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=30))
+    # only as_of <= now-24h AND with >= 18 observed hours in the window count
+    assert out["versions"] and out["versions"][0]["n"] == 7  # as_of 0..6 h: <= now-24h with >= 18 observed hours
+    assert out["versions"][0]["enough_data"] is False
+    assert out["trend"] is None
+
+
+def test_skill_empty(engine):
+    out = skill.compute_skill(engine, "sqlite", "P.1", 24)
+    assert out["versions"] == [] and out["current"] is None and out["trend"] is None

tests/test_hii_collector.py

@@ -353,6 +353,29 @@ class TestHiiApiEndpoints:
         self._get(web_api, "get_hii_rainfall_latest", hours=48)
         assert calls["n"] == 2
 
+    def test_rainfall_catchment(self, web_api):
+        """One gauge in the box (CHM005, 19.12N 98.94E) is below the
+        MIN_GAUGES_PER_HOUR floor, so the catchment mean is NaN -> null, the
+        openmeteo_rain table does not exist in this store, and the comparison
+        reports no overlap. Shape is what matters: the endpoint must not 500
+        on a fresh database."""
+        payload, response = self._get(web_api, "get_hii_rainfall_catchment", days=7)
+        assert "x-data-stale" not in response.headers
+        assert list(payload) == ["box", "gauge", "openmeteo", "comparison_24h_sums"]
+        assert payload["openmeteo"] == []
+        assert payload["comparison_24h_sums"] == {"overlap_hours": 0}
+        assert len(payload["gauge"]) == 1
+        assert payload["gauge"][0]["rain_mm"] is None  # < MIN_GAUGES_PER_HOUR
+
+    def test_rainfall_catchment_disabled(self, monkeypatch):
+        from src import web_api
+
+        monkeypatch.setitem(web_api.app_state, "hii_collector", None)
+        web_api.HII_CACHE.clear()
+        web_api._REFRESH_IN_FLIGHT.clear()
+        payload, _ = self._get(web_api, "get_hii_rainfall_catchment", days=7)
+        assert payload["gauge"] == [] and payload["openmeteo"] == []
+
     def test_stale_served_on_recompute_failure(self, web_api, monkeypatch):
         # Prime the cache, expire it, break the DB: the stale copy is served
         # and flagged via the X-Data-Stale header.

tests/test_hii_rain.py

@@ -0,0 +1,47 @@
+"""HII gauge-rain aggregate: pure-function tests (no DB)."""
+
+import numpy as np
+import pandas as pd
+
+from src.ml import hii_rain
+
+
+def _hourly(start, n):
+    return pd.date_range(start, periods=n, freq="h")
+
+
+def test_compare_identical_series_has_zero_bias():
+    idx = _hourly("2026-08-12", 200)
+    rng = np.random.default_rng(1)
+    rain = pd.Series(rng.exponential(0.5, len(idx)), index=idx)
+    out = hii_rain.compare_with_openmeteo(rain, rain.copy(), window_h=24)
+    assert out["overlap_hours"] == 200
+    assert out["bias_mm"] == 0.0
+    assert out["mae_mm"] == 0.0
+    assert out["corr"] > 0.999
+
+
+def test_compare_reports_constant_bias():
+    idx = _hourly("2026-08-12", 100)
+    gauge = pd.Series(1.0, index=idx)
+    model = pd.Series(1.5, index=idx)  # model wetter by 0.5 mm/h
+    out = hii_rain.compare_with_openmeteo(gauge, model, window_h=24)
+    assert abs(out["bias_mm"] - 12.0) < 1e-9  # 0.5 mm/h x 24 h
+
+
+def test_compare_uses_overlap_only():
+    gauge = pd.Series(1.0, index=_hourly("2026-08-12", 100))
+    model = pd.Series(1.0, index=_hourly("2026-08-14", 100))  # 52 h overlap
+    out = hii_rain.compare_with_openmeteo(gauge, model, window_h=24)
+    assert out["overlap_hours"] == 52
+
+
+def test_compare_no_overlap():
+    gauge = pd.Series(1.0, index=_hourly("2026-01-01", 10))
+    model = pd.Series(1.0, index=_hourly("2026-06-01", 10))
+    assert hii_rain.compare_with_openmeteo(gauge, model) == {"overlap_hours": 0}
+
+
+def test_load_gauge_mean_without_db_returns_none(monkeypatch):
+    monkeypatch.setattr(hii_rain, "resolve_db_url", lambda *a, **k: None)
+    assert hii_rain.load_gauge_mean() is None

tests/test_notify.py

@@ -0,0 +1,285 @@
+"""ntfy notification state machine: transitions only, hysteresis, restart-safe."""
+
+import datetime
+
+import pytest
+
+from src import notify
+
+
+class FakePublisher(notify.NtfyPublisher):
+    def __init__(self):
+        super().__init__("http://ntfy.test", prefix="ping")
+        self.sent = []
+
+    def publish(self, n):
+        self.sent.append(n)
+        return True
+
+
+@pytest.fixture
+def pub():
+    return FakePublisher()
+
+
+def _reading(code, level, ts="2026-09-24T12:00:00"):
+    return {"station_code": code, "water_level": level, "timestamp": ts}
+
+
+def _fc(p, peak=None):
+    return [
+        {
+            "station_code": "P.1",
+            "horizon_hours": 24,
+            "p_warning": p,
+            "predicted_max_level": peak,
+            "source": "model",
+        }
+    ]
+
+
+NOW = datetime.datetime(2026, 9, 24, 12, 30)
+
+
+def topics(pub):
+    return [n.topic for n in pub.sent]
+
+
+def test_quiet_river_sends_nothing(pub):
+    state = notify.InMemoryState()
+    for h in range(48):
+        notify.evaluate(
+            [_reading("P.1", 1.6), _reading("P.103", 3.2)],
+            _fc(0.01),
+            state,
+            pub,
+            now=NOW,
+        )
+    assert pub.sent == []
+
+
+def test_warning_crossing_once_then_silence_then_clear(pub):
+    state = notify.InMemoryState()
+    # rising through 3.70 (P.1 warning)
+    notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
+    assert pub.sent == []
+    notify.evaluate([_reading("P.1", 3.72)], [], state, pub, now=NOW)
+    assert topics(pub) == ["ping-p1-warning", "ping-warning"]
+    assert pub.sent[0].priority == 4 and "3.72 m" in pub.sent[0].message
+    # stays above: no repeats for many hours
+    for level in (3.80, 3.95, 4.05, 3.90, 3.75):
+        notify.evaluate([_reading("P.1", level)], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    # dips to 3.65: within hysteresis, still no message
+    notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    # 3.55: clear
+    notify.evaluate([_reading("P.1", 3.55)], [], state, pub, now=NOW)
+    assert topics(pub)[2:] == ["ping-p1-warning", "ping-warning"]
+    assert "back to normal" in pub.sent[2].title
+
+
+def test_danger_escalation_and_deescalation(pub):
+    state = notify.InMemoryState()
+    notify.evaluate([_reading("P.1", 3.9)], [], state, pub, now=NOW)  # warning
+    notify.evaluate(
+        [_reading("P.1", 4.25)], [], state, pub, now=NOW
+    )  # danger (>= 4.20)
+    assert topics(pub) == [
+        "ping-p1-warning",
+        "ping-warning",
+        "ping-p1-danger",
+        "ping-danger",
+    ]
+    assert pub.sent[2].priority == 5
+    notify.evaluate(
+        [_reading("P.1", 4.15)], [], state, pub, now=NOW
+    )  # hysteresis: still danger
+    assert len(pub.sent) == 4
+    notify.evaluate([_reading("P.1", 4.05)], [], state, pub, now=NOW)  # back to warning
+    assert topics(pub)[4:] == ["ping-p1-danger", "ping-warning"]
+    assert "below danger" in pub.sent[4].title
+
+
+def test_jump_straight_to_danger(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [_reading("P.103", 7.0)], [], state, pub, now=NOW
+    )  # P.103 danger 6.75
+    assert topics(pub) == ["ping-p103-danger", "ping-danger"]
+
+
+def test_basin_digest_groups_stations(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [_reading("P.1", 3.8), _reading("P.103", 6.0), _reading("P.67", 1.0)],
+        [],
+        state,
+        pub,
+        now=NOW,
+    )
+    basin = [n for n in pub.sent if n.topic == "ping-warning"]
+    assert len(basin) == 1 and "P.1" in basin[0].message and "P.103" in basin[0].message
+
+
+def test_outlook_on_off_with_hysteresis(pub):
+    state = notify.InMemoryState()
+    r = [_reading("P.1", 2.9)]
+    notify.evaluate(r, _fc(0.30), state, pub, now=NOW)
+    assert pub.sent == []
+    notify.evaluate(r, _fc(0.55, 3.9), state, pub, now=NOW)
+    assert topics(pub) == ["ping-p1-outlook"]
+    assert "55%" in pub.sent[0].message and "3.90 m" in pub.sent[0].message
+    assert "not an official warning" in pub.sent[0].message
+    notify.evaluate(
+        r, _fc(0.40), state, pub, now=NOW
+    )  # between OFF and ON: stays on, silent
+    assert len(pub.sent) == 1
+    notify.evaluate(r, _fc(0.20), state, pub, now=NOW)
+    assert len(pub.sent) == 2 and "easing" in pub.sent[1].title
+
+
+def test_heuristic_forecast_ignored(pub):
+    state = notify.InMemoryState()
+    fc = [
+        {
+            "station_code": "P.1",
+            "horizon_hours": 24,
+            "p_warning": 0.9,
+            "source": "heuristic",
+        }
+    ]
+    notify.evaluate([_reading("P.1", 2.0)], fc, state, pub, now=NOW)
+    assert pub.sent == []
+
+
+def test_stale_feed_and_recovery(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=NOW
+    )
+    assert pub.sent == []
+    later = NOW + datetime.timedelta(hours=4)
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=later
+    )
+    assert topics(pub) == ["ping-status"] and "stale" in pub.sent[0].title
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T12:00:00")],
+        [],
+        state,
+        pub,
+        now=later + datetime.timedelta(hours=1),
+    )
+    assert len(pub.sent) == 1  # still stale, no repeat
+    notify.evaluate(
+        [_reading("P.1", 1.6, "2026-09-24T17:00:00")],
+        [],
+        state,
+        pub,
+        now=later + datetime.timedelta(hours=1),
+    )
+    assert len(pub.sent) == 2 and "recovered" in pub.sent[1].title
+
+
+def test_capacity_guard_blocks_stale_threshold(pub):
+    """P.77 2026-09: 3.02 m >= 2.85 m 'warning' at 22 % capacity -> not a flood."""
+    state = notify.InMemoryState()
+    r = {
+        "station_code": "P.77",
+        "water_level": 4.40,
+        "timestamp": "2026-09-24T12:00:00",
+        "discharge_percent": 10.3,
+    }
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert pub.sent == [] and state.get("level:P.77") is None
+    # same level with capacity agreeing -> alert
+    r["discharge_percent"] = 82.0
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert topics(pub) == ["ping-p77-warning", "ping-warning"]
+
+
+def test_capacity_guard_exempts_p1_and_missing_pct(pub):
+    state = notify.InMemoryState()
+    notify.evaluate(
+        [
+            {
+                "station_code": "P.1",
+                "water_level": 3.75,
+                "timestamp": "2026-09-24T12:00:00",
+                "discharge_percent": 40.0,
+            }
+        ],
+        [],
+        state,
+        pub,
+        now=NOW,
+    )
+    assert topics(pub) == ["ping-p1-warning", "ping-warning"]
+    pub.sent.clear()
+    notify.evaluate(
+        [
+            {
+                "station_code": "P.103",
+                "water_level": 6.0,
+                "timestamp": "2026-09-24T12:00:00",
+            }
+        ],
+        [],
+        state,
+        pub,
+        now=NOW,
+    )
+    assert topics(pub) == ["ping-p103-warning", "ping-warning"]
+
+
+def test_capacity_guard_does_not_block_clearing(pub):
+    """Guard applies only to the clear->alert edge; the all-clear always goes out."""
+    state = notify.InMemoryState()
+    r = {
+        "station_code": "P.67",
+        "water_level": 2.6,
+        "timestamp": "2026-09-24T12:00:00",
+        "discharge_percent": 90.0,
+    }
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    r.update(water_level=2.2, discharge_percent=30.0)
+    notify.evaluate([r], [], state, pub, now=NOW)
+    assert "back to normal" in pub.sent[2].title
+
+
+def test_state_survives_restart_via_sql(tmp_path, pub):
+    from sqlalchemy import create_engine
+
+    eng = create_engine(f"sqlite:///{tmp_path / 'n.db'}")
+    state = notify.NotificationState(eng, "sqlite")
+    notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
+    assert len(pub.sent) == 2
+    # "restart": new state object on the same DB, same reading -> nothing re-sent
+    state2 = notify.NotificationState(eng, "sqlite")
+    notify.evaluate([_reading("P.1", 3.8)], [], state2, pub, now=NOW)
+    assert len(pub.sent) == 2
+
+
+def test_publish_failure_does_not_advance_state():
+    """If ntfy is down the transition must be retried next cycle, not lost."""
+
+    class Down(notify.NtfyPublisher):
+        def __init__(self):
+            super().__init__("http://ntfy.test")
+            self.calls = 0
+
+        def publish(self, n):
+            self.calls += 1
+            return False
+
+    pub = Down()
+    state = notify.InMemoryState()
+    notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
+    assert pub.calls == 2 and state.get("level:P.1") is None
+    # next cycle, ntfy back: the crossing is delivered
+    good = FakePublisher()
+    notify.evaluate([_reading("P.1", 3.8)], [], state, good, now=NOW)
+    assert topics(good) == ["ping-p1-warning", "ping-warning"]
+    assert state.get("level:P.1") == "warning"