feat: recalibrate P.77/P.75 thresholds; capacity guard on alerts

The first ntfy cycle announced "Warning level at P.77" at 3.02 m. That gauge's 2.85 m threshold sat below its own dry-season baseline (2.6-2.7 m at 8-14 % channel capacity): P.77 had been "above warning" for 761 of the last 2 146 hours, at 22 % capacity. Across 2018-2024, 75-85 % capacity reads 3.35-4.57 m and 95-105 % reads 4.27-5.08 m; set 4.30 / 4.90. P.75 moved 2.75/3.50 -> 3.20/3.65 on the same evidence (2024: 3.45 / 3.72). The predictor already handles changed thresholds (regression-derived probabilities until the Oct 1 retrain). Second line of defence in notify.py: a clear->alert transition is only announced when RID's discharge_percent for the reading is >= 60 %, so a re-rated or datum-shifted gauge cannot page subscribers again. P.1 is exempt (its stages come from the inundation map, not capacity); readings without a capacity figure fall back to level only; the all-clear edge is never blocked. 3 tests.
fix: ntfy listens on the Tailscale address; monitor publishes to it directly
2026-09-12 00:34:59 +02:00 · 2026-09-12 00:28:29 +02:00 · 2026-09-12 00:22:03 +02:00 · 2026-09-12 00:18:38 +02:00 · 2026-09-11 23:54:44 +02:00 · 2026-09-11 23:44:44 +02:00
124 changed files with 13909 additions and 7399 deletions
@@ -1,87 +0,0 @@
 # Northern Thailand Ping River Monitor Configuration
 # Copy this file to .env and customize for your environment
 # Database Configuration
 DB_TYPE=postgresql
 # Options: sqlite, mysql, postgresql, influxdb, victoriametrics
 # SQLite Configuration (default)
 WATER_DB_PATH=water_levels.db
 # VictoriaMetrics Configuration
 VM_HOST=localhost
 VM_PORT=8428
 VM_URL=
 # InfluxDB Configuration
 INFLUX_HOST=localhost
 INFLUX_PORT=8086
 INFLUX_DATABASE=ping_river_monitoring
 INFLUX_USERNAME=
 INFLUX_PASSWORD=
 # PostgreSQL Configuration (Remote Server)
 # Option 1: Full connection string (URL encode special characters in password)
 #POSTGRES_CONNECTION_STRING=postgresql://username:url_encoded_password@your-postgres-host:5432/water_monitoring
 # Option 2: Individual components (password will be automatically URL encoded)
 POSTGRES_HOST=10.0.10.201
 POSTGRES_PORT=5432
 POSTGRES_DB=ping_river
 POSTGRES_USER=ping_river
 POSTGRES_PASSWORD=3_%m]k:+16"rx?M#`swIA
 # Examples for connection string:
 # - Local: postgresql://postgres:password@localhost:5432/water_monitoring
 # - Remote: postgresql://user:pass@192.168.1.100:5432/water_monitoring
 # - With special chars: postgresql://user:my%3Apass%40word@host:5432/db
 # - With SSL: postgresql://user:pass@host:port/db?sslmode=require
 # - Connection pooling: postgresql://user:pass@host:port/db?pool_size=20&max_overflow=0
 # Special character URL encoding:
 # : → %3A    @ → %40    # → %23    ? → %3F    & → %26    / → %2F    % → %25
 # MySQL Configuration
 MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/ping_river_monitoring
 # API Configuration
 API_HOST=0.0.0.0
 API_PORT=8000
 API_WORKERS=1
 # Data Collection Settings
 SCRAPING_INTERVAL_HOURS=1
 REQUEST_TIMEOUT=30
 MAX_RETRIES=3
 RETRY_DELAY_SECONDS=60
 # Data Retention
 DATA_RETENTION_DAYS=365
 # Logging Configuration
 LOG_LEVEL=INFO
 LOG_FILE=water_monitor.log
 # Security (for production)
 SECRET_KEY=your-secret-key-here
 API_KEY=your-api-key-here
 # Monitoring
 ENABLE_METRICS=true
 ENABLE_HEALTH_CHECKS=true
 # Geographic Settings
 TIMEZONE=Asia/Bangkok
 DEFAULT_LATITUDE=18.7875
 DEFAULT_LONGITUDE=99.0045
 # External Services
 NOTIFICATION_EMAIL=
 SMTP_SERVER=
 SMTP_PORT=587
 SMTP_USERNAME=
 SMTP_PASSWORD=
 # Development Settings
 DEBUG=false
 DEVELOPMENT_MODE=false
@@ -84,6 +84,19 @@ SMTP_PORT=587
 SMTP_USERNAME=
 SMTP_PASSWORD=
 # Public push notifications via self-hosted ntfy (https://ntfy.sh, single binary).
 # Leave NTFY_SERVER empty to disable. Topics published: <prefix>-<station>-warning,
 # <prefix>-<station>-danger, <prefix>-warning, <prefix>-danger, <prefix>-p1-outlook,
 # <prefix>-status. See docs/NOTIFICATIONS.md.
 NTFY_SERVER=
 # Where the monitor POSTs (defaults to NTFY_SERVER). Use the local ntfy
 # address (loopback or Tailscale IP) so publishing does not depend on
 # DNS / the reverse proxy being up.
 NTFY_PUBLISH_URL=
 NTFY_TOPIC_PREFIX=ping
 NTFY_TOKEN=
 PUBLIC_URL=https://water.buildfor.life/
 # Matrix Alerting Configuration
 MATRIX_HOMESERVER=https://matrix.org
 MATRIX_ACCESS_TOKEN=
@@ -1,2 +0,0 @@
 DB_TYPE=postgresql
 POSTGRES_CONNECTION_STRING=postgresql://postgres:password@localhost:5432/water_monitoring
@@ -1,342 +1,75 @@
-name: CI/CD Pipeline - Northern Thailand Ping River Monitor
+name: CI
 # What this checks, on every push and PR to master:
 #   1. formatting contract (black + isort, config in pyproject.toml)
 #   2. flake8 hard-error gate (syntax, undefined names)
 #   3. the pytest suite (synthetic data, no DB/network; ~1 min)
 # Docker build / staging / production / perf jobs from the original template
 # were removed: there is no registry, no staging host, and production is a
 # systemd unit deployed by `git pull` on the server (docs/FLOOD_FORECASTING.md
 # section 6, scripts/install.sh). Re-add a job when the thing it deploys exists.
 on:
  push:
-    branches: [ master, develop ]
+    branches: [master, develop]
  pull_request:
-    branches: [ master ]
+    branches: [master]
  schedule:
-    # Run tests daily at 2 AM UTC
+    # daily, catches dependency drift / upstream API changes in the tests
-    - cron: '0 2 * * *'
+    - cron: "0 2 * * *"
  workflow_dispatch:
 env:
-  PYTHON_VERSION: '3.11'
+  PYTHON_VERSION: "3.11" # pandas 2.0.3 ships no 3.12 wheels; psycopg2-binary 2.9.9 breaks on 3.13
  REGISTRY: git.b4l.co.th
  IMAGE_NAME: b4l/northern-thailand-ping-river-monitor
  # GitHub token for better rate limits and authentication
  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 jobs:
-  # Test job
+  lint:
    name: Format & lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
          cache-dependency-path: requirements-dev.txt
      - name: Install tools
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore black==26.5.1 isort==5.12.0 flake8==6.1.0
      - name: black
        run: black --check --diff src/ *.py
      - name: isort
        run: isort --check-only --diff src/ *.py
      - name: flake8 (errors only)
        run: flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
  test:
-    name: Test Suite
+    name: Test suite
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.11', '3.12']
    steps:
-    - name: Checkout code
+      - uses: actions/checkout@v4
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
-    - name: Set up Python ${{ matrix.python-version }}
+      - uses: actions/setup-python@v5
-      uses: actions/setup-python@v4
+        with:
-      with:
+          python-version: ${{ env.PYTHON_VERSION }}
-        python-version: ${{ matrix.python-version }}
+          cache: pip
          cache-dependency-path: |
            requirements.txt
            requirements-dev.txt
-    - name: Cache pip dependencies
+      - name: Install dependencies
-      uses: actions/cache@v3
+        run: |
-      with:
+          python -m pip install --upgrade pip --root-user-action=ignore
-        path: ~/.cache/pip
+          pip install --root-user-action=ignore -r requirements.txt
-        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
+          pip install --root-user-action=ignore pytest==9.1.1 pytest-asyncio==0.21.1
        restore-keys: |
          ${{ runner.os }}-pip-
-    - name: Install dependencies
+      - name: pytest
-      run: |
+        env:
-        python -m pip install --upgrade pip --root-user-action=ignore
+          DB_TYPE: sqlite
-        pip install --root-user-action=ignore -r requirements.txt
+        run: pytest -q -p no:cacheprovider
        pip install --root-user-action=ignore -r requirements-dev.txt
    - name: Lint with flake8
      run: |
        flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 src/ --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
    - name: Type check with mypy (advisory)
      run: |
        # 86 pre-existing errors; blocking typing gate deferred until the debt is paid down
        mypy src/ --ignore-missing-imports || true
    - name: Format check with black
      run: |
        black --check src/ *.py
    - name: Import sort check
      run: |
        isort --check-only src/ *.py
    - name: Run integration tests
      run: |
        python tests/test_integration.py
    - name: Run station management tests
      run: |
        python tests/test_station_management.py
    - name: Test application startup
      run: |
        timeout 10s python run.py --test || true
    - name: Security scan with bandit
      run: |
        bandit -r src/ -f json -o bandit-report.json || true
    - name: Upload test artifacts
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: test-results-${{ matrix.python-version }}
        path: |
          bandit-report.json
          *.log
  # Code quality job
  code-quality:
    name: Code Quality
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip --root-user-action=ignore
        pip install --root-user-action=ignore -r requirements-dev.txt
    - name: Run safety check
      run: |
        safety check -r requirements.txt --json --output safety-report.json || true
    - name: Run bandit security scan
      run: |
        bandit -r src/ -f json -o bandit-report.json || true
    - name: Upload security reports
      uses: actions/upload-artifact@v3
      with:
        name: security-reports
        path: |
          safety-report.json
          bandit-report.json
  # Build Docker image
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    needs: test
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
    - name: Log in to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITEA_TOKEN }}
    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=ref,event=branch
          type=ref,event=pr
          type=sha,prefix={{branch}}-
          type=raw,value=latest,enable={{is_default_branch}}
    - name: Build and push Docker image
      uses: docker/build-push-action@v5
      with:
        context: .
        platforms: linux/amd64,linux/arm64
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max
      env:
        GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
    - name: Test Docker image
      run: |
        docker run --rm ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} python run.py --test
  # Integration test with services
  integration-test:
    name: Integration Test with Services
    runs-on: ubuntu-latest
    needs: build
    services:
      victoriametrics:
        image: victoriametrics/victoria-metrics:latest
        ports:
          - 8428:8428
        options: >-
          --health-cmd "wget --quiet --tries=1 --spider http://localhost:8428/health"
          --health-interval 30s
          --health-timeout 10s
          --health-retries 3
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Wait for VictoriaMetrics
      run: |
        timeout 60s bash -c 'until curl -f http://localhost:8428/health; do sleep 2; done'
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip --root-user-action=ignore
        pip install --root-user-action=ignore -r requirements.txt
    - name: Test with VictoriaMetrics
      env:
        DB_TYPE: victoriametrics
        VM_HOST: localhost
        VM_PORT: 8428
      run: |
        python run.py --test
    - name: Start API server
      env:
        DB_TYPE: victoriametrics
        VM_HOST: localhost
        VM_PORT: 8428
      run: |
        python run.py --web-api &
        sleep 10
    - name: Test API endpoints
      run: |
        curl -f http://localhost:8000/health
        curl -f http://localhost:8000/stations
        curl -f http://localhost:8000/metrics
  # Deploy to staging (only on develop branch)
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: [test, build, integration-test]
    if: github.ref == 'refs/heads/develop'
    environment:
      name: staging
      url: https://staging.ping-river-monitor.b4l.co.th
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Deploy to staging
      run: |
        echo "Deploying to staging environment..."
        # Add your staging deployment commands here
        # Example: kubectl, docker-compose, or webhook call
    - name: Health check staging
      run: |
        sleep 30
        curl -f https://staging.ping-river-monitor.b4l.co.th/health
  # Deploy to production (only on main branch, manual approval)
  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: [test, build, integration-test]
    if: github.ref == 'refs/heads/master'
    environment:
      name: production
      url: https://ping-river-monitor.b4l.co.th
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Deploy to production
      run: |
        echo "Deploying to production environment..."
        # Add your production deployment commands here
    - name: Health check production
      run: |
        sleep 30
        curl -f https://ping-river-monitor.b4l.co.th/health
    - name: Notify deployment
      run: |
        echo "✅ Production deployment successful!"
        echo "🌐 URL: https://ping-river-monitor.b4l.co.th"
        echo "📊 Grafana: https://grafana.ping-river-monitor.b4l.co.th"
  # Performance test (only on main branch)
  performance-test:
    name: Performance Test
    runs-on: ubuntu-latest
    needs: deploy-production
    if: github.ref == 'refs/heads/master'
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Install Apache Bench
      run: |
        sudo apt-get update
        sudo apt-get install -y apache2-utils
    - name: Performance test API endpoints
      run: |
        # Test health endpoint
        ab -n 100 -c 10 https://ping-river-monitor.b4l.co.th/health
        # Test stations endpoint
        ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/stations
        # Test metrics endpoint
        ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/metrics
  # Cleanup old artifacts
  cleanup:
    name: Cleanup
    runs-on: ubuntu-latest
    if: always()
    needs: [test, build, integration-test]
    steps:
    - name: Clean up old Docker images
      run: |
        echo "Cleaning up old Docker images..."
        # Add cleanup commands for old images/artifacts
@@ -1,368 +1,99 @@
-name: Documentation
+name: Docs
 # Checks that the documentation the project actually ships stays consistent:
 #   - every relative link / image path in docs/*.md and README.md resolves
 #     inside the repo (external URLs are NOT fetched: localhost examples,
 #     rate-limited hosts and the Tailscale-era links made that gate permanently
 #     red, and a 200 on a curl --head proves nothing about a doc anyway)
 #   - the FastAPI app imports and its OpenAPI schema is exportable (that is
 #     the reference at https://water.buildfor.life/docs)
 # The previous Sphinx/apidoc jobs produced artifacts nobody read and were
 # removed. Reference docs live in docs/*.md; the public overview is at
 # https://buildfor.life/docs/tooling/ping-river-monitor/.
 on:
  push:
-    branches: [ master, develop ]
+    branches: [master, develop]
    paths:
-      - 'docs/**'
+      - "docs/**"
-      - 'README.md'
+      - "README.md"
-      - 'CONTRIBUTING.md'
+      - "CONTRIBUTING.md"
-      - 'src/**/*.py'
+      - "src/web_api.py"
      - "src/schemas.py"
      - ".gitea/workflows/docs.yml"
  pull_request:
    paths:
-      - 'docs/**'
+      - "docs/**"
-      - 'README.md'
+      - "README.md"
-      - 'CONTRIBUTING.md'
+      - "CONTRIBUTING.md"
  workflow_dispatch:
 env:
-  PYTHON_VERSION: '3.11'
+  PYTHON_VERSION: "3.11"
 jobs:
-  # Validate documentation
+  docs:
-  validate-docs:
+    name: Validate documentation
    name: Validate Documentation
    runs-on: ubuntu-latest
    steps:
-    - name: Checkout code
+      - uses: actions/checkout@v4
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
-    - name: Set up Python
+      - name: Relative links and images resolve
-      uses: actions/setup-python@v4
+        run: |
-      with:
+          python3 - <<'PY'
-        python-version: ${{ env.PYTHON_VERSION }}
+          import re, sys, pathlib
          root = pathlib.Path(".")
          files = [root / "README.md", root / "CONTRIBUTING.md", *root.glob("docs/**/*.md")]
          link = re.compile(r"!?\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
          bad = []
          for md in files:
              if not md.exists():
                  continue
              for m in link.finditer(md.read_text(encoding="utf-8")):
                  target = m.group(1)
                  if target.startswith(("http://", "https://", "mailto:", "#")):
                      continue
                  path = target.split("#", 1)[0]
                  if not path:
                      continue
                  resolved = (md.parent / path).resolve()
                  if not resolved.exists():
                      bad.append(f"{md}: {target}")
          if bad:
              print("Broken relative links:")
              print("\n".join("  " + b for b in bad))
              sys.exit(1)
          print(f"checked {len(files)} files, all relative links resolve")
          PY
-    - name: Install documentation tools
+      - uses: actions/setup-python@v5
-      run: |
+        with:
-        python -m pip install --upgrade pip
+          python-version: ${{ env.PYTHON_VERSION }}
-        pip install -r requirements.txt
+          cache: pip
-        pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
+          cache-dependency-path: requirements.txt
        pip install markdown-link-check || true
-    - name: Check markdown links
+      - name: Install dependencies
-      run: |
+        run: |
-        echo "🔗 Checking markdown links..."
+          python -m pip install --upgrade pip --root-user-action=ignore
-        find . -name "*.md" -not -path "./.git/*" -not -path "./node_modules/*" | while read file; do
+          pip install --root-user-action=ignore -r requirements.txt
          echo "Checking $file"
          # Basic link validation (you can enhance this)
          grep -o 'http[s]*://[^)]*' "$file" | while read url; do
            if curl -s --head "$url" | head -n 1 | grep -q "200 OK"; then
              echo "✅ $url"
            else
              echo "❌ $url (in $file)"
            fi
          done
        done
-    - name: Validate README structure
+      - name: OpenAPI schema exports
-      run: |
+        env:
-        echo "📋 Validating README structure..."
+          DB_TYPE: sqlite
        run: |
          python - <<'PY'
          import json
          from src.web_api import app
          spec = app.openapi()
          paths = sorted(spec["paths"])
          required = {"/forecast", "/measurements/latest", "/measurements/history/{station_code}", "/stations", "/api/stats", "/health"}
          missing = required - set(paths)
          assert not missing, f"documented endpoints missing from the app: {missing}"
          json.dump(spec, open("openapi.json", "w"), indent=1)
          print(f"{len(paths)} paths; schema written to openapi.json")
          PY
-        required_sections=(
+      - uses: actions/upload-artifact@v3
-          "# Northern Thailand Ping River Monitor"
+        with:
-          "## Features"
+          name: openapi-${{ github.run_number }}
-          "## Quick Start"
+          path: openapi.json
          "## Installation"
          "## Usage"
          "## API Endpoints"
          "## Docker"
          "## Contributing"
          "## License"
        )
        for section in "${required_sections[@]}"; do
          if grep -q "$section" README.md; then
            echo "✅ Found: $section"
          else
            echo "❌ Missing: $section"
          fi
        done
    - name: Check documentation completeness
      run: |
        echo "📚 Checking documentation completeness..."
        # Check if all Python modules have docstrings
        python -c "
        import ast
        import os
        def check_docstrings(filepath):
            with open(filepath, 'r', encoding='utf-8') as f:
                tree = ast.parse(f.read())
            missing_docstrings = []
            for node in ast.walk(tree):
                if isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef)):
                    if not ast.get_docstring(node):
                        missing_docstrings.append(f'{node.name} in {filepath}')
            return missing_docstrings
        all_missing = []
        for root, dirs, files in os.walk('src'):
            for file in files:
                if file.endswith('.py') and not file.startswith('__'):
                    filepath = os.path.join(root, file)
                    missing = check_docstrings(filepath)
                    all_missing.extend(missing)
        if all_missing:
            print('⚠️ Missing docstrings:')
            for item in all_missing[:10]:  # Show first 10
                print(f'  - {item}')
            if len(all_missing) > 10:
                print(f'  ... and {len(all_missing) - 10} more')
        else:
            print('✅ All functions and classes have docstrings')
        "
  # Generate API documentation
  generate-api-docs:
    name: Generate API Documentation
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Generate OpenAPI spec
      run: |
        echo "📝 Generating OpenAPI specification..."
        python -c "
        import json
        import sys
        sys.path.insert(0, 'src')
        try:
            from web_api import app
            openapi_spec = app.openapi()
            with open('openapi.json', 'w') as f:
                json.dump(openapi_spec, f, indent=2)
            print('✅ OpenAPI spec generated: openapi.json')
        except Exception as e:
            print(f'❌ Failed to generate OpenAPI spec: {e}')
        "
    - name: Generate API documentation
      run: |
        echo "📖 Generating API documentation..."
        # Create API documentation from OpenAPI spec
        if [ -f openapi.json ]; then
          cat > api-docs.md << 'EOF'
        # API Documentation
        This document describes the REST API endpoints for the Northern Thailand Ping River Monitor.
        ## Base URL
        - Production: `https://ping-river-monitor.b4l.co.th`
        - Staging: `https://staging.ping-river-monitor.b4l.co.th`
        - Development: `http://localhost:8000`
        ## Authentication
        Currently, the API does not require authentication. This may change in future versions.
        ## Endpoints
        EOF
          # Extract endpoints from OpenAPI spec
          python -c "
        import json
        with open('openapi.json', 'r') as f:
            spec = json.load(f)
        for path, methods in spec.get('paths', {}).items():
            for method, details in methods.items():
                print(f'### {method.upper()} {path}')
                print()
                print(details.get('summary', 'No description available'))
                print()
                if 'parameters' in details:
                    print('**Parameters:**')
                    for param in details['parameters']:
                        print(f'- `{param[\"name\"]}` ({param.get(\"in\", \"query\")}): {param.get(\"description\", \"No description\")}')
                    print()
                print('---')
                print()
          " >> api-docs.md
          echo "✅ API documentation generated: api-docs.md"
        fi
    - name: Upload documentation artifacts
      uses: actions/upload-artifact@v3
      with:
        name: documentation-${{ github.run_number }}
        path: |
          openapi.json
          api-docs.md
  # Build Sphinx documentation
  build-sphinx-docs:
    name: Build Sphinx Documentation
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.GITEA_TOKEN }}
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
    - name: Create Sphinx configuration
      run: |
        mkdir -p docs/sphinx
        cat > docs/sphinx/conf.py << 'EOF'
        import os
        import sys
        sys.path.insert(0, os.path.abspath('../../src'))
        project = 'Northern Thailand Ping River Monitor'
        copyright = '2025, Ping River Monitor Team'
        author = 'Ping River Monitor Team'
        version = '3.1.3'
        release = '3.1.3'
        extensions = [
            'sphinx.ext.autodoc',
            'sphinx.ext.viewcode',
            'sphinx.ext.napoleon',
            'sphinx_autodoc_typehints',
        ]
        templates_path = ['_templates']
        exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
        html_theme = 'sphinx_rtd_theme'
        html_static_path = ['_static']
        autodoc_default_options = {
            'members': True,
            'member-order': 'bysource',
            'special-members': '__init__',
            'undoc-members': True,
            'exclude-members': '__weakref__'
        }
        EOF
        cat > docs/sphinx/index.rst << 'EOF'
        Northern Thailand Ping River Monitor Documentation
        ================================================
        .. toctree::
           :maxdepth: 2
           :caption: Contents:
           modules
        Indices and tables
        ==================
        * :ref:`genindex`
        * :ref:`modindex`
        * :ref:`search`
        EOF
    - name: Generate module documentation
      run: |
        cd docs/sphinx
        sphinx-apidoc -o . ../../src
    - name: Build documentation
      run: |
        cd docs/sphinx
        sphinx-build -b html . _build/html
    - name: Upload Sphinx documentation
      uses: actions/upload-artifact@v3
      with:
        name: sphinx-docs-${{ github.run_number }}
        path: docs/sphinx/_build/html/
  # Documentation summary
  docs-summary:
    name: Documentation Summary
    runs-on: ubuntu-latest
    needs: [validate-docs, generate-api-docs, build-sphinx-docs]
    if: always()
    steps:
    - name: Generate documentation summary
      run: |
        echo "# 📚 Documentation Build Summary" > docs-summary.md
        echo "" >> docs-summary.md
        echo "**Build Date:** $(date -u)" >> docs-summary.md
        echo "**Repository:** ${{ github.repository }}" >> docs-summary.md
        echo "**Commit:** ${{ github.sha }}" >> docs-summary.md
        echo "" >> docs-summary.md
        echo "## 📊 Results" >> docs-summary.md
        echo "" >> docs-summary.md
        if [ "${{ needs.validate-docs.result }}" = "success" ]; then
          echo "- ✅ **Documentation Validation**: Passed" >> docs-summary.md
        else
          echo "- ❌ **Documentation Validation**: Failed" >> docs-summary.md
        fi
        if [ "${{ needs.generate-api-docs.result }}" = "success" ]; then
          echo "- ✅ **API Documentation**: Generated" >> docs-summary.md
        else
          echo "- ❌ **API Documentation**: Failed" >> docs-summary.md
        fi
        if [ "${{ needs.build-sphinx-docs.result }}" = "success" ]; then
          echo "- ✅ **Sphinx Documentation**: Built" >> docs-summary.md
        else
          echo "- ❌ **Sphinx Documentation**: Failed" >> docs-summary.md
        fi
        echo "" >> docs-summary.md
        echo "## 🔗 Available Documentation" >> docs-summary.md
        echo "" >> docs-summary.md
        echo "- [README.md](../README.md)" >> docs-summary.md
        echo "- [API Documentation](../docs/)" >> docs-summary.md
        echo "- [Contributing Guide](../CONTRIBUTING.md)" >> docs-summary.md
        echo "- [Deployment Checklist](../DEPLOYMENT_CHECKLIST.md)" >> docs-summary.md
        cat docs-summary.md
    - name: Upload documentation summary
      uses: actions/upload-artifact@v3
      with:
        name: docs-summary-${{ github.run_number }}
        path: docs-summary.md
@@ -1,293 +1,109 @@
-name: Security & Dependency Updates
+name: Security
 # Two gates that can actually fail, plus one report:
 #   - pip-audit against requirements.txt: any known vulnerability in a runtime
 #     dependency fails the job (dev-only tools are reported, not gated)
 #   - bandit on src/: HIGH severity findings fail; medium/low are listed.
 #     B104 (bind 0.0.0.0) is skipped: the service is meant to listen on all
 #     interfaces behind Cloudflare/Caddy.
 #   - pip-licenses report as an artifact (informational; the project is MIT
 #     and its runtime deps are MIT/BSD/Apache/PSF)
 # The old file ran safety/bandit/semgrep with `|| true` and could not go red.
 on:
  schedule:
-    # Run security scans daily at 3 AM UTC
+    - cron: "0 3 * * 1" # weekly, Monday 03:00 UTC
    - cron: "0 3 * * *"
  workflow_dispatch:
  push:
    paths:
      - "requirements*.txt"
-      - "Dockerfile"
+      - "pyproject.toml"
      - "uv.lock"
      - "src/**/*.py"
      - ".gitea/workflows/security.yml"
  pull_request:
    paths:
      - "requirements*.txt"
      - "pyproject.toml"
      - "src/**/*.py"
 env:
  PYTHON_VERSION: "3.11"
  # GitHub token for better rate limits and authentication
  GH_TOKEN: ${{ secrets.GH_TOKEN }}
 jobs:
-  # Dependency vulnerability scan
+  dependencies:
-  dependency-scan:
+    name: Dependency vulnerabilities
    name: Dependency Security Scan
    runs-on: ubuntu-latest
    steps:
-      - name: Checkout code
+      - uses: actions/checkout@v4
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
-      - name: Set up Python
+      - uses: actions/setup-python@v5
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install dependencies
+      - name: Install pip-audit
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore safety bandit semgrep
+          pip install --root-user-action=ignore pip-audit
-      - name: Run Safety check
+      - name: Runtime dependencies (gate)
-        run: |
+        run: pip-audit -r requirements.txt --strict --desc on
          safety check -r requirements.txt --json --output safety-report.json || true
          safety check -r requirements-dev.txt --json --output safety-dev-report.json || true
-      - name: Run Bandit security scan
+      - name: Dev dependencies (report only)
-        run: |
+        run: pip-audit -r requirements-dev.txt --desc on || echo "::warning::dev-only dependency advisories above"
          bandit -r src/ -f json -o bandit-report.json || true
-      - name: Run Semgrep security scan
+  code:
-        run: |
+    name: Static analysis
          semgrep --config=auto src/ --json --output=semgrep-report.json || true
      - name: Upload security reports
        uses: actions/upload-artifact@v3
        with:
          name: security-reports-${{ github.run_number }}
          path: |
            safety-report.json
            safety-dev-report.json
            bandit-report.json
            semgrep-report.json
      - name: Check for critical vulnerabilities
        run: |
          echo "Checking for critical vulnerabilities..."
          # Check Safety results
          if [ -f safety-report.json ]; then
            critical_count=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null || echo "0")
            if [ "$critical_count" -gt 0 ]; then
              echo "Found $critical_count dependency vulnerabilities"
              jq '.vulnerabilities[] | "- \(.package_name) \(.installed_version): \(.vulnerability_id)"' safety-report.json
            else
              echo "No dependency vulnerabilities found"
            fi
          fi
          # Check Bandit results
          if [ -f bandit-report.json ]; then
            high_severity=$(jq '.results[] | select(.issue_severity == "HIGH") | length' bandit-report.json 2>/dev/null | wc -l)
            if [ "$high_severity" -gt 0 ]; then
              echo "Found $high_severity high-severity security issues"
            else
              echo "No high-severity security issues found"
            fi
          fi
  # License compliance check
  license-check:
    name: License Compliance
    runs-on: ubuntu-latest
    steps:
-      - name: Checkout code
+      - uses: actions/checkout@v4
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
-      - name: Set up Python
+      - uses: actions/setup-python@v5
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install pip-licenses
+      - name: Install bandit
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
-          pip install --root-user-action=ignore pip-licenses
+          pip install --root-user-action=ignore bandit
          pip install --root-user-action=ignore -r requirements.txt
-      - name: Check licenses
+      - name: bandit (HIGH fails; medium/low listed)
        run: |
-          echo "Checking dependency licenses..."
+          bandit -r src/ -q --skip B104 -ll -ii || true
          bandit -r src/ -q --skip B104 --severity-level high --confidence-level medium
  licenses:
    name: License report
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
          cache-dependency-path: requirements.txt
      # A fresh venv, not the runner's site-packages: the report must list the
      # project's runtime deps, not whatever the runner image or a previous
      # workflow happened to leave installed (semgrep once showed up here).
      - name: Install into a clean venv
        run: |
          python -m venv .lic && . .lic/bin/activate
          pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore -r requirements.txt pip-licenses
      - name: Report
        run: |
          . .lic/bin/activate
          pip-licenses --format=markdown --with-urls --output-file=licenses.md
          pip-licenses --format=json --output-file=licenses.json
-          pip-licenses --format=markdown --output-file=licenses.md
+          echo "Copyleft licenses among runtime deps (informational; LGPL is fine to link from MIT):"
          pip-licenses --format=plain --ignore-packages pip-licenses | grep -iE 'GPL|AGPL|LGPL' || echo "  none"
-          # Check for problematic licenses
+      - uses: actions/upload-artifact@v3
          problematic_licenses=("GPL" "AGPL" "LGPL")
          for license in "${problematic_licenses[@]}"; do
            if grep -i "$license" licenses.json; then
              echo "Found potentially problematic license: $license"
            fi
          done
          echo "License check completed"
      - name: Upload license report
        uses: actions/upload-artifact@v3
        with:
-          name: license-report-${{ github.run_number }}
+          name: licenses-${{ github.run_number }}
          path: |
            licenses.json
            licenses.md
-
+            licenses.json
  # Dependency update check
  dependency-update:
    name: Check for Dependency Updates
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Install pip-check-updates equivalent
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore pip-review
      - name: Check for outdated packages
        run: |
          echo "Checking for outdated packages..."
          pip install --root-user-action=ignore -r requirements.txt
          pip list --outdated --format=json > outdated-packages.json || true
          if [ -s outdated-packages.json ]; then
            echo "Outdated packages found:"
            cat outdated-packages.json | jq -r '.[] | "- \(.name): \(.version) -> \(.latest_version)"'
          else
            echo "All packages are up to date"
          fi
      - name: Upload dependency reports
        uses: actions/upload-artifact@v3
        with:
          name: dependency-reports-${{ github.run_number }}
          path: |
            outdated-packages.json
  # Code quality metrics
  code-quality:
    name: Code Quality Metrics
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITEA_TOKEN }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Install quality tools
        run: |
          python -m pip install --upgrade pip --root-user-action=ignore
          pip install --root-user-action=ignore radon xenon vulture
          pip install --root-user-action=ignore -r requirements.txt
      - name: Calculate code complexity
        run: |
          echo "Calculating code complexity..."
          radon cc src/ --json > complexity-report.json
          radon mi src/ --json > maintainability-report.json
          echo "Complexity Summary:"
          radon cc src/ --average
          echo "Maintainability Summary:"
          radon mi src/
      - name: Find dead code
        run: |
          echo "Checking for dead code..."
          vulture src/ --json > dead-code-report.json || true
      - name: Check for code smells
        run: |
          echo "Checking for code smells..."
          xenon --max-absolute B --max-modules A --max-average A src/ || true
      - name: Upload quality reports
        uses: actions/upload-artifact@v3
        with:
          name: code-quality-reports-${{ github.run_number }}
          path: |
            complexity-report.json
            maintainability-report.json
            dead-code-report.json
  # Security summary
  security-summary:
    name: Security Summary
    runs-on: ubuntu-latest
    needs: [dependency-scan, license-check, code-quality]
    if: always()
    steps:
      - name: Download all artifacts
        uses: actions/download-artifact@v3
      - name: Generate security summary
        run: |
          echo "# Security Scan Summary" > security-summary.md
          echo "" >> security-summary.md
          echo "**Scan Date:** $(date -u)" >> security-summary.md
          echo "**Repository:** ${{ github.repository }}" >> security-summary.md
          echo "**Commit:** ${{ github.sha }}" >> security-summary.md
          echo "" >> security-summary.md
          echo "## Results" >> security-summary.md
          echo "" >> security-summary.md
          # Dependency scan results
          if [ -f security-reports-*/safety-report.json ]; then
            vuln_count=$(jq '.vulnerabilities | length' security-reports-*/safety-report.json 2>/dev/null || echo "0")
            if [ "$vuln_count" -eq 0 ]; then
              echo "- Dependency Scan: No vulnerabilities found" >> security-summary.md
            else
              echo "- Dependency Scan: $vuln_count vulnerabilities found" >> security-summary.md
            fi
          else
            echo "- Dependency Scan: Results not available" >> security-summary.md
          fi
          # Docker scan results (removed Trivy)
          echo "- Docker Scan: Skipped (Trivy removed)" >> security-summary.md
          # License check results
          if [ -f license-report-*/licenses.json ]; then
            echo "- License Check: Completed" >> security-summary.md
          else
            echo "- License Check: Results not available" >> security-summary.md
          fi
          # Code quality results
          if [ -f code-quality-reports-*/complexity-report.json ]; then
            echo "- Code Quality: Analyzed" >> security-summary.md
          else
            echo "- Code Quality: Results not available" >> security-summary.md
          fi
          echo "" >> security-summary.md
          echo "## Detailed Reports" >> security-summary.md
          echo "" >> security-summary.md
          echo "Detailed reports are available in the workflow artifacts." >> security-summary.md
          cat security-summary.md
      - name: Upload security summary
        uses: actions/upload-artifact@v3
        with:
          name: security-summary-${{ github.run_number }}
          path: security-summary.md
@@ -148,3 +148,28 @@ grafana_data/
 models/*.joblib
 models/cache/
 models/metrics.json
 # scripts/retrain.sh working dirs (staging + one rollback generation)
 models/.staging/
 models/.previous/
 # Playwright MCP browser artifacts (screenshots/snapshots from agent sessions)
 .playwright-mcp/
 # Agent tooling state (whole dirs; the entries above only covered subpaths)
 .claude/
 .claude-flow/
 .swarm/
 # local MCP server wiring, not project config
 .mcp.json
 # CLAUDE.md is intentionally NOT ignored — track it if you want the agent
 # conventions shared with collaborators; it is untracked today.
 # Model evaluation output (regenerate with scripts/evaluate_variants.py)
 models/eval_*.json
 # Editor/merge leftovers and stray shell-redirect artifacts. The repo root once
 # collected 56 zero-byte files named after fragments of shell commands.
 *.orig
 *.rej
 *.bak
 *~
@@ -1,129 +0,0 @@
 # GitLab CI/CD Pipeline for Northern Thailand Ping River Monitor
 stages:
  - test
  - build
  - deploy
 variables:
  PYTHON_VERSION: "3.11"
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
 cache:
  paths:
    - .cache/pip
    - venv/
 # Test stage
 test:
  stage: test
  image: python:${PYTHON_VERSION}-slim
  before_script:
    - apt-get update && apt-get install -y build-essential
    - python -m venv venv
    - source venv/bin/activate
    - pip install --upgrade pip
    - pip install -r requirements-dev.txt
  script:
    - python test_integration.py
    - python test_station_management.py
    - flake8 src/ --max-line-length=100
    - mypy src/
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths:
      - htmlcov/
    expire_in: 1 week
 # Code quality
 code_quality:
  stage: test
  image: python:${PYTHON_VERSION}-slim
  before_script:
    - python -m venv venv
    - source venv/bin/activate
    - pip install black isort flake8 mypy
  script:
    - black --check src/ *.py
    - isort --check-only src/ *.py
    - flake8 src/ --max-line-length=100
    - mypy src/
  allow_failure: true
 # Security scan
 security_scan:
  stage: test
  image: python:${PYTHON_VERSION}-slim
  before_script:
    - pip install safety bandit
  script:
    - safety check -r requirements.txt
    - bandit -r src/
  allow_failure: true
 # Build Docker image
 build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker build -t $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    - main
    - develop
 # Deploy to staging
 deploy_staging:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
  script:
    - echo "Deploying to staging environment"
    - curl -X POST "$STAGING_WEBHOOK_URL" -H "Content-Type: application/json" -d '{"image":"'$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA'"}'
  environment:
    name: staging
    url: https://staging.ping-river-monitor.example.com
  only:
    - develop
 # Deploy to production
 deploy_production:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
  script:
    - echo "Deploying to production environment"
    - curl -X POST "$PRODUCTION_WEBHOOK_URL" -H "Content-Type: application/json" -d '{"image":"'$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA'"}'
  environment:
    name: production
    url: https://ping-river-monitor.example.com
  when: manual
  only:
    - main
 # Health check after deployment
 health_check:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - sleep 30  # Wait for deployment
    - curl -f $HEALTH_CHECK_URL/health
    - curl -s $HEALTH_CHECK_URL/metrics | jq .
  dependencies:
    - deploy_production
  only:
    - main
@@ -19,22 +19,20 @@ repos:
  # Python code formatting with Black
  - repo: https://github.com/psf/black
-    rev: 23.11.0
+    rev: 26.5.1
    hooks:
      - id: black
        language_version: python3
        args: ['--line-length=120']
  # Import sorting with isort
  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
        args: ['--profile', 'black', '--line-length', '120']
  # Linting with flake8
  - repo: https://github.com/pycqa/flake8
    rev: 6.1.0
    hooks:
      - id: flake8
-        args: ['--max-line-length=120', '--extend-ignore=E203,W503']
+        args: ['--max-line-length=100', '--extend-ignore=E203,W503']
@@ -0,0 +1,40 @@
 # CLAUDE.md
 Guidance for AI coding agents working in this repository.
 ## What this is
 Flood monitoring and forecasting for the Ping River, Chiang Mai. Public dashboard and
 API at https://water.buildfor.life/ (never publish the server's private/Tailscale IP).
 Production: one systemd unit on a small VPS, `/opt/thailand-water-monitor`, user
 `water-monitor`, interpreter `.venv/bin/python` (uv-managed), updated by `git pull`.
 ## Rules
 - Python 3.11 only. `uv sync --python 3.11`; run everything as `uv run ...`.
 - `make format` (black 88 / isort black profile, config in pyproject.toml) before
  committing; CI fails on formatting. `make test` must stay green — tests are
  synthetic-data only, never add one that needs the DB or network.
 - Timestamps everywhere are Asia/Bangkok wall-clock with no offset. The dashboard
  parses them with `parseTs()` and renders with `timeZone: TZ`; keep it that way.
 - Model changes go through the rolling-origin harness (`scripts/evaluate_variants.py`)
  and are judged on first-alert LEAD and false alarms, not MAE. Record results, positive
  or negative, in `docs/FLOOD_FORECASTING.md` section 5. Do not change what is deployed
  (`rise_rain` / hgb-v3) without a harness result that beats it on lead.
 - `train_all()` must never silently produce a gauge-only (v2) model; the guard that
  raises `RainUnavailableError` stays.
 - No `git add -A`: zero-byte shell-accident files (`#`, `$(wc`, ...) have been committed
  before. Stage files by name.
 - Do not add Co-Authored-By trailers.
 - The dashboard is a single file, `src/static/dashboard.html`, EN + TH via the `t()`
  table: every user-visible string needs both languages.
 ## Where things are
 - `src/web_api.py` FastAPI app; `src/water_scraper_v3.py` RID collector;
  `src/hii_collector.py` ThaiWater/HII; `src/ml/` features/train/evaluate/predict,
  `rain.py` (Open-Meteo), `dam.py`, `hii_rain.py`.
 - `scripts/retrain.sh` + `water-monitor-retrain.timer`: monthly retrain with staged
  promote. `scripts/dev_proxy.py`: serve the working-copy dashboard against the live API.
 - `docs/FLOOD_FORECASTING.md` is the authoritative model write-up; `docs/DATA_SOURCES.md`
  the source catalog.
@@ -1,268 +0,0 @@
 # 🚀 Deployment Checklist - Northern Thailand Ping River Monitor
 ## ✅ Pre-Deployment Checklist
 ### **Code Quality**
 - [ ] All tests pass (`make test`)
 - [ ] Code formatting applied (`make format`)
 - [ ] Linting checks pass (`make lint`)
 - [ ] No security vulnerabilities (`safety check`)
 - [ ] Documentation updated
 - [ ] Version number updated in `setup.py` and `src/__init__.py`
 ### **Configuration**
 - [ ] Environment variables configured (`.env` file)
 - [ ] Database connection tested
 - [ ] API endpoints tested
 - [ ] Log levels appropriate for environment
 - [ ] Security settings configured (API keys, secrets)
 - [ ] Resource limits set (memory, CPU)
 ### **Dependencies**
 - [ ] All required packages in `requirements.txt`
 - [ ] No unused dependencies
 - [ ] Security updates applied
 - [ ] Compatible Python version (3.9+)
 ## 🐳 Docker Deployment
 ### **Pre-Docker Checklist**
 - [ ] Dockerfile tested locally
 - [ ] Docker Compose configuration verified
 - [ ] Volume mounts configured correctly
 - [ ] Network settings configured
 - [ ] Health checks working
 - [ ] Resource limits set
 ### **Docker Commands**
 ```bash
 # Build and test locally
 make docker-build
 docker run --rm ping-river-monitor python run.py --test
 # Deploy with Docker Compose
 make docker-run
 # Verify deployment
 make health-check
 ```
 ### **Post-Docker Checklist**
 - [ ] All services running (`docker-compose ps`)
 - [ ] Health checks passing
 - [ ] Logs showing normal operation
 - [ ] API accessible (`curl http://localhost:8000/health`)
 - [ ] Database connectivity verified
 - [ ] Grafana dashboards loading
 ## 🌐 Production Deployment
 ### **Infrastructure Requirements**
 - [ ] Server specifications adequate (CPU, RAM, Storage)
 - [ ] Network connectivity to external APIs
 - [ ] SSL certificates configured (if HTTPS)
 - [ ] Firewall rules configured
 - [ ] Backup strategy implemented
 - [ ] Monitoring alerts configured
 ### **Security Checklist**
 - [ ] API keys secured (environment variables)
 - [ ] Database credentials secured
 - [ ] HTTPS enabled for web interface
 - [ ] Input validation enabled
 - [ ] Rate limiting configured
 - [ ] Log sanitization enabled
 ### **Performance Checklist**
 - [ ] Database indexes created
 - [ ] Connection pooling configured
 - [ ] Caching enabled where appropriate
 - [ ] Resource monitoring enabled
 - [ ] Performance baselines established
 ## 📊 Monitoring Setup
 ### **Health Monitoring**
 - [ ] Health check endpoints responding
 - [ ] Database health monitoring
 - [ ] API response time monitoring
 - [ ] Memory usage monitoring
 - [ ] Disk space monitoring
 ### **Alerting**
 - [ ] Critical error alerts configured
 - [ ] Performance degradation alerts
 - [ ] Database connectivity alerts
 - [ ] Disk space alerts
 - [ ] API availability alerts
 ### **Logging**
 - [ ] Log rotation configured
 - [ ] Log levels appropriate
 - [ ] Structured logging enabled
 - [ ] Log aggregation configured (if applicable)
 - [ ] Log retention policy set
 ## 🔄 CI/CD Pipeline
 ### **GitLab CI/CD**
 - [ ] `.gitlab-ci.yml` configured
 - [ ] Pipeline variables set
 - [ ] Test stage passing
 - [ ] Build stage creating artifacts
 - [ ] Deploy stage configured
 - [ ] Rollback procedure documented
 ### **Pipeline Stages**
 - [ ] **Test**: Unit tests, integration tests, linting
 - [ ] **Build**: Docker image creation, artifact generation
 - [ ] **Deploy**: Staging deployment, production deployment
 - [ ] **Verify**: Health checks, smoke tests
 ## 🗄️ Database Setup
 ### **Database Configuration**
 - [ ] Database server running and accessible
 - [ ] Database created with correct permissions
 - [ ] Connection string configured
 - [ ] Migration scripts run (if applicable)
 - [ ] Backup strategy implemented
 - [ ] Performance tuning applied
 ### **Database-Specific Checklist**
 #### **SQLite**
 - [ ] Database file permissions set correctly
 - [ ] WAL mode enabled for better concurrency
 - [ ] Regular backup scheduled
 #### **MySQL/PostgreSQL**
 - [ ] User accounts created with minimal privileges
 - [ ] Connection pooling configured
 - [ ] Query performance optimized
 - [ ] Replication configured (if applicable)
 #### **InfluxDB**
 - [ ] Retention policies configured
 - [ ] Continuous queries set up (if needed)
 - [ ] Backup strategy implemented
 #### **VictoriaMetrics**
 - [ ] Storage configuration optimized
 - [ ] Retention period set
 - [ ] Resource limits configured
 ## 🌐 Web Interface
 ### **API Deployment**
 - [ ] FastAPI server running
 - [ ] All endpoints responding correctly
 - [ ] API documentation accessible (`/docs`)
 - [ ] CORS configured correctly
 - [ ] Rate limiting working
 - [ ] Authentication configured (if applicable)
 ### **Frontend Integration**
 - [ ] Grafana dashboards configured
 - [ ] Data sources connected
 - [ ] Visualizations working
 - [ ] Alerts configured
 - [ ] User access configured
 ## 📈 Performance Verification
 ### **Load Testing**
 - [ ] API endpoints tested under load
 - [ ] Database performance under load
 - [ ] Memory usage under load
 - [ ] Response times acceptable
 - [ ] Error rates acceptable
 ### **Capacity Planning**
 - [ ] Expected data volume calculated
 - [ ] Storage growth projected
 - [ ] Scaling strategy documented
 - [ ] Resource monitoring thresholds set
 ## 🔧 Operational Procedures
 ### **Maintenance**
 - [ ] Update procedure documented
 - [ ] Backup and restore procedures tested
 - [ ] Rollback procedure documented
 - [ ] Monitoring runbooks created
 - [ ] Incident response procedures documented
 ### **Documentation**
 - [ ] Deployment guide updated
 - [ ] API documentation current
 - [ ] Configuration documentation complete
 - [ ] Troubleshooting guide available
 - [ ] Contact information updated
 ## ✅ Post-Deployment Verification
 ### **Functional Testing**
 - [ ] Data collection working
 - [ ] API endpoints responding
 - [ ] Database writes successful
 - [ ] Web interface accessible
 - [ ] Station management working
 ### **Integration Testing**
 - [ ] External API connectivity
 - [ ] Database integration
 - [ ] Monitoring integration
 - [ ] Alert system working
 - [ ] Backup system working
 ### **Performance Testing**
 - [ ] Response times acceptable
 - [ ] Memory usage normal
 - [ ] CPU usage normal
 - [ ] Disk I/O normal
 - [ ] Network usage normal
 ## 🚨 Rollback Plan
 ### **Rollback Triggers**
 - [ ] Critical errors in production
 - [ ] Performance degradation
 - [ ] Data corruption
 - [ ] Security vulnerabilities
 - [ ] Service unavailability
 ### **Rollback Procedure**
 1. [ ] Stop current deployment
 2. [ ] Restore previous Docker images
 3. [ ] Restore database backup (if needed)
 4. [ ] Verify system functionality
 5. [ ] Update monitoring and alerts
 6. [ ] Document incident and lessons learned
 ## 📞 Support Information
 ### **Emergency Contacts**
 - [ ] System administrator contact
 - [ ] Database administrator contact
 - [ ] Network administrator contact
 - [ ] Application developer contact
 ### **Documentation Links**
 - [ ] Deployment guide
 - [ ] API documentation
 - [ ] Troubleshooting guide
 - [ ] Configuration reference
 - [ ] Monitoring dashboards
 ---
 **Deployment Date**: ___________  
 **Deployed By**: ___________  
 **Version**: v3.1.3  
 **Environment**: ___________  
 **Sign-off**:
 - [ ] Technical Lead: ___________
 - [ ] Operations Team: ___________
 - [ ] Security Team: ___________
@@ -1,193 +0,0 @@
 # Final GitHub Publication Checklist ✅
 This checklist ensures the Thailand Water Level Monitor project is ready for GitHub publication.
 ## 🎯 **Project Preparation Complete**
 ### ✅ **Core Repository Files**
 - [x] **README.md** - Comprehensive project documentation with badges and quick start
 - [x] **LICENSE** - MIT License for open source distribution
 - [x] **CONTRIBUTING.md** - Detailed contributor guidelines
 - [x] **.gitignore** - Comprehensive ignore rules for all file types
 - [x] **requirements.txt** - All Python dependencies listed and tested
 ### ✅ **Source Code Organization**
 - [x] **src/** directory created with clean separation
 - [x] **scripts/** directory for utility scripts and system files
 - [x] **docs/** directory with comprehensive documentation
 - [x] **grafana/** directory with visualization configuration
 - [x] All temporary files removed (*.db, *.log, __pycache__)
 ### ✅ **Documentation Quality**
 - [x] **Installation guides** for all platforms and databases
 - [x] **Configuration examples** for 5 different database types
 - [x] **Troubleshooting guides** for common deployment issues
 - [x] **Migration guides** for updating existing systems
 - [x] **API references** documenting Thai government data sources
 - [x] **Notable documents** section with official resources
 ### ✅ **Production Readiness**
 - [x] **Docker support** with Dockerfile and docker-compose
 - [x] **Systemd service** configuration for Linux deployment
 - [x] **Multi-database support** (SQLite, PostgreSQL, MySQL, InfluxDB, VictoriaMetrics)
 - [x] **Geolocation support** for Grafana geomap visualization
 - [x] **Migration scripts** for safe database schema updates
 - [x] **HTTPS configuration** guide for secure deployment
 ### ✅ **Code Quality**
 - [x] **Modular architecture** with clean separation of concerns
 - [x] **Error handling** and comprehensive logging
 - [x] **Configuration management** via environment variables
 - [x] **Database abstraction** layer for multiple backends
 - [x] **Testing utilities** (demo_databases.py)
 ### ✅ **Features Verified**
 - [x] **Real-time data collection** from 16 Thai water stations
 - [x] **15-minute scheduling** with intelligent retry logic
 - [x] **Gap filling** for missing historical data
 - [x] **Data validation** and error recovery
 - [x] **Geolocation integration** with sample coordinates
 - [x] **Grafana dashboards** with pre-built visualizations
 ## 🚀 **Ready for GitHub Publication**
 ### **Repository Information**
 - **Name**: `thailand-water-monitor`
 - **Description**: "Real-time water level monitoring system for Thailand's Royal Irrigation Department stations with Grafana visualization"
 - **Topics**: `water-monitoring`, `thailand`, `grafana`, `timeseries`, `python`, `iot`, `environmental-monitoring`
 - **License**: MIT
 - **Language**: Python
 ### **Repository Settings**
 - [x] Enable Issues for bug reports and feature requests
 - [x] Enable Discussions for community support
 - [x] Enable Wiki for extended documentation
 - [x] Set up GitHub Pages for documentation hosting
 - [x] Configure branch protection for main branch
 ### **Initial Release (v1.0.0)**
 - **Release Title**: "Thailand Water Level Monitor v1.0.0 - Complete Monitoring Solution"
 - **Release Notes**: 
  - Complete real-time monitoring system
  - Multi-database backend support
  - Grafana geomap integration
  - Production-ready deployment
  - Comprehensive documentation
 ## 📊 **Project Statistics**
 ### **Code Metrics**
 - **Total Files**: 25+ files
 - **Python Source Files**: 4 main modules
 - **Documentation Files**: 12 comprehensive guides
 - **Configuration Files**: 6 deployment configurations
 - **Lines of Code**: ~2,000+ lines of Python
 - **Documentation**: ~15,000+ words
 ### **Feature Coverage**
 - **Database Backends**: 5 different types supported
 - **Monitoring Stations**: 16 across Thailand
 - **Data Collection**: Every 15 minutes
 - **Data Points**: ~300 measurements per collection cycle
 - **Geolocation**: GPS coordinates and geohash support
 - **Visualization**: Pre-built Grafana dashboards
 ### **Documentation Coverage**
 - **Installation**: Complete setup for all platforms
 - **Configuration**: All database types documented
 - **Deployment**: Docker, systemd, manual options
 - **Troubleshooting**: Common issues and solutions
 - **Migration**: Safe upgrade procedures
 - **API**: External data source documentation
 ## 🌟 **Key Selling Points**
 ### **For Water Management Professionals**
 - Real-time monitoring of 16 stations across Thailand
 - Historical data analysis and trend visualization
 - Alert capabilities for critical water levels
 - Integration with official Thai government data sources
 ### **For Developers**
 - Clean, modular Python codebase
 - Multiple database backend options
 - Docker containerization for easy deployment
 - Comprehensive API documentation
 ### **For System Administrators**
 - Production-ready deployment configurations
 - Systemd service integration
 - HTTPS and security configuration
 - Monitoring and logging capabilities
 ### **For Data Scientists**
 - Time-series data with geolocation
 - Grafana visualization and analysis tools
 - Historical data gap filling
 - Export capabilities for further analysis
 ## 🎯 **Post-Publication Roadmap**
 ### **Immediate (Week 1)**
 - [ ] Create GitHub repository and upload files
 - [ ] Set up initial release v1.0.0
 - [ ] Configure repository settings and templates
 - [ ] Create project documentation website
 ### **Short-term (Month 1)**
 - [ ] Add GitHub Actions for CI/CD
 - [ ] Create issue and PR templates
 - [ ] Set up automated testing
 - [ ] Add code quality badges
 ### **Medium-term (Quarter 1)**
 - [ ] Community feedback integration
 - [ ] Additional database backends
 - [ ] Mobile app development
 - [ ] Advanced alerting system
 ### **Long-term (Year 1)**
 - [ ] Predictive analytics features
 - [ ] Machine learning integration
 - [ ] Multi-country expansion
 - [ ] Commercial support options
 ## 🏆 **Success Metrics**
 ### **Community Engagement**
 - GitHub stars and forks
 - Issue reports and feature requests
 - Community contributions
 - Documentation feedback
 ### **Technical Adoption**
 - Download and deployment statistics
 - Database backend usage patterns
 - Performance benchmarks
 - User success stories
 ### **Impact Measurement**
 - Water management improvements
 - Early warning system effectiveness
 - Data accessibility improvements
 - Research and academic usage
 ---
 ## ✅ **FINAL VERIFICATION**
 **All checklist items completed successfully!**
 The Thailand Water Level Monitor project is now:
 - ✅ **Professionally organized** with clean structure
 - ✅ **Comprehensively documented** with guides for all use cases
 - ✅ **Production ready** with multiple deployment options
 - ✅ **Community friendly** with contribution guidelines
 - ✅ **Feature complete** with real-time monitoring capabilities
 **🚀 Ready for GitHub publication and community engagement!** 🌊
 ---
 *Last updated: July 30, 2025*
 *Project status: Ready for publication*
@@ -1,233 +0,0 @@
 # 🎉 Gitea Actions Setup Complete!
 ## 🚀 **What's Been Created**
 Your **Northern Thailand Ping River Monitor** now has a complete CI/CD pipeline with Gitea Actions! Here's what's been set up:
 ### **🔄 Gitea Actions Workflows**
 ```
 .gitea/workflows/
 ├── ci.yml          # Main CI/CD pipeline
 ├── release.yml     # Automated releases
 ├── security.yml    # Security & dependency scanning
 └── docs.yml        # Documentation generation
 ```
 ### **📊 Workflow Features**
 #### **1. CI/CD Pipeline (`ci.yml`)**
 - ✅ **Multi-Python Testing** (3.9, 3.10, 3.11, 3.12)
 - ✅ **Code Quality Checks** (flake8, mypy, black, isort)
 - ✅ **Docker Multi-Arch Builds** (amd64, arm64)
 - ✅ **Integration Testing** with VictoriaMetrics
 - ✅ **Automated Staging Deployment** (develop branch)
 - ✅ **Manual Production Deployment** (main branch)
 - ✅ **Performance Testing** after deployment
 #### **2. Release Management (`release.yml`)**
 - 🏷️ **Tag-Based Releases** (`v*.*.*` pattern)
 - 📝 **Automatic Changelog Generation**
 - 🐳 **Multi-Architecture Docker Images**
 - 🔒 **Security Scanning** before release
 - ✅ **Comprehensive Validation** after deployment
 #### **3. Security Monitoring (`security.yml`)**
 - 🔒 **Daily Security Scans** (3 AM UTC)
 - 📦 **Dependency Vulnerability Detection**
 - 🐳 **Docker Image Security Scanning**
 - 📄 **License Compliance Checking**
 - 📊 **Code Quality Metrics**
 - 🔄 **Automated Update Notifications**
 #### **4. Documentation (`docs.yml`)**
 - 📚 **API Documentation Generation**
 - 🔗 **Link Validation**
 - 📖 **Sphinx Documentation Building**
 - ✅ **Documentation Completeness Checking**
 ## 🔧 **Setup Instructions**
 ### **1. Configure Repository Secrets**
 In your Gitea repository settings, add these secrets:
 ```bash
 # Required
 GITEA_TOKEN          # For container registry access
 # Optional (for notifications)
 SLACK_WEBHOOK_URL    # Slack notifications
 STAGING_WEBHOOK_URL  # Staging deployment webhook
 PRODUCTION_WEBHOOK_URL # Production deployment webhook
 ```
 ### **2. Enable Actions**
 1. Go to your repository settings in Gitea
 2. Enable "Actions" if not already enabled
 3. Configure runners if using self-hosted runners
 ### **3. Push to Repository**
 ```bash
 # Initialize and push
 git init
 git remote add origin https://git.b4l.co.th/grabowski/Northern-Thailand-Ping-River-Monitor.git
 git add .
 git commit -m "Initial commit with Gitea Actions workflows"
 git push -u origin main
 ```
 ## 🎯 **Workflow Triggers**
 ### **Automatic Triggers**
 - **Push to main/develop** → CI/CD Pipeline
 - **Pull Request to main** → Testing & Validation
 - **Daily at 2 AM UTC** → CI/CD Health Check
 - **Daily at 3 AM UTC** → Security Scanning
 - **Git Tag `v*.*.*`** → Release Pipeline
 - **Documentation Changes** → Documentation Build
 ### **Manual Triggers**
 - **Manual Dispatch** → Any workflow can be triggered manually
 - **Release Creation** → Manual release with custom version
 ## 📊 **Monitoring & Status**
 ### **Status Badges**
 Your README now includes comprehensive status badges:
 - CI/CD Pipeline Status
 - Security Scan Status  
 - Documentation Build Status
 - Python Version Support
 - FastAPI Version
 - Docker Ready
 - License Information
 - Current Version
 ### **Workflow Artifacts**
 Each workflow generates useful artifacts:
 - **Test Results** and coverage reports
 - **Security Scan Reports** (JSON format)
 - **Docker Images** (multi-architecture)
 - **Documentation** (HTML and PDF)
 - **Performance Reports**
 ## 🚀 **Usage Examples**
 ### **Development Workflow**
 ```bash
 # Create feature branch
 git checkout -b feature/new-station-type
 # Make changes
 git add .
 git commit -m "Add support for new station type"
 git push origin feature/new-station-type
 # Create PR in Gitea → Triggers testing
 ```
 ### **Release Workflow**
 ```bash
 # Create and push release tag
 git tag v3.1.1
 git push origin v3.1.1
 # → Triggers automated release pipeline
 ```
 ### **Security Monitoring**
 - **Daily scans** run automatically
 - **Security reports** available in Actions artifacts
 - **Notifications** sent for critical vulnerabilities
 ## 🔍 **Validation Commands**
 Test your setup locally:
 ```bash
 # Validate workflow syntax
 make validate-workflows
 # Test workflow components
 make workflow-test
 # Run full test suite
 make test
 # Build Docker image
 make docker-build
 ```
 ## 📈 **Performance & Optimization**
 ### **Caching Strategy**
 - **Pip dependencies** cached across runs
 - **Docker layers** cached for faster builds
 - **Workflow artifacts** retained for analysis
 ### **Parallel Execution**
 - **Matrix builds** for multiple Python versions
 - **Independent jobs** for security and testing
 - **Conditional execution** to skip unnecessary steps
 ### **Resource Management**
 - **Appropriate timeouts** prevent hanging workflows
 - **Artifact cleanup** manages storage usage
 - **Efficient Docker builds** with multi-stage approach
 ## 🔒 **Security Best Practices**
 ### **Implemented Security**
 - ✅ **Secret management** via Gitea repository secrets
 - ✅ **Multi-stage Docker builds** for minimal attack surface
 - ✅ **Non-root containers** for better security
 - ✅ **Vulnerability scanning** before deployment
 - ✅ **Dependency monitoring** with automated alerts
 ### **Security Scanning Coverage**
 - **Python dependencies** (Safety, Bandit)
 - **Docker images** (Trivy)
 - **Code quality** (Semgrep)
 - **License compliance** (pip-licenses)
 ## 📚 **Documentation**
 ### **Available Documentation**
 - [Gitea Workflows Guide](docs/GITEA_WORKFLOWS.md) - Detailed workflow documentation
 - [Contributing Guide](CONTRIBUTING.md) - How to contribute
 - [Deployment Checklist](DEPLOYMENT_CHECKLIST.md) - Production deployment
 - [Project Structure](docs/PROJECT_STRUCTURE.md) - Architecture overview
 ### **Generated Documentation**
 - **API Documentation** - Auto-generated from OpenAPI spec
 - **Code Documentation** - Sphinx-generated from docstrings
 - **Security Reports** - Automated vulnerability reports
 ## 🎉 **Ready for Production!**
 Your repository is now equipped with:
 - 🔄 **Enterprise-grade CI/CD pipeline**
 - 🔒 **Comprehensive security monitoring**
 - 📊 **Automated quality assurance**
 - 🚀 **Streamlined release management**
 - 📚 **Automated documentation**
 - 🐳 **Multi-architecture Docker support**
 - 📈 **Performance monitoring**
 - 🔍 **Comprehensive testing**
 ## 🚀 **Next Steps**
 1. **Push to Gitea** and watch the workflows run
 2. **Configure deployment environments** (staging/production)
 3. **Set up monitoring dashboards** for workflow metrics
 4. **Configure notifications** for team collaboration
 5. **Create your first release** with `git tag v3.1.3`
 Your **Northern Thailand Ping River Monitor** is now ready for professional development and deployment! 🎊
 ---
 **Workflow Version**: v3.1.3  
 **Setup Date**: 2025-08-12  
 **Repository**: https://git.b4l.co.th/grabowski/Northern-Thailand-Ping-River-Monitor
@@ -1,203 +0,0 @@
 # GitHub Publication Summary
 This document summarizes the Thailand Water Level Monitor project preparation for GitHub publication.
 ## 📁 **Final Project Structure**
 ```
 thailand-water-monitor/
 ├── 📄 README.md                           # Main project documentation
 ├── 📄 LICENSE                             # MIT License
 ├── 📄 CONTRIBUTING.md                     # Contributor guidelines
 ├── 📄 requirements.txt                    # Python dependencies
 ├── 📄 .gitignore                          # Git ignore rules
 ├── 📄 Dockerfile                          # Container definition
 ├── 📄 docker-compose.victoriametrics.yml # Complete stack deployment
 │
 ├── 📂 src/                                # Source Code
 │   ├── 🐍 water_scraper_v3.py           # Main application
 │   ├── 🐍 database_adapters.py          # Multi-database support
 │   ├── 🐍 config.py                     # Configuration management
 │   └── 🐍 demo_databases.py             # Database testing utility
 │
 ├── 📂 scripts/                           # Utility Scripts
 │   ├── 🐍 migrate_geolocation.py        # Database migration script
 │   └── ⚙️ water-monitor.service          # Systemd service file
 │
 ├── 📂 docs/                              # Documentation
 │   ├── 📖 DATABASE_DEPLOYMENT_GUIDE.md   # Complete setup guide
 │   ├── 📖 ENHANCED_SCHEDULER_GUIDE.md    # 15-minute scheduling
 │   ├── 📖 GEOLOCATION_GUIDE.md          # Grafana geomap integration
 │   ├── 📖 GAP_FILLING_GUIDE.md          # Data integrity management
 │   ├── 📖 MIGRATION_QUICKSTART.md       # Quick migration guide
 │   ├── 📖 VICTORIAMETRICS_SETUP.md      # High-performance deployment
 │   ├── 📖 HTTPS_CONFIGURATION.md        # Secure deployment
 │   ├── 📖 DEBIAN_TROUBLESHOOTING.md     # Linux deployment issues
 │   ├── 📖 PROJECT_STATUS.md             # Development status
 │   └── 📂 references/
 │       └── 📖 NOTABLE_DOCUMENTS.md       # Official Thai government resources
 │
 └── 📂 grafana/                          # Grafana Configuration
    ├── 📂 dashboards/
    │   └── 📊 water-monitoring-dashboard.json
    └── 📂 provisioning/
        ├── 📂 dashboards/
        │   └── ⚙️ dashboard.yml
        └── 📂 datasources/
            └── ⚙️ victoriametrics.yml
 ```
 ## ✅ **GitHub Readiness Checklist**
 ### **Core Files**
 - ✅ **README.md** - Comprehensive project documentation with badges, features, quick start
 - ✅ **LICENSE** - MIT License for open source distribution
 - ✅ **CONTRIBUTING.md** - Detailed contributor guidelines and development setup
 - ✅ **.gitignore** - Comprehensive ignore rules for Python, databases, logs, IDE files
 - ✅ **requirements.txt** - All Python dependencies listed
 ### **Source Code Organization**
 - ✅ **src/** directory - Clean separation of source code
 - ✅ **scripts/** directory - Utility scripts and system files
 - ✅ **docs/** directory - Comprehensive documentation
 - ✅ **grafana/** directory - Visualization configuration
 ### **Documentation Quality**
 - ✅ **Installation guides** - Multiple deployment options
 - ✅ **Configuration examples** - All database types covered
 - ✅ **Troubleshooting guides** - Common issues and solutions
 - ✅ **Migration guides** - Updating existing systems
 - ✅ **API references** - External data sources documented
 ### **Production Readiness**
 - ✅ **Docker support** - Containerization ready
 - ✅ **Systemd service** - Linux service configuration
 - ✅ **Multi-database support** - 5 different database options
 - ✅ **Geolocation support** - Grafana geomap integration
 - ✅ **Migration scripts** - Safe database updates
 ## 🌟 **Key Features for GitHub**
 ### **Real-time Monitoring**
 - 16 water stations across Thailand
 - 15-minute data collection frequency
 - Automatic gap filling and data validation
 - Multi-database backend support
 ### **Visualization Ready**
 - Pre-built Grafana dashboards
 - Geomap integration with coordinates
 - Real-time alerts and notifications
 - Historical trend analysis
 ### **Production Deployment**
 - Docker containerization
 - VictoriaMetrics high-performance backend
 - HTTPS and security configuration
 - Comprehensive logging and monitoring
 ### **Developer Friendly**
 - Clean, modular code structure
 - Comprehensive documentation
 - Multiple database adapters
 - Easy local development setup
 ## 📊 **Project Statistics**
 ### **Code Metrics**
 - **Python Files**: 4 main source files
 - **Documentation**: 10+ comprehensive guides
 - **Database Support**: 5 different backends
 - **Monitoring Stations**: 16 across Thailand
 - **Data Points**: ~300 every 15 minutes
 ### **Documentation Coverage**
 - **Installation**: Complete setup guides for all platforms
 - **Configuration**: All database types documented
 - **Deployment**: Docker, systemd, and manual options
 - **Troubleshooting**: Common issues and solutions
 - **Migration**: Safe upgrade procedures
 ### **Features Implemented**
 - ✅ Real-time data collection
 - ✅ Multi-database support
 - ✅ Geolocation integration
 - ✅ Gap filling and data validation
 - ✅ Grafana visualization
 - ✅ Docker deployment
 - ✅ Production monitoring
 - ✅ Migration tools
 ## 🚀 **Ready for GitHub Publication**
 ### **Repository Setup**
 1. **Create GitHub repository** - "thailand-water-monitor"
 2. **Upload all files** - Complete project structure
 3. **Configure repository settings**:
   - Add description: "Real-time water level monitoring for Thailand's RID stations"
   - Add topics: `water-monitoring`, `thailand`, `grafana`, `timeseries`, `python`
   - Enable Issues and Discussions
   - Set up GitHub Pages for documentation
 ### **Initial Release**
 - **Version**: v1.0.0
 - **Release Notes**: Complete feature set with multi-database support
 - **Assets**: Include sample configuration files
 - **Documentation**: Link to comprehensive guides
 ### **Community Features**
 - **Issues Template**: Bug reports and feature requests
 - **Pull Request Template**: Contribution guidelines
 - **Discussions**: Community support and questions
 - **Wiki**: Extended documentation and tutorials
 ## 🎯 **Post-Publication Tasks**
 ### **Community Building**
 - Create detailed issue templates
 - Set up GitHub Actions for CI/CD
 - Add code quality badges
 - Create project roadmap
 ### **Documentation Enhancement**
 - Add video tutorials
 - Create API documentation
 - Add performance benchmarks
 - Create deployment examples
 ### **Feature Development**
 - Mobile app integration
 - Additional database backends
 - Advanced alerting system
 - Predictive analytics
 ## 📞 **Support Channels**
 - **GitHub Issues**: Bug reports and feature requests
 - **GitHub Discussions**: Community support and questions
 - **Documentation**: Comprehensive guides in docs/ directory
 - **Examples**: Working configurations and deployments
 ## 🏆 **Project Highlights**
 ### **Technical Excellence**
 - Clean, modular architecture
 - Comprehensive error handling
 - Production-ready deployment
 - Multi-database abstraction
 ### **Documentation Quality**
 - Step-by-step installation guides
 - Troubleshooting for common issues
 - Migration procedures for updates
 - API and configuration references
 ### **Community Ready**
 - Open source MIT license
 - Contributor guidelines
 - Development setup instructions
 - Code quality standards
 ---
 **The Thailand Water Level Monitor project is now fully prepared for GitHub publication with a professional structure, comprehensive documentation, and production-ready features.** 🌊
@@ -1,114 +0,0 @@
 # 🔑 GitHub Token Setup Guide
 ## 🎯 **Why You Need This**
 The Gitea Actions workflows use Trivy for security scanning, which needs to download vulnerability databases from GitHub. Without a GitHub token, you'll hit rate limits and the security scans will fail.
 ## 🚀 **Quick Setup (5 minutes)**
 ### **Step 1: Create GitHub Personal Access Token**
 1. **Go to GitHub**: https://github.com/settings/tokens
 2. **Click "Generate new token"** → "Generate new token (classic)"
 3. **Configure the token**:
   - **Note**: `B4L Ping River Monitor - Gitea Actions`
   - **Expiration**: `90 days` (or longer)
   - **Scopes**: Select `public_repo` (for public repositories)
 4. **Click "Generate token"**
 5. **Copy the token** (you won't see it again!)
 ### **Step 2: Add Token to Gitea Repository**
 1. **Go to your repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
 2. **Click "Settings"** (in the repository)
 3. **Click "Secrets"** in the left sidebar
 4. **Click "Add Secret"**
 5. **Configure the secret**:
   - **Name**: `GITHUB_TOKEN`
   - **Value**: Paste the token you copied from GitHub
 6. **Click "Add Secret"**
 ### **Step 3: Verify It's Working**
 1. **Trigger a workflow** by pushing a commit or manually running the security workflow
 2. **Check the Actions tab** in your repository
 3. **Look for the message**: `✅ GITHUB_TOKEN is configured`
 ## 🔒 **Security Best Practices**
 ### **Token Permissions**
 - **Minimum required**: `public_repo` scope
 - **Never use**: `repo` scope unless you need private repo access
 - **Avoid**: Admin or write permissions
 ### **Token Management**
 - **Set expiration**: Don't create tokens that never expire
 - **Regular rotation**: Update tokens every 90 days
 - **Monitor usage**: Check GitHub token usage in settings
 ### **Repository Security**
 - **Only trusted contributors**: Should have access to repository secrets
 - **Audit regularly**: Review who has access to secrets
 - **Use organization secrets**: For multiple repositories
 ## 🧪 **Testing the Setup**
 ### **Manual Test**
 ```bash
 # Trigger the security workflow manually
 # Go to: Repository → Actions → Security & Dependency Updates → Run workflow
 ```
 ### **Automatic Test**
 ```bash
 # Push any change to trigger workflows
 git commit --allow-empty -m "Test GitHub token setup"
 git push
 ```
 ### **Check Workflow Logs**
 1. Go to Actions tab in your repository
 2. Click on the latest "Security & Dependency Updates" run
 3. Click on "Docker Security Scan" job
 4. Look for: `✅ GITHUB_TOKEN is configured`
 ## ❌ **Troubleshooting**
 ### **"GITHUB_TOKEN not configured" message**
 - **Problem**: Token not added to repository secrets
 - **Solution**: Follow Step 2 above, ensure exact name `GITHUB_TOKEN`
 ### **"Bad credentials" error**
 - **Problem**: Token is invalid or expired
 - **Solution**: Generate a new token and update the secret
 ### **Rate limit errors**
 - **Problem**: Token doesn't have correct permissions
 - **Solution**: Ensure token has `public_repo` scope
 ### **Trivy still failing**
 - **Problem**: Network issues or GitHub API problems
 - **Solution**: Wait and retry, or check GitHub status page
 ## 🎉 **Success Indicators**
 When everything is working correctly, you'll see:
 ✅ **In workflow logs**: `✅ GITHUB_TOKEN is configured`  
 ✅ **Security scans**: Complete without authentication errors  
 ✅ **Trivy reports**: Generated and uploaded as artifacts  
 ✅ **No rate limit errors**: In the workflow execution  
 ## 📚 **Additional Resources**
 - [GitHub Personal Access Tokens Documentation](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
 - [Gitea Secrets Documentation](https://docs.gitea.io/en-us/usage/actions/#secrets)
 - [Trivy Action Documentation](https://github.com/aquasecurity/trivy-action)
 ---
 **Setup Time**: ~5 minutes  
 **Token Validity**: 90 days (recommended)  
 **Security Level**: High (read-only public repo access)  
 Your workflows will now run smoothly with proper GitHub API authentication! 🚀
@@ -120,10 +120,6 @@ docker-logs:
 docs:
 	cd docs && make html
 # Database management
 db-migrate:
 	uv run python scripts/migrate_geolocation.py
 # Monitoring
 health-check:
 	curl -f http://localhost:8000/health || exit 1
@@ -149,9 +145,6 @@ setup-postgres:
 test-postgres:
 	uv run python -c "from scripts.setup_postgres import test_postgres_connection; from src.config import Config; config = Config.get_database_config(); test_postgres_connection(config['connection_string'])"
 encode-password:
 	uv run python scripts/encode_password.py
 migrate-sqlite:
 	uv run python scripts/migrate_sqlite_to_postgres.py
@@ -161,16 +154,6 @@ migrate-fast:
 analyze-sqlite:
 	uv run python scripts/migrate_sqlite_to_postgres.py --dry-run
 # Distribution
 build-exe:
 	uv run python build_simple.py
 package: build-exe
 	@echo "Creating distribution package..."
 	@if exist dist\ping-river-monitor-distribution.zip del dist\ping-river-monitor-distribution.zip
 	@cd dist && powershell -Command "Compress-Archive -Path * -DestinationPath ping-river-monitor-distribution.zip -Force"
 	@echo "✅ Distribution package created: dist/ping-river-monitor-distribution.zip"
 # Git helpers
 git-setup:
 	git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
@@ -1,509 +1,157 @@
-# Northern Thailand Ping River Monitor 🏔️
+# Northern Thailand Ping River Monitor
-A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.
+Live water levels, discharge, rainfall and machine-learning flood forecasts for the
 Ping River basin around Chiang Mai. Collects hourly gauge data from public sources,
 keeps the full history in PostgreSQL, and serves a bilingual dashboard, an open REST
 API, and 6/12/24-hour flood-risk forecasts per gauge.
-[![CI/CD](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Documentation](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [![Python](https://img.shields.io/badge/Python-3.9+-blue.svg)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-0.104+-green.svg)](https://fastapi.tiangolo.com) [![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Version](https://img.shields.io/badge/Version-v3.1.3-blue.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/releases)
+**Live: [water.buildfor.life](https://water.buildfor.life/)** · API reference at
 [/docs](https://water.buildfor.life/docs) · built by [buildfor.life](https://buildfor.life)
 after the [October 2024 flood](https://buildfor.life/blog/chiang-mai-flood-2024/) —
 background in [Teaching a Model to See the Ping River Rise 13 Hours Early](https://buildfor.life/blog/ping-river-monitor/).
-## 🌟 Features
+[![CI](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/ci.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Security](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/security.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Docs](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions/workflows/docs.yml/badge.svg)](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions)
 [![Python 3.11](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
-### 📊 **Real-time Data Collection**
+## What it does
 - **16 Monitoring Stations** across Thailand
 - **15-minute Collection Frequency** with intelligent scheduling
 - **Automatic Gap Filling** for missing historical data
 - **Data Validation** and error recovery mechanisms
 - **Rate Limiting** to prevent API abuse
-### 🌐 **Web API Interface (NEW!)**
+- **Collects** hourly water level and discharge from 16 Royal Irrigation Department
- **FastAPI-powered REST API** with interactive documentation
+  (RID) telemetry gauges, Chiang Dao to the southern basin, since 2018-08; hourly
- **Station Management** - Add, update, and remove monitoring stations
+  rainfall and water level from 400+ ThaiWater/HII stations; Open-Meteo catchment
- **Real-time health monitoring** and system status
+  rainfall (archive + 48 h forecast); daily Mae Ngat reservoir state. Every source and
- **Manual data collection triggers** via web interface
+  its quirks: [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md).
- **Comprehensive metrics** and performance monitoring
+- **Fills gaps.** The raw RID grid had readings for ~56 % of hours; a full-history
- **CORS support** for web applications
+  re-fetch plus HII cross-fill brought it to ~93 %. `GET /api/stats` reports the
  current figure.
 - **Forecasts.** Per gauge and horizon, a gradient-boosted model predicts the rise
  within 6/12/24 h and the probability of crossing the station's warning and danger
  levels. Trained on the monitor's own history plus catchment rain; evaluated
  rolling-origin, event by event. On the October 2024 record flood, trained only on
  data through August 2024, the first alert came **13 hours before** P.1 crossed
  3.70 m. Everything about the model, including what did not work:
  [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md).
 - **Shows it.** A Leaflet map with the river drawn as OSM geometry and styled by live
  discharge, rain gauges, the Chiang Mai inundation zones, per-station history, the
  forecast card, a replay of the 2024 flood, English/Thai, light/dark.
 - **Notifies.** Public push alerts over a self-hosted [ntfy](https://ntfy.sh): one
  message when a gauge crosses its warning or danger level, one all-clear on the
  way down, an opt-in early-warning topic from the model, nothing in between.
  Subscribe from the free app, no account. Matrix room alerts for a team are
  also supported.
-### 🗄️ **Multi-Database Support**
+## Quick start
 - **VictoriaMetrics** (Recommended) - High-performance time-series
 - **InfluxDB** - Purpose-built time-series database
 - **PostgreSQL + TimescaleDB** - Relational with time-series optimization
 - **MySQL** - Traditional relational database
 - **SQLite** - Local development and testing
-### 🗺️ **Geolocation Support**
+Python **3.11** (3.13 breaks the pinned `psycopg2-binary`), PostgreSQL for anything
- **Grafana Geomap** integration ready
+beyond a quick look, [uv](https://docs.astral.sh/uv/).
 - **GPS coordinates** and geohash support
 - **Interactive mapping** of water stations
 ### 📈 **Visualization & Monitoring**
 - **Pre-built Grafana dashboards**
 - **Real-time alerts** and notifications
 - **Historical trend analysis**
 - **Built-in metrics collection** (counters, gauges, histograms)
 - **Health checks** for database, API, and system resources
 ### 🚀 **Production Ready**
 - **Docker containerization** with multi-service support
 - **Systemd service** configuration
 - **HTTPS support** with SSL certificates
 - **Comprehensive logging** with rotation and colored output
 - **Type safety** with Pydantic models and type hints
 - **Custom exception handling** for better error management
 ## 🚀 Quick Start
 ### Prerequisites
 - Python 3.9 or higher
 - Internet connection for data fetching
 - Database server (optional - SQLite works out of the box)
 ### Installation
 ```bash
 # Clone the repository
 git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 cd Northern-Thailand-Ping-River-Monitor
-
+uv sync --python 3.11
-# Quick setup with Make
+cp .env.example .env            # DB_TYPE, POSTGRES_CONNECTION_STRING, optional MATRIX_*
-make dev-setup
+uv run python run.py --web-api  # dashboard + API on http://localhost:8000
 # Or manual setup:
 python -m venv venv
 source venv/bin/activate  # Windows: venv\Scripts\activate
 pip install -r requirements.txt
 cp .env.example .env
 ```
-### Basic Usage
+`DB_TYPE=sqlite` works for the dashboard and API; the forecasting path expects the
 PostgreSQL history.
 ```bash
-# Test run with SQLite (default)
+uv run python run.py --status          # collector status
-make run-test
+uv run python run.py --test            # one collection cycle
-# or: python run.py --test
+uv run python run.py --fill-gaps 7     # re-fetch the last 7 days from RID
-
+uv run python run.py --collect-hii     # one ThaiWater/HII collection cycle
-# Run continuous monitoring
+uv run python run.py --alert-check     # evaluate thresholds, notify Matrix
-make run
+uv run python scripts/train_flood_model.py --stations all   # retrain (~12 min)
-# or: python run.py
+make test                              # pytest, synthetic data, no network
-
+make format                            # black + isort (the CI contract)
 # Start web API server (NEW!)
 make run-api
 # or: python run.py --web-api
 # Run all tests
 make test
 # Demo different databases
 python src/demo_databases.py
 ```
-### 🌐 Web API Interface (NEW!)
+## API
-The system now includes a comprehensive FastAPI web interface:
+Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
 Asia/Bangkok wall-clock without an offset suffix.
 | Endpoint | Returns |
 | --- | --- |
 | `GET /stations` | The 16 RID gauges: code, Thai/English names, coordinates |
 | `GET /measurements/latest?limit=N` | Newest reading per station |
 | `GET /measurements/history/{code}?hours=N` | Hourly history; or `?start=YYYY-MM-DD&end=YYYY-MM-DD`; `limit` ≤ 100000 |
 | `GET /forecast` | Current flood-risk forecast, every station × horizon, with thresholds and P.1 inundation-stage probabilities |
 | `GET /api/forecast/history/{code}?hours=N&horizon=24` | Forecasts as issued, for auditing lead time after the fact |
 | `GET /api/hii/rainfall/latest`, `/api/hii/waterlevel/latest` | Latest ThaiWater/HII gauge readings |
 | `GET /api/hii/rainfall/catchment?days=N` | HII gauge catchment-mean rain next to the Open-Meteo series the model uses |
 | `GET /api/forecast/skill?station_code=P.1` | Issued forecasts vs what happened, per deployed model version |
 | `GET /api/notifications` | ntfy server and topic names for the subscribe panel |
 | `GET /api/stats` | Row counts per source, date range, coverage |
 | `GET /health` | DB / upstream / memory checks |
 Interactive reference with schemas: [water.buildfor.life/docs](https://water.buildfor.life/docs).
 Responses are cached briefly server-side; poll no faster than once a minute — the data
 changes hourly.
 ## Deployment
 Production is a systemd unit on a small VPS behind Cloudflare, updated by `git pull`.
 `scripts/install.sh` (run as root from a checkout) creates the `water-monitor` user,
 deploys to `/opt/thailand-water-monitor`, runs `uv sync` into `.venv`, installs
 `water-monitor.service` and the monthly `water-monitor-retrain.timer`.
 ```bash
 # Start the web API
 python run.py --web-api
 # Access the API at:
 # - Dashboard: http://localhost:8000
 # - Interactive docs: http://localhost:8000/docs
 # - Health check: http://localhost:8000/health
 # - Latest data: http://localhost:8000/measurements/latest
 ```
 **Key API Endpoints:**
 - `GET /` - Web dashboard
 - `GET /health` - System health status
 - `GET /metrics` - Application metrics
 - `GET /stations` - List all monitoring stations
 - `POST /stations` - Add new monitoring station
 - `PUT /stations/{id}` - Update station information
 - `DELETE /stations/{id}` - Remove monitoring station
 - `GET /measurements/latest` - Latest measurements
 - `GET /measurements/station/{code}` - Station-specific data
 - `POST /scrape/trigger` - Trigger manual data collection
 ## 📊 Station Information
 The system monitors **16 water stations** along the Ping River Basin in Northern Thailand:
 | Station | Thai Name | English Name | Location |
 |---------|-----------|--------------|----------|
 | P.1 | สะพานนวรัฐ | Nawarat Bridge | Nakhon Sawan |
 | P.5 | สะพานท่านาง | Tha Nang Bridge | - |
 | P.20 | บ้านเชียงดาว | Ban Chiang Dao | Chiang Mai |
 | P.21 | บ้านริมใต้ | Ban Rim Tai | - |
 | P.4A | บ้านแม่แตง | Ban Mae Taeng | Chiang Mai |
 | P.67 | บ้านแม่แต | Ban Tae | - |
 | P.75 | บ้านช่อแล | Ban Chai Lat | - |
 | P.76 | บ้านแม่อีไฮ | Banb Mae I Hai | - |
 | P.77 | บ้านสบแม่สะป๊วด | Baan Sop Mae Sapuord | - |
 | P.81 | บ้านโป่ง | Ban Pong | - |
 | P.82 | บ้านสบวิน | Ban Sob win | - |
 | P.84 | บ้านพันตน | Ban Panton | - |
 | P.85 | บ้านหล่ายแก้ว | Baan Lai Kaew | - |
 | P.87 | บ้านป่าซาง | Ban Pa Sang | - |
 | P.92 | บ้านเมืองกึ๊ด | Ban Muang Aut | - |
 | P.103 | สะพานวงแหวนรอบ 3 | Ring Bridge 3 | Bangkok |
 ### Data Metrics
 - **Water Level**: Measured in meters (m)
 - **Discharge**: Flow rate in cubic meters per second (cms)
 - **Discharge Percentage**: Relative to station capacity
 - **Timestamp**: Thai time (UTC+7) with Buddhist calendar support
 ## 🗄️ Database Configuration
 ### VictoriaMetrics (Recommended)
 **High-performance time-series database with excellent compression and query speed.**
 ```bash
 # Environment variables
 export DB_TYPE=victoriametrics
 export VM_HOST=localhost
 export VM_PORT=8428
 # Quick start with Docker
 docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428
 ```
 ### Complete Stack with Grafana
 ```bash
 # Start the complete monitoring stack
 docker-compose -f docker-compose.victoriametrics.yml up -d
 # Access Grafana at http://localhost:3000
 # Username: admin, Password: admin_password
 ```
 ### Other Database Options
 <details>
 <summary>InfluxDB Configuration</summary>
 ```bash
 export DB_TYPE=influxdb
 export INFLUX_HOST=localhost
 export INFLUX_PORT=8086
 export INFLUX_DATABASE=water_monitoring
 export INFLUX_USERNAME=water_user
 export INFLUX_PASSWORD=your_password
 ```
 </details>
 <details>
 <summary>PostgreSQL Configuration</summary>
 ```bash
 export DB_TYPE=postgresql
 export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring
 ```
 </details>
 <details>
 <summary>MySQL Configuration</summary>
 ```bash
 export DB_TYPE=mysql
 export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring
 ```
 </details>
 ## 📈 Grafana Dashboards
 ### Pre-built Dashboard Features
 - **Real-time water levels** across all stations
 - **Historical trends** and patterns
 - **Discharge monitoring** with percentage indicators
 - **Station status** and health monitoring
 - **Geomap visualization** of station locations
 - **Alert thresholds** for critical water levels
 ### Sample Queries
 **VictoriaMetrics/Prometheus:**
 ```promql
 # Current water levels
 water_level
 # High discharge alerts
 water_discharge_percent > 80
 # Station-specific data
 water_level{station_code="P.1"}
 ```
 **SQL Databases:**
 ```sql
 -- Latest readings from all stations
 SELECT s.station_code, s.english_name, m.water_level, m.discharge
 FROM stations s
 JOIN water_measurements m ON s.id = m.station_id
 WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);
 ```
 ## 🚀 Production Deployment
 ### Docker Deployment
 ```bash
 # Build the image
 docker build -t thailand-water-monitor .
 # Run with environment variables
 docker run -d \
  --name water-monitor \
  -e DB_TYPE=victoriametrics \
  -e VM_HOST=victoriametrics \
  thailand-water-monitor
 ```
 ### Systemd Service (Linux)
 The install script sets everything up: a dedicated `water-monitor` system user,
 a deploy to `/opt/thailand-water-monitor`, a uv-managed virtualenv, and the
 enabled systemd unit.
 ```bash
 # From a checkout of the repo, as root:
 sudo bash scripts/install.sh
 # Then start and check:
 sudo systemctl start water-monitor.service
-systemctl status water-monitor.service
+systemctl list-timers water-monitor-retrain.timer
 ```
-Fill in `/opt/thailand-water-monitor/.env` (Matrix token/room, DB settings)
+The retrain timer runs `scripts/retrain.sh`, which trains into `models/.staging`,
-before starting if the script reports it is missing.
+refuses to promote anything that is not a rain-enabled (`hgb-v3+`) set covering the
 expected stations, and renames the bundles into place. Details and the operations
 runbook: [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) sections 6–8.
-<details>
+## Repository layout
 <summary>Manual setup (if you prefer not to use the script)</summary>
 ```bash
 sudo useradd --system --no-create-home --shell /usr/sbin/nologin water-monitor
 sudo cp scripts/water-monitor.service /etc/systemd/system/
 sudo systemctl enable water-monitor.service
 sudo systemctl start water-monitor.service
 ```
 </details>
 ### Migration for Existing Systems
 If you have an existing installation, use the migration script to add geolocation support:
 ```bash
 # Stop the service
 sudo systemctl stop water-monitor
 # Run migration
 python scripts/migrate_geolocation.py
 # Restart the service
 sudo systemctl start water-monitor
 ```
 ## 🔧 Command Line Tools
 ### Main Application
 ```bash
 python src/water_scraper_v3.py                    # Run continuous monitoring
 python src/water_scraper_v3.py --test             # Single test cycle
 python src/water_scraper_v3.py --help             # Show help
 ```
 ### Data Management
 ```bash
 python src/water_scraper_v3.py --check-gaps 7     # Check for missing data (7 days)
 python src/water_scraper_v3.py --fill-gaps 7      # Fill missing data gaps
 python src/water_scraper_v3.py --update-data 2    # Update existing data (2 days)
 ```
 ### Database Testing
 ```bash
 python src/demo_databases.py                      # SQLite demo
 python src/demo_databases.py victoriametrics      # VictoriaMetrics demo
 python src/demo_databases.py all                  # Test all databases
 ```
 ## 📚 Documentation
 ### Core Documentation
 - **[Installation Guide](docs/DATABASE_DEPLOYMENT_GUIDE.md)** - Complete setup instructions
 - **[Scheduler Guide](docs/ENHANCED_SCHEDULER_GUIDE.md)** - 15-minute scheduling system
 - **[Geolocation Guide](docs/GEOLOCATION_GUIDE.md)** - Grafana geomap integration
 - **[Gap Filling Guide](docs/GAP_FILLING_GUIDE.md)** - Data integrity management
 ### Deployment Guides
 - **[VictoriaMetrics Setup](docs/VICTORIAMETRICS_SETUP.md)** - High-performance deployment
 - **[HTTPS Configuration](docs/HTTPS_CONFIGURATION.md)** - Secure deployment
 - **[Debian Troubleshooting](docs/DEBIAN_TROUBLESHOOTING.md)** - Linux deployment issues
 ### References
 - **[Notable Documents](docs/references/NOTABLE_DOCUMENTS.md)** - Official Thai government resources
 - **[Migration Guide](docs/MIGRATION_QUICKSTART.md)** - Updating existing systems
 ## 🔍 Troubleshooting
 ### Common Issues
 **Database Connection Errors:**
 ```bash
 # Check database status
 python src/demo_databases.py
 # Test specific database
 python src/demo_databases.py victoriametrics
 ```
 **Missing Data:**
 ```bash
 # Check for gaps
 python src/water_scraper_v3.py --check-gaps 7
 # Fill missing data
 python src/water_scraper_v3.py --fill-gaps 7
 ```
 **Service Issues:**
 ```bash
 # Check service status
 sudo systemctl status water-monitor
 # View logs
 sudo journalctl -u water-monitor -f
 ```
 ### Health Checks
 ```bash
 # VictoriaMetrics health
 curl http://localhost:8428/health
 # Check latest data
 curl "http://localhost:8428/api/v1/query?query=water_level"
 # Application logs
 tail -f water_monitor.log
 ```
 ## 🌐 API Integration
 ### VictoriaMetrics API Examples
 ```bash
 # Query current water levels
 curl "http://localhost:8428/api/v1/query?query=water_level"
 # Query discharge rates for last hour
 curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"
 # Query specific station
 curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
 # High discharge alerts
 curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
 ```
 ## 📊 Performance
 ### System Requirements
 - **CPU**: 1-2 cores (minimal load)
 - **RAM**: 512MB - 2GB (depending on database)
 - **Storage**: 1GB+ (for historical data)
 - **Network**: Stable internet connection
 ### Performance Metrics
 - **Data Collection**: ~300 data points every 15 minutes
 - **Database Write Speed**: 1000+ points/second (VictoriaMetrics)
 - **Query Response**: <100ms for recent data
 - **Storage Efficiency**: 70x compression vs. raw data
 ## 🤝 Contributing
 Contributions are welcome! Please:
 1. Fork the repository
 2. Create a feature branch
 3. Make your changes
 4. Add tests if applicable
 5. Submit a pull request
 ### Development Setup
 ```bash
 # Clone your fork
 git clone https://github.com/your-username/thailand-water-monitor.git
 cd thailand-water-monitor
 # Install development dependencies
 pip install -r requirements.txt
 pip install pytest black flake8
 # Run tests
 pytest
 # Format code
 black src/
 ```
 ## 📄 License
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
 ## 🙏 Acknowledgments
 - **Royal Irrigation Department (RID)** of Thailand for providing the data API
 - **VictoriaMetrics** team for the excellent time-series database
 - **Grafana** team for the visualization platform
 - **Python community** for the amazing libraries and tools
 ## 📞 Support
 - **Issues**: [GitHub Issues](https://github.com/your-username/thailand-water-monitor/issues)
 - **Discussions**: [GitHub Discussions](https://github.com/your-username/thailand-water-monitor/discussions)
 - **Documentation**: [Project Wiki](https://github.com/your-username/thailand-water-monitor/wiki)
 ---
 ## 📁 Project Structure
 ```
-Northern-Thailand-Ping-River-Monitor/
+src/                 collector, API (web_api.py), dashboard (static/dashboard.html)
-├── src/                    # Main application code
+src/ml/              features, training, evaluation harness, prediction, rain/dam/HII loaders
-├── tests/                  # Test suite
+scripts/             train_flood_model.py, retrain.sh, evaluate_variants.py, install.sh, dev_proxy.py
-├── docs/                   # Documentation
+tests/               pytest suite (synthetic data; no DB or network)
-├── grafana/               # Grafana dashboards
+docs/                FLOOD_FORECASTING.md, DATA_SOURCES.md, deployment and station guides
-├── scripts/               # Utility scripts
+models/              trained bundles + metrics.json (gitignored) and evaluation results (tracked)
-├── docker-compose.yml     # Docker deployment
+.gitea/workflows/    ci (format/lint/tests), security (pip-audit/bandit), docs (link + OpenAPI checks)
 ├── Makefile              # Development tasks
 └── requirements.txt      # Dependencies
 ```
-See [docs/PROJECT_STRUCTURE.md](docs/PROJECT_STRUCTURE.md) for detailed architecture information.
+## Documentation
-## 🔄 CI/CD & Automation
+- [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md) — the model: data, features, evaluation, measured performance, negatives, deployment, retraining
 - [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) — every ingested and candidate source, endpoints, quirks
 - [docs/STATION_MANAGEMENT_GUIDE.md](docs/STATION_MANAGEMENT_GUIDE.md) — adding/editing gauges
 - [docs/DATABASE_DEPLOYMENT_GUIDE.md](docs/DATABASE_DEPLOYMENT_GUIDE.md), [POSTGRESQL_SETUP.md](POSTGRESQL_SETUP.md) — database setup
 - [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md) — public push alerts: topics, semantics, ntfy deployment
 - [docs/MATRIX_QUICK_START.md](docs/MATRIX_QUICK_START.md) — Matrix room alerts for a team
 - [docs/GAP_FILLING_GUIDE.md](docs/GAP_FILLING_GUIDE.md) — data integrity tooling
 - [docs/references/NOTABLE_DOCUMENTS.md](docs/references/NOTABLE_DOCUMENTS.md) — official Thai government resources
 - Public overview: [buildfor.life/docs/tooling/ping-river-monitor](https://buildfor.life/docs/tooling/ping-river-monitor/)
-The project includes comprehensive Gitea Actions workflows:
+Other database backends (VictoriaMetrics, InfluxDB, MySQL, SQLite) and the Grafana
 dashboards under `grafana/` are supported by the adapters but not what production
 runs; see [docs/VICTORIAMETRICS_SETUP.md](docs/VICTORIAMETRICS_SETUP.md) if you want them.
- **🧪 CI/CD Pipeline** - Automated testing, building, and deployment
+## Contributing
 - **🔒 Security Scanning** - Daily vulnerability and dependency checks
 - **📚 Documentation** - Automated API docs and validation
 - **🚀 Release Management** - Automated releases with multi-arch Docker builds
-See [docs/GITEA_WORKFLOWS.md](docs/GITEA_WORKFLOWS.md) for detailed workflow documentation.
+`make format` before committing (black 88 columns, isort black profile — the CI gate),
 `make test` must stay green, tests use synthetic data only. See
 [CONTRIBUTING.md](CONTRIBUTING.md). Issues and merge requests on
 [git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
-## 🔗 Repository
+## Data sources and thanks
- **Main Repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
+Royal Irrigation Department (RID) gauge telemetry; Hydro-Informatics Institute (HII) /
- **Issues**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
+ThaiWater open API; Open-Meteo; OpenStreetMap contributors for the river geometry;
- **Actions**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
+Chiang Mai Municipality for the inundation map the P.1 stages are keyed to. All
- **Documentation**: [docs/](docs/)
+instruments are theirs; we aggregate, store, fill gaps and forecast.
-**Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin**
+## License
 MIT — see [LICENSE](LICENSE).
@@ -1,311 +0,0 @@
 #!/usr/bin/env python3
 """
 Build script to create a standalone executable for Northern Thailand Ping River Monitor
 """
 import os
 import shutil
 import sys
 from pathlib import Path
 def create_spec_file():
    """Create PyInstaller spec file"""
    spec_content = """
 # -*- mode: python ; coding: utf-8 -*-
 block_cipher = None
 # Data files to include
 data_files = [
    ('.env', '.'),
    ('sql/*.sql', 'sql'),
    ('README.md', '.'),
    ('POSTGRESQL_SETUP.md', '.'),
    ('SQLITE_MIGRATION.md', '.'),
 ]
 # Hidden imports that PyInstaller might miss
 hidden_imports = [
    'psycopg2',
    'psycopg2-binary',
    'sqlalchemy.dialects.postgresql',
    'sqlalchemy.dialects.sqlite',
    'sqlalchemy.dialects.mysql',
    'influxdb',
    'pymysql',
    'dotenv',
    'pydantic',
    'fastapi',
    'uvicorn',
    'schedule',
    'pandas',
    'requests',
    'psutil',
 ]
 a = Analysis(
    ['run.py'],
    pathex=['.'],
    binaries=[],
    datas=data_files,
    hiddenimports=hidden_imports,
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[
        'tkinter',
        'matplotlib',
        'PIL',
        'jupyter',
        'notebook',
        'IPython',
    ],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
 )
 pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
 exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    [],
    name='ping-river-monitor',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
    icon='icon.ico' if os.path.exists('icon.ico') else None,
 )
 """
    with open("ping-river-monitor.spec", "w") as f:
        f.write(spec_content.strip())
    print("[OK] Created ping-river-monitor.spec")
 def install_pyinstaller():
    """Install PyInstaller if not present"""
    try:
        import PyInstaller
        print("[OK] PyInstaller already installed")
    except ImportError:
        print("Installing PyInstaller...")
        os.system("uv add --dev pyinstaller")
        print("[OK] PyInstaller installed")
 def build_executable():
    """Build the executable"""
    print("🔨 Building executable...")
    # Clean previous builds
    if os.path.exists("dist"):
        shutil.rmtree("dist")
    if os.path.exists("build"):
        shutil.rmtree("build")
    # Build with PyInstaller using uv
    result = os.system("uv run pyinstaller ping-river-monitor.spec --clean --noconfirm")
    if result == 0:
        print("✅ Executable built successfully!")
        # Copy additional files to dist directory
        dist_dir = Path("dist")
        if dist_dir.exists():
            # Copy .env file if it exists
            if os.path.exists(".env"):
                shutil.copy2(".env", dist_dir / ".env")
                print("✅ Copied .env file")
            # Copy documentation
            for doc in ["README.md", "POSTGRESQL_SETUP.md", "SQLITE_MIGRATION.md"]:
                if os.path.exists(doc):
                    shutil.copy2(doc, dist_dir / doc)
                    print(f"✅ Copied {doc}")
            # Copy SQL files
            if os.path.exists("sql"):
                shutil.copytree("sql", dist_dir / "sql", dirs_exist_ok=True)
                print("✅ Copied SQL files")
            print(f"\n🎉 Executable created: {dist_dir / 'ping-river-monitor.exe'}")
            print(f"📁 All files in: {dist_dir.absolute()}")
    else:
        print("❌ Build failed!")
        return False
    return True
 def create_batch_files():
    """Create convenient batch files"""
    batch_files = {
        "start.bat": """@echo off
 echo Starting Ping River Monitor...
 ping-river-monitor.exe
 pause
 """,
        "start-api.bat": """@echo off
 echo Starting Ping River Monitor Web API...
 ping-river-monitor.exe --web-api
 pause
 """,
        "test.bat": """@echo off
 echo Running Ping River Monitor test...
 ping-river-monitor.exe --test
 pause
 """,
        "status.bat": """@echo off
 echo Checking Ping River Monitor status...
 ping-river-monitor.exe --status
 pause
 """,
    }
    dist_dir = Path("dist")
    for filename, content in batch_files.items():
        batch_file = dist_dir / filename
        with open(batch_file, "w") as f:
            f.write(content)
        print(f"✅ Created {filename}")
 def create_readme():
    """Create deployment README"""
    readme_content = """# Ping River Monitor - Standalone Executable
 This is a standalone executable version of the Northern Thailand Ping River Monitor.
 ## Quick Start
 1. **Configure Database**: Edit `.env` file with your PostgreSQL settings
 2. **Test Connection**: Double-click `test.bat`
 3. **Start Monitoring**: Double-click `start.bat`
 4. **Web Interface**: Double-click `start-api.bat`
 ## Files Included
 - `ping-river-monitor.exe` - Main executable
 - `.env` - Configuration file (EDIT THIS!)
 - `start.bat` - Start continuous monitoring
 - `start-api.bat` - Start web API server
 - `test.bat` - Run a test cycle
 - `status.bat` - Check system status
 - `README.md`, `POSTGRESQL_SETUP.md` - Documentation
 - `sql/` - Database initialization scripts
 ## Configuration
 Edit `.env` file:
 ```
 DB_TYPE=postgresql
 POSTGRES_HOST=your-server-ip
 POSTGRES_PORT=5432
 POSTGRES_DB=water_monitoring
 POSTGRES_USER=your-username
 POSTGRES_PASSWORD=your-password
 ```
 ## Usage
 ### Command Line
 ```cmd
 # Continuous monitoring
 ping-river-monitor.exe
 # Single test run
 ping-river-monitor.exe --test
 # Web API server
 ping-river-monitor.exe --web-api
 # Check status
 ping-river-monitor.exe --status
 ```
 ### Batch Files
 - Just double-click the `.bat` files for easy operation
 ## Troubleshooting
 1. **Database Connection Issues**
   - Check `.env` file settings
   - Verify PostgreSQL server is accessible
   - Test with `test.bat`
 2. **Permission Issues**
   - Run as administrator if needed
   - Check firewall settings for API mode
 3. **Log Files**
   - Check `water_monitor.log` for detailed logs
   - Logs are created in the same directory as the executable
 ## Support
 For issues or questions, check the documentation files included.
 """
    with open("dist/DEPLOYMENT_README.txt", "w") as f:
        f.write(readme_content)
    print("✅ Created DEPLOYMENT_README.txt")
 def main():
    """Main build process"""
    print("Building Ping River Monitor Executable")
    print("=" * 50)
    # Check if we're in the right directory
    if not os.path.exists("run.py"):
        print(
            "❌ Error: run.py not found. Please run this from the project root directory."
        )
        return False
    # Install PyInstaller
    install_pyinstaller()
    # Create spec file
    create_spec_file()
    # Build executable
    if not build_executable():
        return False
    # Create convenience files
    create_batch_files()
    create_readme()
    print("\n" + "=" * 50)
    print("🎉 BUILD COMPLETE!")
    print("📁 Check the 'dist' folder for your executable")
    print("💡 Edit the .env file before distributing")
    print("🚀 Ready for deployment!")
    return True
 if __name__ == "__main__":
    success = main()
    sys.exit(0 if success else 1)
@@ -1,112 +0,0 @@
 #!/usr/bin/env python3
 """
 Simple build script for standalone executable
 """
 import os
 import shutil
 import sys
 from pathlib import Path
 def main():
    print("Building Ping River Monitor Executable")
    print("=" * 50)
    # Check if PyInstaller is installed
    try:
        import PyInstaller
        print("[OK] PyInstaller available")
    except ImportError:
        print("[INFO] Installing PyInstaller...")
        os.system("uv add --dev pyinstaller")
    # Clean previous builds
    if os.path.exists("dist"):
        shutil.rmtree("dist")
        print("[CLEAN] Removed old dist directory")
    if os.path.exists("build"):
        shutil.rmtree("build")
        print("[CLEAN] Removed old build directory")
    # Build command with all necessary options
    cmd = [
        "uv",
        "run",
        "pyinstaller",
        "--onefile",
        "--console",
        "--name=ping-river-monitor",
        "--add-data=.env;.",
        "--add-data=sql;sql",
        "--add-data=README.md;.",
        "--add-data=POSTGRESQL_SETUP.md;.",
        "--add-data=SQLITE_MIGRATION.md;.",
        "--hidden-import=psycopg2",
        "--hidden-import=sqlalchemy.dialects.postgresql",
        "--hidden-import=sqlalchemy.dialects.sqlite",
        "--hidden-import=dotenv",
        "--hidden-import=pydantic",
        "--hidden-import=fastapi",
        "--hidden-import=uvicorn",
        "--hidden-import=schedule",
        "--hidden-import=pandas",
        "--clean",
        "--noconfirm",
        "run.py",
    ]
    print("[BUILD] Running PyInstaller...")
    print("[CMD] " + " ".join(cmd))
    result = os.system(" ".join(cmd))
    if result == 0:
        print("[SUCCESS] Executable built successfully!")
        # Copy .env file to dist if it exists
        if os.path.exists(".env") and os.path.exists("dist"):
            shutil.copy2(".env", "dist/.env")
            print("[COPY] .env file copied to dist/")
        # Create batch files for easy usage
        batch_files = {
            "start.bat": """@echo off
 echo Starting Ping River Monitor...
 ping-river-monitor.exe
 pause
 """,
            "start-api.bat": """@echo off
 echo Starting Web API...
 ping-river-monitor.exe --web-api
 pause
 """,
            "test.bat": """@echo off
 echo Running test...
 ping-river-monitor.exe --test
 pause
 """,
        }
        for filename, content in batch_files.items():
            if os.path.exists("dist"):
                with open(f"dist/{filename}", "w") as f:
                    f.write(content)
                print(f"[CREATE] {filename}")
        print("\n" + "=" * 50)
        print("BUILD COMPLETE!")
        print(f"Executable: dist/ping-river-monitor.exe")
        print("Batch files: start.bat, start-api.bat, test.bat")
        print("Don't forget to edit .env file before using!")
        return True
    else:
        print("[ERROR] Build failed!")
        return False
 if __name__ == "__main__":
    success = main()
    sys.exit(0 if success else 1)
@@ -0,0 +1,263 @@
 # Data Sources & External API Catalog
 Catalog of every data source available to the Ping River Monitor — what we ingest
 today, what the ThaiWater/HII ecosystem exposes, and vetted external feeds for
 future model inputs (rainfall, dam releases, forecasts).
 All "verified" claims below were empirically probed on **2026-08-11**. Endpoints
 marked *(catalog)* were recovered from the thaiwater.net frontend JS bundle but
 not exercised (auth required).
 ---
 ## 1. Currently ingested — RID hydrology telemetry
 The only source persisted to the database and used by the ML pipeline.
 | | |
 |---|---|
 | Endpoint | `POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx` |
 | Agency | Royal Irrigation Department (RID) |
 | Auth | None |
 | Cadence | Hourly (`hourlytime` 1.00–24.00; hour 24 = midnight next day) |
 | Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
 | Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
 | Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
 | Client | `src/water_scraper_v3.py` |
 Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
 ---
 ## 2. ThaiWater / HII ecosystem
 ThaiWater (<https://twa.thaiwater.net>) is the National Hydroinformatics
 Institute (HII) portal. It sits on **two distinct API layers** with very
 different access rules.
 ### 2.1 `api-v3.thaiwater.net` — open, no authentication ✅
 Base: `https://api-v3.thaiwater.net/api/v1/thaiwater30/public/`
 Undocumented backend of the portal. No key, no session. No published rate
 limits or terms of use — be a good citizen (hourly polls, cache, filter to
 Ping basin `basin_code == 6`).
 #### `waterlevel_load` — national water-level snapshot (verified)
 ```
 GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/waterlevel_load
 ```
 - ~2.4 MB, 1,426 stations nationwide; **125 in Ping Basin, 61 in Chiang Mai
  province, 34 with discharge**.
 - Per station: `waterlevel_m`, `waterlevel_msl`, `waterlevel_msl_previous`,
  `flow_rate`, `discharge`, `storage_percent`, `situation_level` (1–4 flood
  severity), `diff_wl_bank`, bank/critical levels (`min_bank`,
  `critical_level_msl`, `warning_level_m`, `critical_level_m`, `qmax`),
  `river_name`, basin, geocode, agency, lat/long, `is_key_station`.
 - Ping key stations present: P.1, P.67, P.75, P.81, P.92, P.17, P.87, P.4A,
  P.7A, P.77, P.2A — plus RID mirrors (`ridhydro_P.1`, …), Tak-reach `TP.`/
  `TUP.` codes, and HII sensor clusters (`PIN001-011`, `CHM001-005`).
 - **P.1 = internal station id `3226`** (lat 18.786961, long 99.005089).
 #### `waterlevel_graph` — hourly historical time series (verified) ⭐
 ```
 GET .../waterlevel_graph?station_type=tele_waterlevel&station_id=3226&start_date=2024-09-25&end_date=2024-10-08
 ```
 - Returns `{data: {graph_data: [{datetime, value, value_out, discharge}]}}`,
  hourly. `value` is **water level in m MSL** (not gauge datum).
 - **Respects arbitrary date ranges. Archive verified back to at least
  2019-08** (P.1 returned data for 2019-08-01). 14-day windows tested OK;
  maximum window size not probed.
 - Oct 2024 record flood fully present: peak 305.8 m MSL @ 2024-10-05 12:00,
  discharge 656 m³/s.
 - Datum conversion at P.1: 305.8 MSL peak = 5.30 m gauge ⇒
  **gauge ≈ MSL − 300.5 m** (verify per station before use; each station has
  its own datum offset).
 - Value: independent second historical source for cross-validating / gap-filling
  the RID feed (RID grid is only ~56% filled).
 #### `rain_24h` — national rainfall snapshot (verified)
 ```
 GET https://api-v3.thaiwater.net/api/v1/thaiwater30/public/rain_24h
 ```
 - ~4.5 MB, 4,445 stations; **321 in Chiang Mai province**; agencies include
  HII, DWR, RID.
 - Per station: `rain_1h`, `rain_24h` (mm), `rainfall_datetime`,
  `station.id` (small int — the graph key), `station.tele_station_oldcode`
  (e.g. `CHM005`, `STN0410`, `ridtele_TUP.14`), `sub_basin_id`, lat/long,
  basin, geocode, agency.
 - **This is the missing rainfall input** for the flood model — near-real-time
  hourly gauge rain across the upper Ping catchment.
 #### `rain_24h_graph` — trailing-window rainfall series (verified, limited) ⚠️
 ```
 GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&end_date=...
 ```
 - `station_id` is the **small `station.id`** from `rain_24h` (e.g. 418 =
  CHM005 "Chiang Mai 5", Mae Taeng), *not* the top-level record id.
 - Returns hourly `{rainfall_datetime, rainfall_value}` — **but the date range
  is IGNORED**: every request returns the same trailing ~36-hour window
  (39 rows). Requests for 2020/2024 return identical data to today.
 - Consequence: **no rainfall history via this API**. To build training data,
  persist `rain_24h` from now on and backfill history from satellite QPE or an
  HII data request (§4).
 #### Probed and NOT available on api-v3 (all HTTP 404)
 `dam_daily`, `dam`, `dam_json`, `big_dam`, `mainstream_dam`, `weather`,
 `rain_graph`, `rainfall_graph`, `rain24hr_graph`. Dam data is v2-only (§2.2).
 ### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒
 The layer our existing `src/thaiwater.py` client uses
 (`GET /v2/waterlevel` with `x-api-key: $THAIWATER_API_KEY`; wired to
 `GET /sensors/thaiwater` in the web API, display-only, never persisted).
 Without a key: HTTP 401/500. No public key-registration page was found —
 obtain a sanctioned key from HII (<https://hii.or.th>).
 Full endpoint catalog *(catalog — recovered from frontend JS, not exercised)*:
 - **Water level / discharge**: `/v2/waterlevel`, `/v2/waterlevel/list`,
  `/v2/waterlevel/{id}/detail`, `/v2/waterlevel/canal`,
  `/v2/waterlevel/sea-waterlevel`, `/v2/waterlevel-discharge`,
  `/v2/waterlevel-discharge/list`, `/v2/waterlevel-discharge/{id}/detail`,
  `/v2/waterlevel-discharge/{id}/forecast-table`,
  `/v2/waterlevel-discharge/forecast`, `/v2/waterlevel-discharge/forecast/list`,
  `/v2/watergate`, `/v2/waterload-tide`
 - **Dams** (incl. Bhumibol): `/v2/large-dam/daily-geo-json`,
  `/v2/large-dam/daily/list`, `/v2/large-dam/hourly/list`,
  `/v2/large-dam/daily/{id}/detail`, `/v2/large-dam/hourly/{id}/detail`,
  `/v2/medium-dam/daily-geo-json`, `/v2/medium-dam/daily/list`,
  `/v2/medium-dam/{id}/detail`, `/v2/summary/summary4dam`,
  `/v2/summary/dam-summary`, `/v2/summary/dam-crisis`
 - **Rainfall**: `/v2/rainfall/{type}`, `/v2/rainfall/{type}/list`,
  `/v2/district-rain/actual-measure`, `/v2/district-rain/forecast`,
  `/v2/district-rain/accumulate`, `/v2/summary/rainfall24h-ranking-province`,
  `/v2/summary/rainfall-24hr-forecast`, `/v2/summary/rainfall-forecast`,
  `/v2/summary/warning-rainfall-24h`, `/v2/summary/warning-rainfall-48h`
 - **Weather / hazards**: `/v2/weather`, `/v2/storm`, `/v2/wave`, `/v2/pm25`,
  `/v2/pm10`, `/v2/flood/flash-flood`, `/v2/flood/flash-flood-alert`,
  `/v2/drought/alert`, `/v2/drought/risk-area/list`,
  `/v2/summary/weather-summary`, `/v2/summary/temperature-forecast`,
  `/v2/summary-area/rainfall`
 - **Time-series / graph** (base `/data/platform/v1/public/`):
  `tele_waterlevel/graph`, `flow/graph`, `latest_waterlevel/forecast/graph`,
  `latest_watertide/forecast/graph`, `dam_pdaily_sum_by_date`,
  `dam_pdaily_sum_by_region_graph`, `dam_rulecurve/graph`,
  `medium_dam/graph_year`, `monthly_rainfall/stations`,
  `monthly_rainfall/anomaly-stations`, `tele_watergate/graph`,
  `salinity_forecast_cpy/graph`, `sea_waterlevel_forecast/graph`,
  `latest_weather_area`, `latest_weather_area_daily`
 ### 2.3 Other HII hosts
 | Host | What | Access |
 |---|---|---|
 | `https://standard.thaiwater.net` | **Official water-data standard** — canonical station/basin/province code registries, data-exchange formats, warning-level definitions (Thai) | Open, docs site |
 | `https://api.hii.or.th/tiservice/v1/ws/{token}/isohyet/daily/latest/province/{code}` | Daily isohyet rainfall by province | Token in path |
 | `https://live1.hii.or.th/product/latest/rain/one_map/data/*.tif` | Rainfall anomaly & 1–6-month forecast GeoTIFF rasters | Open |
 | `https://data.hii.or.th` | HII open-data catalog — 36 datasets (rainfall telemetry, water level, weather, climate) | Open browsing |
 | `https://tiwrm.hii.or.th` | Legacy reports | Open |
 Historical bulk telemetry: HII documents a request channel at
 **nhcsoc@hii.or.th**.
 ---
 ## 3. Dams & reservoirs
 > **Geography matters: Bhumibol Dam (Tak) is ~240 km DOWNSTREAM of P.1** and
 > cannot influence Chiang Mai water levels. Do not use it as a P.1 feature.
 The predictive upstream reservoir is **Mae Ngat Somboon Chon** (Mae Ngat
 tributary, joins the Ping above Chiang Mai; spilled 110 m³/s during the
 Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
 | Source | What | Access |
 |---|---|---|
 | `https://app.rid.go.th/reservoir/api/dams` | **INGESTED** — daily snapshot of all ~35 large dams (storage/inflow/outflow MCM, % of usable). `POST` with form field `date=YYYY-MM-DD` (empty = today); GET returns 404 "Unknown method." Archive ≥ 2009; `level_msl` (`DMD_Q`) populated in older years only. Mae Ngat = `DAM_ID 200103` — hit 113% usable capacity, ~19 MCM/day inflow, in Oct 2024. Collected daily by `src/rid_reservoir.py` into `rid_dams` + `rid_reservoir_daily`; backfill via `scripts/backfill_rid_reservoir.py` | Open, no auth |
 | `https://lsim.rid.go.th/ForeCast?reservoirid=22` | Mae Ngat daily status/forecast (RID) | **UNREACHABLE — do not plan around it.** Probed 2026-08-13 from a Thai consumer ISP (AIS Fibre, TH) *and* from abroad: DNS resolves (122.154.18.207) but ICMP is 100% loss and ports 80/443/8080 are filtered, while `app.rid.go.th` answers in 0.27 s over the same connection. Down or RID-internal-only — not a geo-block |
 | `https://app.rid.go.th/reservoir/api/dam` | **Per-dam daily series in ONE request** — `GET` with `dam_id=200103&date_start=YYYY-MM-DD&date_end=YYYY-MM-DD&percent=`. Archive to 2009 (scattered single-day gaps). Far cheaper than the per-day `api/dams` loop the backfill used (one request vs ~2,900); prefer it for gap repair and for adding other dams. Sibling `api/damgraph` takes the same params | Open, no auth |
 | `https://bigdata-api.rid.go.th` (SWOC) | **Intraday reservoir state** — RID SWOC telemetry, hourly with an explicit `hourly_time_utc` stamp; includes Mae Ngat (`TUP.16`) reservoir level m MSL and % capacity. **Snapshot-only — no archive**, so it can only be accumulated forward | Open, no auth |
 | ThaiWater `public/waterlevel_load` stations `ridhydro_TUP.16` (at the dam) / `ridhydro_TUP.11` (dam outlet) | Hourly Mae Ngat reservoir level and outlet stage/flow. **Snapshot-only — `waterlevel_graph` returns empty grids for these ids at every era** (verified 2026-08-13). Collected hourly by our HII collector since 2026-08-11; accumulating forward | Open, no auth |
 | ThaiWater `public/waterlevel_graph` station `P.75` (id 3253) | **The practical dam-release signal**: hourly stage+discharge 3.8 km below the Mae Ngat dam, history to 2019. Already ingested as a core RID station and a model feature since v1 | Open, no auth |
 | ThaiWater `public/waterlevel_graph` station `MOU301` "สะพานน้ำแม่งัด" (id 1475118) | 10-minute stage on the Mae Ngat *above* the reservoir (inflow arm). History only from ~mid-2025; level only, no discharge | Open, no auth |
 | ThaiWater `api-v3 .../analyst/dam` (dam.id 53) | EGAT-sourced copy of Mae Ngat carrying reservoir **level in m MSL historically** — the field RID's own API stopped populating (`DMD_Q`) after ~2013. Daily, no observation time | Open, no auth |
 | `https://tiwrm.hii.or.th/DATA/REPORT/php/rid_bigcm_raw.php?sdate=YYYY-MM-DD` | HII HTML mirror of the RID large-dam daily table. Daily and *intermittent* (2026 YTD publishes ~108 of 225 days) — a cross-check, not a primary source | Open, no auth |
 | `https://water.egat.co.th` | EGAT dams (Bhumibol/Sirikit) hourly+daily inflow/outflow/level | Endpoint catalog not public; contact EGAT (0-2436-8186). Only relevant downstream of Bhumibol |
 | ThaiWater `/v2/large-dam/*`, `dam_rulecurve/graph` | All large/medium dams incl. hourly | Requires HII API key (§2.2) |
 ---
 ## 4. Rainfall & weather (external)
 ### Near-real-time (usable in the live inference path)
 | Source | Cadence / latency | Access | Notes |
 |---|---|---|---|
 | HII `rain_24h` (§2.1) | Hourly, near-real-time | Open JSON | Primary rain-gauge feed; persist from now on |
 | GSMaP NRT (JAXA) | Hourly, ~4 h latency, 0.1° | Free JAXA registration (FTP); or Google Earth Engine `JAXA/GPM_L3/GSMaP/v8/operational` (no registration) | Gauge-corrected `hourlyPrecipRateGC`; catchment-average rain where gauges are sparse |
 | NASA IMERG **Early Run** | Half-hourly, ~4 h latency, 0.1° | Free Earthdata login | NASA-stack alternative to GSMaP |
 ### Forecasts (the only way past the ~17 h physical lead-time cap)
 | Source | What | Access |
 |---|---|---|
 | **Open-Meteo** (<https://open-meteo.com>) — ✅ **INGESTED** since 2026-08-12 (`src/ml/rain.py`): 5 upper-Ping catchment points feed the hgb-v3 model's rain features (trailing sums + forward-24h forecast, archive 2021+); the leader worker also persists hourly rows to the `openmeteo_rain` table | Hourly precip forecast ≤16 days, any lat/lon; **Historical Forecast API archive from 2021** (train on forecast-as-seen, leakage-free); Previous Runs API (fixed 1–7-day leads from Jan 2024); ERA5 back to 1940 | Free, no key, 10k calls/day, non-commercial w/ attribution |
 | TMD NWP API (`https://data.tmd.go.th/nwpapi/v1/forecast/location/...`) | WRF 4.2 daily/hourly forecasts by place, processed ~06:00 daily | Free Bearer-token registration (`/nwpapi/doc/main/`) |
 | GFS / ECMWF IFS open data | 0.25° global, 4×/day | Free (NOMADS / AWS / data.ecmwf.int); Open-Meteo already wraps both |
 ### Training-only (too slow for live)
 | Source | Cadence | Latency |
 |---|---|---|
 | CHIRPS (`data.chc.ucsb.edu/products/CHIRPS-2.0/`) | Daily, 0.05° | ~2 days prelim / 3+ weeks final |
 | IMERG Late / Final | Half-hourly | ~14 h / ~3.5 months |
 | TMD observation API (`data.tmd.go.th/api/index1.php`) | 3-hourly / daily station obs, XML | Free uid+key registration |
 ---
 ## 5. Historical / open-data portals
 | Portal | Content |
 |---|---|
 | `https://data.hii.or.th` | 36 HII datasets (rainfall telemetry the most viewed) |
 | `https://data.go.th/dataset?organization=rid` | 4 RID datasets (API + ZIP) |
 | `https://gdcatalog.go.th` | Nationwide daily rainfall-station catalogs |
 | `https://hydro-1.net` | RID Upper-Northern Hydrology Center — hourly/daily tables, hydrology yearbooks (rating curves) for P-series stations; scrape/download |
 | `https://water.rid.go.th/flood/flood/daily.pdf` | RID daily flood bulletin (PDF only) |
 ---
 ## 6. Integration status & recommended order
 | Source | Status | Action |
 |---|---|---|
 | RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
 | ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
 | HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
 | HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
 | HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
 | Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
 | Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
 | Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |
 | HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
 **Collector configuration** (`src/hii_collector.py`): runs automatically every
 scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry
 mode) from both `--web-api` and continuous-monitoring modes; one-shot via
 `python -m src.main --collect-hii`. Env vars: `ENABLE_HII_COLLECTION`
 (default `true`), `HII_BASIN_CODE` (default `6` = Ping). Requires a SQL
 `DB_TYPE` (sqlite/postgresql/mysql); tables are created automatically.
 **Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
 without notice. Poll hourly at most, cache aggressively, and pursue official
 HII access for anything production-critical.
@@ -1,293 +0,0 @@
 # Enhanced Scheduler Guide
 This guide explains the new 15-minute scheduling system that runs continuously throughout each hour to ensure comprehensive data coverage.
 ## ✅ **New Scheduling Behavior**
 ### **15-Minute Schedule Pattern**
 - **Timing**: Runs every 15 minutes: 1:00, 1:15, 1:30, 1:45, 2:00, 2:15, 2:30, 2:45, etc.
 - **Hourly Full Checks**: At :00 minutes (includes gap filling and data updates)
 - **Quarter-Hour Quick Checks**: At :15, :30, :45 minutes (data fetch only)
 - **Continuous Coverage**: Ensures no data is missed throughout each hour
 ### **Operation Types**
 - **Full Operations** (at :00): Data fetching + gap filling + data updates
 - **Quick Operations** (at :15, :30, :45): Data fetching only for performance
 ## 🔧 **Technical Implementation**
 ### **Scheduler States**
 ```python
 # State tracking variables
 self.last_successful_update = None  # Timestamp of last successful data update
 self.retry_mode = False            # Whether in quick check mode (skip gap filling)
 self.next_hourly_check = None      # Next scheduled hourly check
 ```
 ### **Quarter-Hour Check Process**
 ```python
 def quarter_hour_check(self):
    """15-minute check for new data"""
    current_time = datetime.datetime.now()
    minute = current_time.minute
    # Determine if this is a full hourly check (at :00) or a quarter-hour check
    if minute == 0:
        logging.info("=== HOURLY CHECK (00:00) ===")
        self.retry_mode = False  # Full check with gap filling and updates
    else:
        logging.info(f"=== 15-MINUTE CHECK ({minute:02d}:00) ===")
        self.retry_mode = True   # Skip gap filling and updates on 15-min checks
    new_data_found = self.run_scraping_cycle()
    if new_data_found:
        self.last_successful_update = datetime.datetime.now()
        if minute == 0:
            logging.info("New data found during hourly check")
        else:
            logging.info(f"New data found during 15-minute check at :{minute:02d}")
    else:
        if minute == 0:
            logging.info("No new data found during hourly check")
        else:
            logging.info(f"No new data found during 15-minute check at :{minute:02d}")
 ```
 ### **Scheduler Setup**
 ```python
 def start_scheduler(self):
    """Start enhanced scheduler with 15-minute checks"""
    # Schedule checks every 15 minutes (at :00, :15, :30, :45)
    schedule.every().hour.at(":00").do(self.quarter_hour_check)
    schedule.every().hour.at(":15").do(self.quarter_hour_check)
    schedule.every().hour.at(":30").do(self.quarter_hour_check)
    schedule.every().hour.at(":45").do(self.quarter_hour_check)
    while True:
        schedule.run_pending()
        time.sleep(30)  # Check every 30 seconds
 ```
 ## 📊 **New Data Detection Logic**
 ### **Smart Detection Algorithm**
 ```python
 def has_new_data(self) -> bool:
    """Check if there is new data available since last successful update"""
    # Get most recent timestamp from database
    latest_data = self.get_latest_data(limit=1)
    # Check if we should have newer data by now
    now = datetime.datetime.now()
    expected_latest = now.replace(minute=0, second=0, microsecond=0)
    # If current time is past 5 minutes after the hour, we should have data
    if now.minute >= 5:
        if latest_timestamp < expected_latest:
            return True  # New data expected
    # Check if we have data for the previous hour
    previous_hour = expected_latest - datetime.timedelta(hours=1)
    if latest_timestamp < previous_hour:
        return True  # Missing recent data
    return False  # Data is up to date
 ```
 ### **Actual Data Verification**
 ```python
 # Compare timestamps before and after scraping
 initial_timestamp = get_latest_timestamp_before_scraping()
 # ... perform scraping ...
 latest_timestamp = get_latest_timestamp_after_scraping()
 if initial_timestamp is None or latest_timestamp > initial_timestamp:
    new_data_found = True
    self.last_successful_update = datetime.datetime.now()
 ```
 ## 🚀 **Operational Modes**
 ### **Mode 1: Full Hourly Operation (at :00)**
 - **Schedule**: Every hour at :00 minutes (1:00, 2:00, 3:00, etc.)
 - **Operations**: 
  - ✅ Fetch current data
  - ✅ Fill data gaps (last 7 days)
  - ✅ Update existing data (last 2 days)
 - **Purpose**: Comprehensive data collection and maintenance
 ### **Mode 2: Quick 15-Minute Checks (at :15, :30, :45)**
 - **Schedule**: Every 15 minutes at quarter-hour marks
 - **Operations**:
  - ✅ Fetch current data only
  - ❌ Skip gap filling (performance optimization)
  - ❌ Skip data updates (performance optimization)
 - **Purpose**: Ensure no new data is missed between hourly checks
 ## 📋 **Logging Output Examples**
 ### **Successful Hourly Check (at :00)**
 ```
 2025-07-26 01:00:00,123 - INFO - === HOURLY CHECK (00:00) ===
 2025-07-26 01:00:00,124 - INFO - Starting scraping cycle...
 2025-07-26 01:00:01,456 - INFO - Successfully fetched 384 data points from API
 2025-07-26 01:00:02,789 - INFO - New data found: 2025-07-26 01:00:00
 2025-07-26 01:00:03,012 - INFO - Filled 5 data gaps
 2025-07-26 01:00:04,234 - INFO - Updated 2 existing measurements
 2025-07-26 01:00:04,235 - INFO - New data found during hourly check
 ```
 ### **15-Minute Quick Check (at :15, :30, :45)**
 ```
 2025-07-26 01:15:00,123 - INFO - === 15-MINUTE CHECK (15:00) ===
 2025-07-26 01:15:00,124 - INFO - Starting scraping cycle...
 2025-07-26 01:15:01,456 - INFO - Successfully fetched 299 data points from API
 2025-07-26 01:15:02,789 - INFO - New data found: 2025-07-26 01:00:00
 2025-07-26 01:15:02,790 - INFO - New data found during 15-minute check at :15
 ```
 ### **Continuous 15-Minute Pattern**
 ```
 2025-07-26 01:00:00,123 - INFO - === HOURLY CHECK (00:00) ===
 2025-07-26 01:00:04,235 - INFO - New data found during hourly check
 2025-07-26 01:15:00,123 - INFO - === 15-MINUTE CHECK (15:00) ===
 2025-07-26 01:15:02,790 - INFO - No new data found during 15-minute check at :15
 2025-07-26 01:30:00,123 - INFO - === 15-MINUTE CHECK (30:00) ===
 2025-07-26 01:30:02,790 - INFO - No new data found during 15-minute check at :30
 2025-07-26 01:45:00,123 - INFO - === 15-MINUTE CHECK (45:00) ===
 2025-07-26 01:45:02,790 - INFO - No new data found during 15-minute check at :45
 2025-07-26 02:00:00,123 - INFO - === HOURLY CHECK (00:00) ===
 2025-07-26 02:00:04,235 - INFO - New data found during hourly check
 ```
 ## ⚙️ **Configuration Options**
 ### **Environment Variables**
 ```bash
 # Retry interval (default: 5 minutes)
 export RETRY_INTERVAL_MINUTES=5
 # Data availability buffer (default: 5 minutes after hour)
 export DATA_BUFFER_MINUTES=5
 # Gap filling days (default: 7 days)
 export GAP_FILL_DAYS=7
 # Update check days (default: 2 days)
 export UPDATE_DAYS=2
 ```
 ### **Scheduler Timing**
 ```python
 # Hourly checks at top of hour
 schedule.every().hour.at(":00").do(self.hourly_check)
 # 5-minute retries (dynamically scheduled)
 schedule.every(5).minutes.do(self.retry_check).tag('retry')
 # Check every 30 seconds for responsive retry scheduling
 time.sleep(30)
 ```
 ## 🔍 **Performance Optimizations**
 ### **Retry Mode Optimizations**
 - **Skip Gap Filling**: Avoids expensive historical data fetching during retries
 - **Skip Data Updates**: Avoids comparison operations during retries
 - **Focused API Calls**: Only fetches current day data during retries
 - **Reduced Database Queries**: Minimal database operations during retries
 ### **Resource Management**
 - **API Rate Limiting**: 1-second delays between API calls
 - **Database Connection Pooling**: Efficient connection reuse
 - **Memory Efficiency**: Selective data processing
 - **Error Recovery**: Automatic retry with exponential backoff
 ## 🛠️ **Troubleshooting**
 ### **Common Scenarios**
 #### **Stuck in Retry Mode**
 ```
 # Check if API is returning data
 curl -X POST https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx
 # Check database connectivity
 python water_scraper_v3.py --check-gaps 1
 # Manual data fetch test
 python water_scraper_v3.py --test
 ```
 #### **Missing Hourly Triggers**
 ```
 # Check system time synchronization
 timedatectl status
 # Verify scheduler is running
 ps aux | grep water_scraper
 # Check logs for scheduler activity
 tail -f water_monitor.log | grep "HOURLY CHECK"
 ```
 #### **False New Data Detection**
 ```
 # Check latest data in database
 sqlite3 water_monitoring.db "SELECT MAX(timestamp) FROM water_measurements;"
 # Verify timestamp parsing
 python -c "
 import datetime
 print('Current hour:', datetime.datetime.now().replace(minute=0, second=0, microsecond=0))
 "
 ```
 ## 📈 **Monitoring and Alerts**
 ### **Key Metrics to Monitor**
 - **Hourly Success Rate**: Percentage of hourly checks that find new data
 - **Retry Duration**: How long system stays in retry mode
 - **Data Freshness**: Time since last successful data update
 - **API Response Time**: Performance of data fetching operations
 ### **Alert Conditions**
 - **Extended Retry Mode**: System in retry mode for > 30 minutes
 - **No Data for 2+ Hours**: No new data found for extended period
 - **High Error Rate**: Multiple consecutive API failures
 - **Database Issues**: Connection or save failures
 ### **Health Check Script**
 ```bash
 #!/bin/bash
 # Check if system is stuck in retry mode
 RETRY_COUNT=$(tail -n 100 water_monitor.log | grep -c "RETRY CHECK")
 if [ $RETRY_COUNT -gt 6 ]; then
    echo "WARNING: System may be stuck in retry mode ($RETRY_COUNT retries in last 100 log entries)"
 fi
 # Check data freshness
 LATEST_DATA=$(sqlite3 water_monitoring.db "SELECT MAX(timestamp) FROM water_measurements;")
 echo "Latest data timestamp: $LATEST_DATA"
 ```
 ## 🎯 **Best Practices**
 ### **Production Deployment**
 1. **Monitor Logs**: Watch for retry mode patterns
 2. **Set Alerts**: Configure notifications for extended retry periods
 3. **Regular Maintenance**: Weekly gap filling and data validation
 4. **Backup Strategy**: Regular database backups before major operations
 ### **Performance Tuning**
 1. **Adjust Buffer Time**: Modify data availability buffer based on API patterns
 2. **Optimize Retry Interval**: Balance between responsiveness and API load
 3. **Database Indexing**: Ensure proper indexes for timestamp queries
 4. **Connection Pooling**: Configure appropriate database connection limits
 This enhanced scheduler ensures reliable, efficient, and intelligent water level monitoring with automatic adaptation to data availability patterns.
@@ -1,227 +0,0 @@
 # 🚀 Northern Thailand Ping River Monitor - Enhancement Summary
 ## 🎯 **What We've Accomplished**
 We've successfully transformed your water monitoring system from a simple scraper into a **production-ready, enterprise-grade monitoring platform** focused on the Ping River Basin in Northern Thailand, with modern web interfaces, station management capabilities, and comprehensive observability.
 ## 🌟 **Major New Features Added**
 ### 1. **FastAPI Web Interface** 🌐
 - **Interactive Dashboard** at `http://localhost:8000`
 - **REST API** with comprehensive endpoints
 - **Station Management** - Add, update, delete monitoring stations
 - **Real-time Health Monitoring** 
 - **Manual Data Collection Triggers**
 - **Interactive API Documentation** at `/docs`
 - **CORS Support** for web applications
 ### 2. **Enhanced Architecture** 🏗️
 - **Type Safety** with Pydantic models and comprehensive type hints
 - **Data Validation Layer** with range checking and error handling
 - **Custom Exception Classes** for better error management
 - **Modular Design** with separated concerns
 ### 3. **Observability & Monitoring** 📊
 - **Metrics Collection System** (counters, gauges, histograms)
 - **Health Checks** for database, API, and system resources
 - **Performance Tracking** with response times and success rates
 - **Enhanced Logging** with colors, rotation, and performance logs
 ### 4. **Production Features** 🚀
 - **Rate Limiting** to prevent API abuse
 - **Request Tracking** with detailed statistics
 - **Configuration Validation** on startup
 - **Graceful Error Handling** and recovery
 - **Background Task Management**
 ## 📁 **New Files Created**
 ```
 src/
 ├── models.py              # Data models and type definitions
 ├── exceptions.py          # Custom exception classes
 ├── validators.py          # Data validation layer
 ├── metrics.py            # Metrics collection system
 ├── health_check.py       # Health monitoring system
 ├── rate_limiter.py       # Rate limiting and request tracking
 ├── logging_config.py     # Enhanced logging configuration
 ├── web_api.py           # FastAPI web interface
 ├── main.py              # Enhanced CLI with multiple modes
 └── __init__.py          # Package initialization
 # Root files
 ├── run.py                # Simple startup script
 ├── test_integration.py   # Integration test suite
 ├── test_api.py          # API endpoint tests
 └── ENHANCEMENT_SUMMARY.md # This file
 ```
 ## 🔧 **Enhanced Existing Files**
 - **`src/water_scraper_v3.py`** - Integrated new features, metrics, validation
 - **`src/config.py`** - Added configuration validation
 - **`requirements.txt`** - Added FastAPI, Pydantic, and monitoring dependencies
 - **`docker-compose.victoriametrics.yml`** - Added web API service
 - **`Dockerfile`** - Updated for new startup script
 - **`README.md`** - Updated with new features and usage instructions
 ## 🌐 **Web API Endpoints**
 | Endpoint | Method | Description |
 |----------|--------|-------------|
 | `/` | GET | Interactive dashboard |
 | `/docs` | GET | API documentation |
 | `/health` | GET | System health status |
 | `/metrics` | GET | Application metrics |
 | `/stations` | GET | List all monitoring stations |
 | `/measurements/latest` | GET | Latest measurements |
 | `/measurements/station/{code}` | GET | Station-specific data |
 | `/scrape/trigger` | POST | Trigger manual data collection |
 | `/scraping/status` | GET | Scraping status and statistics |
 | `/config` | GET | Current configuration (masked) |
 ## 🚀 **Usage Examples**
 ### **Traditional Mode (Enhanced)**
 ```bash
 # Test single cycle
 python run.py --test
 # Continuous monitoring
 python run.py
 # Fill data gaps
 python run.py --fill-gaps 7
 # Show system status
 python run.py --status
 ```
 ### **Web API Mode (NEW!)**
 ```bash
 # Start web API server
 python run.py --web-api
 # Access dashboard
 open http://localhost:8000
 # View API documentation
 open http://localhost:8000/docs
 ```
 ### **Docker Deployment**
 ```bash
 # Start complete stack
 docker-compose -f docker-compose.victoriametrics.yml up -d
 # Services available:
 # - Water API: http://localhost:8000
 # - Grafana: http://localhost:3000
 # - VictoriaMetrics: http://localhost:8428
 ```
 ## 📊 **Monitoring & Observability**
 ### **Built-in Metrics**
 - API request counts and response times
 - Database connection status and save operations
 - Scraping cycle success/failure rates
 - System resource usage (memory, etc.)
 ### **Health Checks**
 - Database connectivity and data freshness
 - External API availability
 - Memory usage monitoring
 - Overall system health status
 ### **Enhanced Logging**
 - Colored console output for better readability
 - File rotation to prevent disk space issues
 - Performance logging for optimization
 - Structured logging with proper levels
 ## 🔒 **Production Ready Features**
 ### **Security & Reliability**
 - Rate limiting to prevent API abuse
 - Input validation and sanitization
 - Graceful error handling and recovery
 - Configuration validation on startup
 ### **Performance**
 - Efficient metrics collection with minimal overhead
 - Background task management
 - Connection pooling and resource management
 - Optimized database operations
 ### **Scalability**
 - Modular architecture for easy extension
 - Async support for high concurrency
 - Configurable resource limits
 - Health checks for load balancer integration
 ## 🧪 **Testing**
 ### **Integration Tests**
 ```bash
 # Run all integration tests
 python test_integration.py
 ```
 ### **API Tests**
 ```bash
 # Test API endpoints (server must be running)
 python test_api.py
 ```
 ## 📈 **Performance Improvements**
 1. **Request Tracking** - Monitor API performance and success rates
 2. **Rate Limiting** - Prevent API abuse and ensure stability
 3. **Data Validation** - Catch errors early and improve data quality
 4. **Metrics Collection** - Identify bottlenecks and optimization opportunities
 5. **Health Monitoring** - Proactive issue detection and alerting
 ## 🎉 **Benefits Achieved**
 ### **For Developers**
 - **Better Developer Experience** with type hints and validation
 - **Easier Debugging** with enhanced logging and error messages
 - **Comprehensive Testing** with integration and API tests
 - **Modern Architecture** following best practices
 ### **For Operations**
 - **Web Dashboard** for easy monitoring and management
 - **Health Checks** for automated monitoring integration
 - **Metrics Collection** for performance analysis
 - **Production-Ready** deployment with Docker support
 ### **For Users**
 - **REST API** for integration with other systems
 - **Real-time Data Access** via web interface
 - **Manual Controls** for triggering data collection
 - **Status Monitoring** for system visibility
 ## 🔮 **Future Enhancement Opportunities**
 1. **Authentication & Authorization** - Add user management and API keys
 2. **Real-time WebSocket Updates** - Live data streaming to web clients
 3. **Advanced Analytics** - Trend analysis and forecasting
 4. **Alert System** - Email/SMS notifications for critical conditions
 5. **Multi-tenant Support** - Support for multiple organizations
 6. **Data Export** - CSV, Excel, and other format exports
 7. **Mobile App** - React Native or Flutter mobile interface
 ## 🏆 **Summary**
 Your Thailand Water Monitor has been transformed from a simple data scraper into a **comprehensive, enterprise-grade monitoring platform** that includes:
 - ✅ **Modern Web Interface** with FastAPI
 - ✅ **Production-Ready Architecture** with proper error handling
 - ✅ **Comprehensive Monitoring** with metrics and health checks
 - ✅ **Type Safety** and data validation
 - ✅ **Enhanced Logging** and observability
 - ✅ **Docker Support** for easy deployment
 - ✅ **Extensive Testing** for reliability
 The system is now ready for production deployment and can serve as a foundation for further enhancements and integrations!
@@ -68,7 +68,7 @@ environment variable, then `Config.get_database_config()` when `DB_TYPE` is
 1. **PostgreSQL** (`_fetch_from_db`) — the primary path. NULL discharge stays
   NULL, which matters because the models must learn from the real missingness
   pattern.
-2. **HTTP API** (`_fetch_from_api`, default `http://100.81.167.42:8000`) — a
+2. **HTTP API** (`_fetch_from_api`, default `https://water.buildfor.life`) — a
   fallback for running off-server. **Caveat:** the public history endpoint
   backfills missing discharge with a synthetic rating-curve estimate, so this
   path is not equivalent to the DB path. It is flagged as
@@ -184,7 +184,7 @@ One `HistGradientBoosting` model per **station × horizon × head**:
 | Head | Type | Target |
 |---|---|---|
-| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | max observed level in (t, t+h] |
+| `max_{h}` | `HistGradientBoostingRegressor` (squared error) | *rise*: max observed level in (t, t+h] minus level at t (v2; serving adds the level back) |
 | `warn_{h}` | `HistGradientBoostingClassifier` | level ≥ 3.0 m anywhere in (t, t+h] |
 | `danger_{h}` | `HistGradientBoostingClassifier` | level ≥ 4.5 m anywhere in (t, t+h] |
@@ -203,16 +203,17 @@ section 6).
 The system degrades in tiers rather than failing:
-1. **Classifier head**, when the training span contains at least
+1. **Belt-and-braces probability** *(since 2026-08-11 — see the re-examination
-   `MIN_POSITIVES_FOR_CLASSIFIER = 30` positive examples. Below that, a
+   note in section 7)*: the sigmoid-of-regression probability
-   classifier would be fitting noise, and the head is recorded in
+   `p = 1/(1 + exp(−(predicted_max − threshold)/σ))` is always computed (σ =
-   `skipped_heads` with its reason.
+   the regressor's test-residual std, floor `MIN_SIGMA = 0.15` m), and when a
-2. **Sigmoid on the regression head**, when the classifier is absent.
+   classifier head exists — trained only if the span had at least
-   `p = 1/(1 + exp(−(predicted_max − threshold)/σ))`, where σ is the standard
+   `MIN_POSITIVES_FOR_CLASSIFIER = 30` positives — the served probability is
-   deviation of the regressor's test residuals (floor `MIN_SIGMA = 0.15` m). This
+   `max(classifier, sigmoid)`. The classifier can raise the alarm but never
-   turns the peak-level prediction into a calibrated-ish probability that widens
+   silence it: on the gap-filled data a trained classifier stayed near zero
-   correctly when the regressor is less accurate at that horizon — at P.1, σ is
+   through the 2024 record crossing while the regression tracked it.
-   0.15 m at 6 and 12 h but 0.166 m at 24 h.
+2. **Sigmoid only**, when the classifier head is absent or skipped
   (recorded in `skipped_heads` with its reason).
 3. **Persistence heuristic** (`predict._heuristic_forecast`), when there is no
   model file at all, or the station's newest reading is more than
   `STALE_AFTER_H = 6` hours old. It extrapolates the last 3 h rate of rise
@@ -255,34 +256,129 @@ invalidates the cache without a restart.
 ### Holdout metrics (`models/metrics.json`)
-Model version `hgb-v1+49a3de0`, generated 2026-08-10. Train ≤ 2024-12-31, test
+Two evaluations exist and they differ sharply — the re-examination note in the
-2025-01-01 → 2026-08-10 — the test span is entirely unseen future data relative
+next section explains why (the hourly grid was gap-filled from ~56% to ~93%
-to training.
+between them, roughly doubling the test rows and collapsing the warning base
 rates).
-P.1 (Nawarat Bridge), the station that matters most:
+**Current model** `hgb-v3` (rise target + Open-Meteo rain features),
 generated 2026-08-12 on the gap-filled DB (~976k rows). Train ≤ 2024-12-31,
 test 2025-01-01 → 2026-08-12. P.1:
-| Horizon | Warning PR-AUC | Recall @1% FAR | Recall @5% FAR | MAE | MAE above 2 m | Test rows | Base rate |
+| Horizon | Warning PR-AUC | MAE | MAE above 2 m | Test rows | Base rate |
-|---|---|---|---|---|---|---|---|
+|---|---|---|---|---|---|
-| 6 h | 0.974 | 98.3% | 100% | 6.1 cm | 9.2 cm | 8,536 | 1.36% |
+| 6 h | 0.783 | 4.9 cm | 5.0 cm | 14,034 | 0.12% |
-| 12 h | 0.904 | 93.8% | 97.7% | 9.0 cm | 15.0 cm | 7,932 | 1.61% |
+| 12 h | 0.508 | 7.2 cm | 10.7 cm | 14,028 | 0.16% |
-| 24 h | 0.900 | 90.1% | 93.4% | 11.3 cm | 24.5 cm | 8,572 | 1.77% |
+| 24 h | 0.288 | 8.7 cm | 18.1 cm | 14,034 | 0.25% |
-Read PR-AUC against the base rate — 0.974 versus a 1.36% positive rate is a wide
+(Progression across the same day's runs — v1 absolute target:
-margin over chance. "Recall at 1% false-alarm rate" is the operationally honest
+5.5/8.1/10.5 cm MAE; v2 rise: 5.0/7.2/9.4; v3 rise+rain: 4.9/7.2/8.7 —
-number: at a threshold that fires on 1% of quiet hours, the 6 h model still
+with above-2 m MAE falling 24.0 → 20.0 → 18.1 cm at 24 h. PR-AUC belongs to
-catches 98.3% of warning exceedances.
+the unchanged classifier heads; serving is belt-and-braces so alerting uses
 the improved regression path regardless.)
-P.103 (Ring Bridge 3) is the only station with enough danger-level events to
+Level accuracy improved; standalone classifier discrimination did not survive
-evaluate a danger head on the 2025–26 span (base rate 5.7–7.4%): PR-AUC 0.979 /
+the data change (which is why serving is now `max(classifier, sigmoid)` — see
-0.953 / 0.892 and recall at 1% FAR of 97.9% / 89.9% / 79.5% at 6 / 12 / 24 h.
+"Head gating"). Recall-at-FAR is null at all horizons on this run. Across
 stations the 6 h warning PR-AUC now spans 0.987 (P.77) / 0.982 (P.5) / 0.956
 (P.85) / 0.950 (P.67) down to 0.436 (P.84), and danger heads are now evaluable
 at nine stations — strongest P.5 (0.958/0.883/0.811 at 6/12/24 h) and P.77
 (0.942/0.863/0.786); P.103's danger metrics, previously the highlight, are null
 on this span.
-Across the other stations the 6 h warning PR-AUC spans 0.996 (P.5) down to 0.302
+**Historical evaluation** (`hgb-v1+49a3de0`, 2026-08-10, pre-gap-fill DB —
-(P.82), and tracks almost exactly with how many exceedances that station saw. The
+kept for the record; these numbers described the sparser 56%-filled grid and do
-strong ones are the frequently-flooded gauges — P.5 0.996, P.81 0.992, P.77 0.968,
+not reproduce on today's data):
-P.85 0.953, P.75 0.927 — and the weak ones are un-routed western tributaries with
+
-almost no positives (P.84 0.570, P.82 0.302 on 0.22% of test hours). P.92 and P.20
+| Horizon | Warning PR-AUC | Recall @1% FAR | MAE | Test rows | Base rate |
-have no evaluable warning metric at all: neither crossed 3.0 m often enough in the
+|---|---|---|---|---|---|
-test span (P.92 not once, P.20 in 0.09% of hours) to score.
+| 6 h | 0.974 | 98.3% | 6.1 cm | 8,536 | 1.36% |
 | 12 h | 0.904 | 93.8% | 9.0 cm | 7,932 | 1.61% |
 | 24 h | 0.900 | 90.1% | 11.3 cm | 8,572 | 1.77% |
 The dramatic PR-AUC difference is mostly the base rate: the filled grid adds
 ~5,500 quiet test hours per horizon while the number of positive hours barely
 changes, so the same ranking quality scores far lower — and the classifier's
 genuine out-of-distribution weakness (see the backtest sections) does the rest.
 ### 2026-08-11 re-examination: fuller data changes the backtest story
 > **Read this before the two backtest sections below.** On 2026-08-11 the
 > backtests were codified into `scripts/backtest_render.py` (previously they
 > were one-off runs) and re-run after the database grew from 592k to ~976k
 > rows (a `--fill-gaps all` pass repaired most of the missing 44% of the
 > hourly grid). Three things changed:
 >
 > 1. **The 2024 crossing was 8 hours earlier than documented.** The recovered
 >    hours show P.1 crossing 3.70 m at **17:00 on 24 September 2024**, not
 >    01:00 on 25 September — confirmed independently by the HII sensor at
 >    Nawarat Bridge (hii_waterlevel, station 3226: 3.73 m at 17:00). The
 >    originally celebrated "24-hour warning" was therefore ~16 hours measured
 >    against the real river.
 > 2. **Retraining on the fuller data improves level accuracy but degrades the
 >    warning classifiers.** P.1 24 h MAE improved (11.3 → 10.5 cm), but the
 >    warning-head PR-AUC collapsed (0.900 → 0.288 at 24 h): with the filled
 >    grid the classifier trains on many more dry-season rows and now stays
 >    silent through the September 2024 record crossing while the regression
 >    head tracks it. Serving was changed to belt-and-braces —
 >    `max(classifier, sigmoid(regression))` — so alerting can never be worse
 >    than the regression path.
 > 3. **Honest current lead times, from the regenerated charts below:** the
 >    retrained configuration first alerts ~18 h *after* the true 24 Sep 2024
 >    crossing and roughly *at* the 27 Sep 2025 crossing. The earlier, better
 >    numbers came from models trained and evaluated on the sparser data. The
 >    conclusion is not that the old system was better — it is that gauge-only
 >    features fundamentally lack lead time for fast rises, which is exactly
 >    the rainfall-input and rise-target work now queued (see "Honest limits").
 >
 > `scripts/backtest_render.py` regenerates all three charts and fails its
 > acceptance gate while the 2024 lead stays under 12 h — keeping this page
 > honest is now automatic.
 >
 > **2026-08-12 follow-up — hgb-v2 (rise target).** A rolling-origin,
 > event-aware evaluation (`scripts/evaluate_variants.py`, one fold per monsoon
 > 2021-2025) compared the absolute-level target against rise-target variants.
 > The rise target — regression predicts *future max minus current level*, the
 > level is added back at serving — won decisively and is now deployed as
 > `hgb-v2`: the regenerated charts below show the 2024 first alert moving from
 > 18 h late to **6 h early** (11:00 vs the 17:00 crossing), the 2025 alert
 > from at-crossing to **45 h early**, the record-peak underprediction
 > eliminated (the model now slightly overshoots 5.30 m rather than capping
 > ~0.4 m below it), and P.1 MAE improving ~11% at every horizon. Weighted and
 > quantile variants were evaluated and rejected (more false alarms, no
 > calibration gain by Brier score). The ≥12 h acceptance gate still fails at
 > +6 h for 2024 — genuine further lead needs rainfall inputs, not modelling.
 >
 > **2026-08-12 follow-up 2 — hgb-v3 (rain features): the gate passes.**
 > Open-Meteo catchment rainfall (five upper-Ping points, forecast-model
 > archive 2021+, `src/ml/rain.py`) added four features: trailing 6/24/72 h
 > rain sums and `rain_fc24`, the forward-24 h forecast sum — the first input
 > that can act before water reaches any gauge. On the rolling-origin harness
 > (`models/eval_rain.json`) rain roughly halved flood-year Brier scores, cut
 > flood-regime MAE 20–40%, and moved the hard 2024 leads from +6 h to +11 h
 > (P.1) and +10 to +19 h (P.103); the marginal 2025 double-crest event trades
 > its artifact +46 h "lead" for a calibrated +2 h with zero false alarms. The
 > regenerated backtest below now shows a **13-hour warning for the 2024
 > record flood (alert 04:00, crossing 17:00) — the ≥12 h acceptance gate
 > passes for the first time**. P.1 MAE improves again to 4.9/7.2/8.7 cm at
 > 6/12/24 h. Serving fetches live rain hourly and degrades to NaN features
 > (never a crash) if Open-Meteo is unreachable.
 ### The September 2025 flood, as the deployed configuration saw it
 ![Observed vs predicted through the September 2025 flood — model trained only
 through 2024](img/backtest-2025-p1.png)
 This uses the deployed configuration (train ≤ 2024-12-31) on an event it never
 saw. *(Chart regenerated 2026-08-12 with the hgb-v2 rise target on the
 gap-filled data — see the re-examination note above for the history of these
 numbers.)* The v3 model first alerts at **16:00 on 27 September 2025 — 2 hours before
 the river crosses 3.70 m** at 18:00. This is a shorter lead than v2's 45 h,
 and deliberately so: v2's long "lead" was an alarm that latched through the
 near-miss 3.51 m crest of the 26th; v3's rain-informed probabilities are far
 better calibrated on this marginal event (Brier halved, zero false-alarm
 episodes on the season) and fire when exceedance actually becomes likely. A
 barely-over-threshold crest is intrinsically a short-notice event.
 ### Headline validation: the October 2024 record flood
@@ -291,6 +387,38 @@ training half. So the model was retrained on data **ending 2024-08-31** and aske
 to forecast September–November 2024 cold, with no knowledge of the event that
 followed. This is the closest thing to a real operational test available.
 ![Observed P.1 level vs the model's 24 h-ahead predicted peak through the
 October 2024 flood, with the warning probability below](img/backtest-2024-p1.png)
 The render above shows the whole event hour by hour *(regenerated 2026-08-12
 with the hgb-v3 rise + rain configuration)*. Top: the observed level (blue)
 against the 24 h-ahead predicted peak the model issued at each hour (amber,
 dashed) — the amber line leads the blue one into both flood waves. Bottom: the
 belt-and-braces probability of flooding within 24 h; the **first alert comes
 at 04:00 on 24 September, 13 hours before the true 17:00 crossing**, while
 the river in town still read 2.9 m — the rain features react to upstream
 precipitation before any gauge rises. The rise target removed the
 cannot-exceed-training-max ceiling, so the record 5.30 m peak is tracked
 rather than capped. The same historic model track drives the dashboard's
 "Replay Oct 2024 flood" feature.
 ![Hour-by-hour detail of the detection window, 22–28 September
 2024](img/backtest-2024-p1-detail.png)
 The hour-by-hour detail of the detection window *(regenerated 2026-08-12,
 hgb-v3)* shows the sequence: the river crosses 3.70 m at **17:00 on
 24 September** (the hours recovered by gap-filling; independently confirmed by
 the HII sensor at the same bridge), and the model's probability crosses 0.5 at
 **04:00 — a 13-hour warning** delivered while the river stood at 2.9 m. The
 same alert under the absolute-level target came 18 hours *after* the crossing
 (v1), and 6 hours before it with the rise target alone (v2); catchment
 rainfall closed the rest. This clears the ≥12 h acceptance gate in
 `scripts/backtest_render.py`.
 The event bullets below quote the original (pre-gap-fill) evaluation of the
 deployed model and are kept for the historical record — see the re-examination
 note above for why the lead times no longer reproduce:
 - **25 September cold start.** P.1's first warning crossing of the episode was
  alerted **24–26 hours ahead**. This is the genuinely impressive case: the river
  was in normal state, and the alert came from upstream routing alone.
@@ -315,11 +443,17 @@ followed. This is the closest thing to a real operational test available.
 time into P.1 is 17 h (P.20), and the strongest predictors are much closer:
 P.103 at 1 h, P.67 at 7 h, P.21 at 9 h. Once a 24 h forecast reaches past roughly
 17 h, there is no observation that has "already happened" to inform it — the model
-is extrapolating basin state and season, not routing a wave. The 2025–26 test
+is extrapolating basin state and season — unless it has rainfall. That is no
-events bear this out: the 25 September 2025 cold-start crossing was called 7 h
+longer hypothetical: hgb-v3's Open-Meteo features (see the re-examination
-ahead by the 12 h model and 9 h ahead by the 24 h model. **Practical lead for P.1
+notes in section 7) took the 2024 record-flood lead from 18 h late (v1
-is ~7–17 h.** Extending it requires rainfall forecasts and Mae Ngat/Mae Kuang dam
+gauge-only, absolute target) to 13 h early, precisely because catchment rain
-release data, neither of which this system currently ingests.
+acts before any gauge rises, and `rain_fc24` — a weather *forecast* — acts
 before the rain itself falls. **Remaining honest limits:** marginal
 just-over-threshold crests (2025: +2 h) are intrinsically short-notice; the
 rain series only exists from 2021-03, so older training rows are rain-blind;
 forecast-rain quality bounds what the feature can add; and Mae Ngat reservoir
 state, though now ingested daily (see `docs/DATA_SOURCES.md`), measurably
 *hurts* alert lead as a model feature — see the 2026-08-13 experiment below.
 **Danger-level skill at P.1 is unproven.** P.1 never crossed 4.5 m in the
 2025-01-01 → 2026-08-10 test span (`base_rate_danger` is 0.0, so every danger
@@ -342,6 +476,114 @@ P.1 additionally reports `stages`: exceedance probability for each of the seven
 official inundation stages (3.70–4.60 m), computed from the regression head and
 its calibration sigma, so they need no retrain and no per-stage classifiers.
 ### 2026-08-13: Mae Ngat dam features — a documented negative result
 With `rid_reservoir_daily` backfilled to 2018 (daily Mae Ngat storage/inflow/
 outflow, `src/ml/dam.py`), the obvious v4 experiment was to feed reservoir
 state to the mainstem models: during the Oct 2024 flood the dam hit 113% of
 usable capacity with 19–22 MCM/day inflow spikes on the crossing days.
 **It fails the acceptance gate.** On the 2024 record-flood backtest (train
 < 1 Sep 2024, belt-and-braces alerting, identical to the deployed pipeline):
 | dam features               | first-alert lead | record-peak err (24 h ahead) |
 |----------------------------|------------------|------------------------------|
 | none (deployed v3 config)  | **+13 h** (PASS) | +0.24 m                      |
 | all four                   | +10 h (FAIL)     | +0.22 m                      |
 | storage % + 3-day delta    | +12 h            | +0.21…+0.27 m                |
 | inflow + outflow           | +10 h (FAIL)     | +0.35 m                      |
 | outflow only               | +12 h            | +0.20 m                      |
 Every subset costs 1–3 h of warning for at most a ~3 cm peak-error gain. The
 mechanism is the publication lag: RID posts the daily report on the morning of
 its own date (features apply it from 07:00, `dam.py`'s leakage rule), so at the
 04:00 first-alert hour of 24 Sep 2024 the freshest dam row still described
 23 Sep — a benign reservoir quietly absorbing inflow (outflow 0.13 MCM/day).
 The columns therefore argue *against* imminent flooding exactly when the rain
 features are (correctly) raising the alarm. The rolling-origin harness agrees:
 `rise_rain_dam` matches `rise_rain` on leads and false alarms, only nudging
 event-peak amplitude (−0.11 → −0.03 m on the Sep 2024 event), and `rise_dam`
 (dam without rain) is strictly worse with alarm-latch artifacts.
 **Disposition:** dam features are OFF by default (`train_all(use_dam=False)`;
 opt-in via `--dam` on the training CLI, `scripts/backtest_render.py --dam`,
 and the `rise_rain_dam` / `rise_dam` harness variants). The collector keeps
 accruing daily rows; revisit post-monsoon when the 2026 season adds dam-era
 flood events.
 **Why the lag is probably not the whole story — P.75 already *is* the dam
 signal.** A 2026-08-13 source sweep put the negative result on firmer
 ground: **P.75 "บ้านช่อแล" sits 3.8 km downstream of the Mae Ngat dam** on
 the Mae Ngat river (nearest other station: P.4A at 10.9 km), it reports
 hourly, and it has been a model input since v1 with a 12 h routed lead into
 P.1. Whatever the reservoir releases flows past P.75 within the hour and the
 model already reads it. The daily reservoir table therefore offers a stale,
 coarser proxy of a signal the features capture hourly and directly — which
 is the more likely reason it adds nothing and costs alarm responsiveness.
 That reframes what a future intraday source would have to beat: not "no dam
 information", but "hourly observed dam *outflow*". Genuine intraday
 reservoir-state feeds do exist and are open (`bigdata-api.rid.go.th` SWOC
 telemetry, and ThaiWater station `ridhydro_TUP.16` *at the dam*), but both
 are **snapshot-only — no archive** (verified: the history endpoint returns
 empty grids for them at every era, including the current one). They can only
 be accumulated forward, so they cannot retrain against 2024/2025 events.
 The HII collector already captures both hourly as of 2026-08-11; revisit
 after the 2026 monsoon, when a season of true intraday reservoir state
 exists alongside its flood events.
 **Shipped from the same work:** the HII gap-fill merge in the data loader
 (`fill_from_hii`, +9,341 h at P.81, +682 h at P.92, +810 h at P.20) is
 lead-neutral — the gate holds at 13 h with fill on — and ships enabled.
 ### 2026-09-12: three candidates on top of hgb-v3 — two rejected, one deferred
 Same rolling-origin harness (`src/ml/evaluate.py`, five monsoon folds
 2021–2025, P.1 and P.103), all variants run from the identical
 `models/cache/` snapshot (`--from-cache`), results in
 `models/eval_2026-09-12*.json`, tables via `scripts/summarize_eval.py`.
 Baseline is `rise_rain`, the deployed configuration.
 **Quantile regression heads (`rise_rain_quantile`, `_uw`) — rejected.** The
 August result that quantile loss beat L2 on MAE held with rain in the model
 (P.1 0.083/0.081 vs 0.087; P.103 0.143/0.124 vs 0.152), and Brier improved a
 hair, but the operational numbers went the wrong way: at P.103 the 2022-08-14
 crossing dropped from +6 h to +1 h lead, 2022-10-02 from +9 h to +5/+3 h, and
 the 2024-09-30 event from +9 h to +4 h; at P.1 2022 dropped +5 → +3/+2 h and
 2025 +2 → +1 h, with one false-alarm episode where the baseline had none. A
 median predicts the *typical* rise, and on the run-up to a crossing the typical
 rise is not the one that matters. MAE is not the objective; lead is.
 **Quantile heads for sigma only (`rise_rain_qsigma`) — no effect.** The
 hybrid keeps the L2 point prediction (so every lead is identical to the
 baseline by construction — p≥0.5 alerts are sigma-independent) and derives a
 per-row sigma from q90−q50. Brier moved 0.0031 → 0.0029 at P.1 and
 0.0061 → 0.0060 at P.103, i.e. within noise, at the cost of three fitted
 heads per horizon instead of one. Per-row uncertainty from this family of
 models is not informative enough here to be worth the training time; the
 0.15 m floor stays.
 **Forward-48 h forecast rain (`rise_rain_fc48`) — deferred.** Adding the
 `(t, t+48]` Open-Meteo sum alongside `rain_fc24` left MAE, Brier and false
 alarms unchanged and every event lead within ±1 h of baseline, *except* the
 2024-10-03 P.1 record crossing, which went from +21 h to +72 h (and +55 → +69 h
 at P.103). That is one event with the highest stakes in the record, on the
 same feature family that already produced the 2024 gain, but n=1 is not
 evidence: the P.103 2025-09-26 event lost 2 h in the same run. Rerun after the
 2026 season adds events; if the 48 h window still moves only the biggest
 onsets, promote it. Serving would need no new data source (`fetch_forecast`
 already pulls `forecast_days=2`).
 **HII gauge rain — not evaluable yet.** `hii_rainfall` (~130 gauges in the
 upper-Ping box, DWR/FOP/HII/RID/TMD) is the obvious independent rain source,
 but the table only exists since 2026-08-11 and the api-v3 archive endpoint
 ignores its date range (see `docs/DATA_SOURCES.md` §2.1), so every training
 row before that is NaN and no fold in the harness has gauge data in its test
 span. `src/ml/hii_rain.py` builds the catchment mean and
 `GET /api/hii/rainfall/catchment` exposes it next to the Open-Meteo series with
 a 24 h-sum bias/MAE/correlation, so the two sources' relationship is on record
 by the time the 2027 fold (train ≤ 2027-04-30, test Jun–Nov 2027) can test it.
 ## 6. Deployment
 ### API
@@ -539,6 +781,32 @@ timestamp in every bundle. Both are echoed in every `/forecast` row, so you can
 tell from the API response alone which code produced a forecast and how old the
 model is.
 **Scheduled retrain (since 2026-09-12).** `scripts/water-monitor-retrain.timer`
 fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
 time (`Persistent=true`, so a missed run catches up at boot). The unit runs
 `scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
 1. trains all stations into `models/.staging/` (the API keeps serving the old
   bundles throughout);
 2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
   least 14 trained stations (exit 3, staging discarded, old models untouched);
 3. renames the new bundles into `models/`, moving the previous generation to
   `models/.previous/` for rollback.
 No API restart: `predict.py` reloads bundles by mtime on the next hourly
 precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
 run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
 flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
 (`scripts/install.sh`) enables the timer.
 **Why the trainer refuses to run without rain (since 2026-09-12).** On
 2026-09-01 the server retrain could not reach the Open-Meteo archive on a
 checkout with no `models/cache/`, logged a warning, and quietly overwrote the
 v3 bundles with gauge-only v2 ones — the 13-hour early warning on the 2024 flood
 became an 18-hour late one and nothing on the dashboard said so. `train_all()`
 now raises `RainUnavailableError` (CLI exit 2) in that situation. Gauge-only
 bundles are still available, but only by asking for them: `--no-rain`.
 ## 8. Operations runbook
 All commands assume the project virtualenv is active (`.venv` locally).
@@ -579,10 +847,12 @@ print({h: (d.get('pr_auc_warn'), d.get('mae')) for h, d in m['stations']['P.1'][
 ```
 Expect fifteen `trained` and one `heuristic` (P.4A). A station that reports
-`failed` names its reason in the same payload. If P.1's 6 h warning PR-AUC has
+`failed` names its reason in the same payload. Compare against the *previous
-dropped materially below ~0.97 or its MAE has risen well above ~6 cm, investigate
+run's* `metrics.json`, not an absolute bar: after the 2026-08-11 gap-fill the
-before deploying — that usually means a data problem (a gauge that went quiet, or
+expected baseline is P.1 6 h warning PR-AUC ≈ 0.78 and MAE ≈ 5.5 cm (the
-a bad backfill) rather than a modelling one.
+historical ~0.97 figure belonged to the sparse pre-fill grid — see section 5).
 A *material drop from the previous run* usually means a data problem (a gauge
 that went quiet, or a bad backfill) rather than a modelling one.
 **Run the tests** (synthetic data only, no database or network required):
@@ -590,10 +860,12 @@ a bad backfill) rather than a modelling one.
 python -m pytest tests/test_flood_forecast.py -v
 ```
-Seven tests covering leakage, label alignment, the coverage gate, forward-fill and
+Tests cover leakage, label alignment, the coverage gate, forward-fill and
-staleness, a train/predict round trip, the heuristic fallback, and feature-name
+staleness, a train/predict round trip, the heuristic fallback, feature-name
-stability. The whole suite runs in about 8 seconds, so there is no excuse for
+stability, and the rain-downgrade guard (no rain series → `RainUnavailableError`,
-skipping it before a deploy.
+nothing written; `--no-rain` → v2; rain present → v3 with the rain columns in
 `feature_names`). The file runs in well under a minute, so there is no excuse
 for skipping it before a deploy.
 **Understanding graceful degradation.** Three things can make a forecast row
 non-model-backed, and all of them are visible in the payload:
@@ -1,475 +0,0 @@
 # Geolocation Support for Grafana Geomap
 This guide explains the geolocation functionality added to the Thailand Water Monitor for use with Grafana's geomap visualization.
 ## ✅ **Implemented Features**
 ### **Database Schema Updates**
 All database adapters now support geolocation fields:
 - **latitude**: Decimal latitude coordinates (DECIMAL(10,8) for SQL, REAL for SQLite)
 - **longitude**: Decimal longitude coordinates (DECIMAL(11,8) for SQL, REAL for SQLite)  
 - **geohash**: Geohash string for efficient spatial indexing (VARCHAR(20)/TEXT)
 ### **Station Data Enhancement**
 Station mapping now includes geolocation fields:
 ```python
 '8': {
    'code': 'P.1', 
    'thai_name': 'สะพานนวรัฐ', 
    'english_name': 'Nawarat Bridge',
    'latitude': 15.6944,    # Decimal degrees
    'longitude': 100.2028,  # Decimal degrees
    'geohash': 'w5q6uuhvfcfp25'  # Geohash for P.1
 }
 ```
 ## 🗄️ **Database Schema**
 ### **Updated Stations Table**
 ```sql
 CREATE TABLE stations (
    id INTEGER PRIMARY KEY,
    station_code TEXT UNIQUE NOT NULL,
    thai_name TEXT NOT NULL,
    english_name TEXT NOT NULL,
    latitude REAL,           -- NEW: Latitude coordinate
    longitude REAL,          -- NEW: Longitude coordinate
    geohash TEXT,           -- NEW: Geohash for spatial indexing
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
 );
 ```
 ### **Database Support**
 - ✅ **SQLite**: REAL columns for coordinates, TEXT for geohash
 - ✅ **PostgreSQL**: DECIMAL(10,8) and DECIMAL(11,8) for coordinates, VARCHAR(20) for geohash
 - ✅ **MySQL**: DECIMAL(10,8) and DECIMAL(11,8) for coordinates, VARCHAR(20) for geohash
 - ✅ **VictoriaMetrics**: Geolocation data included in metric labels
 ## 📊 **Current Station Data**
 ### **P.1 - Nawarat Bridge (Sample)**
 - **Station Code**: P.1
 - **Thai Name**: สะพานนวรัฐ
 - **English Name**: Nawarat Bridge
 - **Latitude**: 15.6944
 - **Longitude**: 100.2028
 - **Geohash**: w5q6uuhvfcfp25
 ### **Remaining Stations**
 The following stations are ready for geolocation data when coordinates become available:
 - P.20 - บ้านเชียงดาว (Ban Chiang Dao)
 - P.75 - บ้านช่อแล (Ban Chai Lat)
 - P.92 - บ้านเมืองกึ๊ด (Ban Muang Aut)
 - P.4A - บ้านแม่แตง (Ban Mae Taeng)
 - P.67 - บ้านแม่แต (Ban Tae)
 - P.21 - บ้านริมใต้ (Ban Rim Tai)
 - P.103 - สะพานวงแหวนรอบ 3 (Ring Bridge 3)
 - P.82 - บ้านสบวิน (Ban Sob win)
 - P.84 - บ้านพันตน (Ban Panton)
 - P.81 - บ้านโป่ง (Ban Pong)
 - P.5 - สะพานท่านาง (Tha Nang Bridge)
 - P.77 - บ้านสบแม่สะป๊วด (Baan Sop Mae Sapuord)
 - P.87 - บ้านป่าซาง (Ban Pa Sang)
 - P.76 - บ้านแม่อีไฮ (Banb Mae I Hai)
 - P.85 - บ้านหล่ายแก้ว (Baan Lai Kaew)
 ## 🗺️ **Grafana Geomap Integration**
 ### **Data Source Configuration**
 The geolocation data is automatically included in all database queries and can be used directly in Grafana:
 #### **SQLite/PostgreSQL/MySQL Query Example**
 ```sql
 SELECT 
    m.timestamp,
    s.station_code,
    s.english_name,
    s.thai_name,
    s.latitude,
    s.longitude,
    s.geohash,
    m.water_level,
    m.discharge,
    m.discharge_percent
 FROM water_measurements m
 JOIN stations s ON m.station_id = s.id
 WHERE s.latitude IS NOT NULL 
  AND s.longitude IS NOT NULL
 ORDER BY m.timestamp DESC
 ```
 #### **VictoriaMetrics Query Example**
 ```promql
 water_level{latitude!="",longitude!=""}
 ```
 ### **Geomap Panel Configuration**
 #### **1. Create Geomap Panel**
 1. Add new panel in Grafana
 2. Select "Geomap" visualization
 3. Configure data source (SQLite/PostgreSQL/MySQL/VictoriaMetrics)
 #### **2. Configure Location Fields**
 - **Latitude Field**: `latitude`
 - **Longitude Field**: `longitude`
 - **Alternative**: Use `geohash` field for geohash-based positioning
 #### **3. Configure Display Options**
 - **Station Labels**: Use `station_code` or `english_name`
 - **Tooltip Information**: Include `thai_name`, `water_level`, `discharge`
 - **Color Mapping**: Map to `water_level` or `discharge_percent`
 #### **4. Sample Geomap Configuration**
 ```json
 {
  "type": "geomap",
  "title": "Thailand Water Stations",
  "targets": [
    {
      "rawSql": "SELECT latitude, longitude, station_code, english_name, water_level, discharge_percent FROM stations s JOIN water_measurements m ON s.id = m.station_id WHERE s.latitude IS NOT NULL AND m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id)",
      "format": "table"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "custom": {
        "hideFrom": {
          "legend": false,
          "tooltip": false,
          "vis": false
        }
      },
      "mappings": [],
      "color": {
        "mode": "continuous-GrYlRd",
        "field": "water_level"
      }
    }
  },
  "options": {
    "view": {
      "id": "coords",
      "lat": 15.6944,
      "lon": 100.2028,
      "zoom": 8
    },
    "controls": {
      "mouseWheelZoom": true,
      "showZoom": true,
      "showAttribution": true
    },
    "layers": [
      {
        "type": "markers",
        "config": {
          "size": {
            "field": "discharge_percent",
            "min": 5,
            "max": 20
          },
          "color": {
            "field": "water_level"
          },
          "showLegend": true
        }
      }
    ]
  }
 }
 ```
 ## 🔧 **Adding New Station Coordinates**
 ### **Method 1: Update Station Mapping**
 Edit `water_scraper_v3.py` and add coordinates to the station mapping:
 ```python
 '1': {
    'code': 'P.20', 
    'thai_name': 'บ้านเชียงดาว', 
    'english_name': 'Ban Chiang Dao',
    'latitude': 19.3056,    # Add actual coordinates
    'longitude': 98.9264,   # Add actual coordinates
    'geohash': 'w4r6...'    # Add actual geohash
 }
 ```
 ### **Method 2: Direct Database Update**
 ```sql
 UPDATE stations 
 SET latitude = 19.3056, longitude = 98.9264, geohash = 'w4r6uuhvfcfp25'
 WHERE station_code = 'P.20';
 ```
 ### **Method 3: Bulk Update Script**
 ```python
 import sqlite3
 coordinates = {
    'P.20': {'lat': 19.3056, 'lon': 98.9264, 'geohash': 'w4r6uuhvfcfp25'},
    'P.75': {'lat': 18.7756, 'lon': 99.1234, 'geohash': 'w4r5uuhvfcfp25'},
    # Add more stations...
 }
 conn = sqlite3.connect('water_monitoring.db')
 cursor = conn.cursor()
 for station_code, coords in coordinates.items():
    cursor.execute("""
        UPDATE stations 
        SET latitude = ?, longitude = ?, geohash = ?
        WHERE station_code = ?
    """, (coords['lat'], coords['lon'], coords['geohash'], station_code))
 conn.commit()
 conn.close()
 ```
 ## 🌐 **Geohash Information**
 ### **What is Geohash?**
 Geohash is a geocoding system that represents geographic coordinates as a short alphanumeric string. It provides:
 - **Spatial Indexing**: Efficient spatial queries
 - **Proximity**: Similar geohashes indicate nearby locations
 - **Hierarchical**: Longer geohashes provide more precision
 ### **Geohash Precision Levels**
 - **5 characters**: ~2.4km precision
 - **6 characters**: ~610m precision  
 - **7 characters**: ~76m precision
 - **8 characters**: ~19m precision
 - **9+ characters**: <5m precision
 ### **Example: P.1 Geohash**
 - **Geohash**: `w5q6uuhvfcfp25`
 - **Length**: 14 characters
 - **Precision**: Sub-meter accuracy
 - **Location**: Nawarat Bridge, Thailand
 ## 📈 **Grafana Visualization Examples**
 ### **1. Station Location Map**
 - **Type**: Geomap with markers
 - **Data**: Current station locations
 - **Color**: Water level or discharge percentage
 - **Size**: Discharge volume
 ### **2. Regional Water Levels**
 - **Type**: Geomap with heatmap
 - **Data**: Water level data across regions
 - **Visualization**: Color-coded intensity map
 - **Filters**: Time range, station groups
 ### **3. Alert Zones**
 - **Type**: Geomap with threshold markers
 - **Data**: Stations exceeding alert thresholds
 - **Visualization**: Red markers for high water levels
 - **Alerts**: Automated notifications for critical levels
 ## 🔄 **Updating a Running System**
 ### **Automated Migration Script**
 Use the provided migration script to safely add geolocation columns to your existing database:
 ```bash
 # Stop the water monitoring service first
 sudo systemctl stop water-monitor
 # Run the migration script
 python migrate_geolocation.py
 # Restart the service
 sudo systemctl start water-monitor
 ```
 ### **Migration Script Features**
 - ✅ **Auto-detects database type** from environment variables
 - ✅ **Checks existing columns** to avoid conflicts
 - ✅ **Supports all database types** (SQLite, PostgreSQL, MySQL)
 - ✅ **Adds sample data** for P.1 station
 - ✅ **Safe operation** - won't break existing data
 ### **Step-by-Step Migration Process**
 #### **1. Stop the Application**
 ```bash
 # If running as systemd service
 sudo systemctl stop water-monitor
 # If running in screen/tmux
 # Use Ctrl+C to stop the process
 # If running as Docker container
 docker stop water-monitor
 ```
 #### **2. Backup Your Database**
 ```bash
 # SQLite backup
 cp water_monitoring.db water_monitoring.db.backup
 # PostgreSQL backup
 pg_dump water_monitoring > water_monitoring_backup.sql
 # MySQL backup
 mysqldump water_monitoring > water_monitoring_backup.sql
 ```
 #### **3. Run Migration Script**
 ```bash
 # Default (uses environment variables)
 python migrate_geolocation.py
 # Or specify database path for SQLite
 SQLITE_DB_PATH=/path/to/water_monitoring.db python migrate_geolocation.py
 ```
 #### **4. Verify Migration**
 ```bash
 # Check SQLite schema
 sqlite3 water_monitoring.db ".schema stations"
 # Check PostgreSQL schema
 psql -d water_monitoring -c "\d stations"
 # Check MySQL schema
 mysql -e "DESCRIBE water_monitoring.stations"
 ```
 #### **5. Update Application Code**
 Ensure you have the latest version of the application with geolocation support:
 ```bash
 # Pull latest code
 git pull origin main
 # Install any new dependencies
 pip install -r requirements.txt
 ```
 #### **6. Restart Application**
 ```bash
 # Systemd service
 sudo systemctl start water-monitor
 # Docker container
 docker start water-monitor
 # Manual execution
 python water_scraper_v3.py
 ```
 ### **Migration Output Example**
 ```
 2025-07-28 17:30:00,123 - INFO - Starting geolocation column migration...
 2025-07-28 17:30:00,124 - INFO - Detected database type: SQLITE
 2025-07-28 17:30:00,125 - INFO - Migrating SQLite database: water_monitoring.db
 2025-07-28 17:30:00,126 - INFO - Current columns in stations table: ['id', 'station_code', 'thai_name', 'english_name', 'created_at', 'updated_at']
 2025-07-28 17:30:00,127 - INFO - Added latitude column
 2025-07-28 17:30:00,128 - INFO - Added longitude column
 2025-07-28 17:30:00,129 - INFO - Added geohash column
 2025-07-28 17:30:00,130 - INFO - Successfully added columns: latitude, longitude, geohash
 2025-07-28 17:30:00,131 - INFO - Updated P.1 station with sample geolocation data
 2025-07-28 17:30:00,132 - INFO - P.1 station geolocation: ('P.1', 15.6944, 100.2028, 'w5q6uuhvfcfp25')
 2025-07-28 17:30:00,133 - INFO - ✅ Migration completed successfully!
 2025-07-28 17:30:00,134 - INFO - You can now restart your water monitoring application
 2025-07-28 17:30:00,135 - INFO - The system will automatically use the new geolocation columns
 ```
 ## 🔍 **Troubleshooting**
 ### **Migration Issues**
 #### **Database Locked Error**
 ```bash
 # Stop all processes using the database
 sudo systemctl stop water-monitor
 pkill -f water_scraper
 # Wait a few seconds, then run migration
 sleep 5
 python migrate_geolocation.py
 ```
 #### **Permission Denied**
 ```bash
 # Check database file permissions
 ls -la water_monitoring.db
 # Fix permissions if needed
 sudo chown $USER:$USER water_monitoring.db
 chmod 664 water_monitoring.db
 ```
 #### **Missing Dependencies**
 ```bash
 # For PostgreSQL
 pip install psycopg2-binary
 # For MySQL
 pip install pymysql
 # For all databases
 pip install -r requirements.txt
 ```
 ### **Verification Issues**
 #### **Missing Coordinates**
 If stations don't appear on the geomap:
 1. Check if latitude/longitude are NULL in database
 2. Verify geolocation data in station mapping
 3. Ensure database schema includes geolocation columns
 4. Run migration script if columns are missing
 #### **Incorrect Positioning**
 If stations appear in wrong locations:
 1. Verify coordinate format (decimal degrees)
 2. Check latitude/longitude order (lat first, lon second)
 3. Validate geohash accuracy
 ### **Rollback Procedure**
 If migration causes issues:
 #### **SQLite Rollback**
 ```bash
 # Stop application
 sudo systemctl stop water-monitor
 # Restore backup
 cp water_monitoring.db.backup water_monitoring.db
 # Restart with old version
 sudo systemctl start water-monitor
 ```
 #### **PostgreSQL Rollback**
 ```sql
 -- Remove added columns
 ALTER TABLE stations DROP COLUMN IF EXISTS latitude;
 ALTER TABLE stations DROP COLUMN IF EXISTS longitude;
 ALTER TABLE stations DROP COLUMN IF EXISTS geohash;
 ```
 #### **MySQL Rollback**
 ```sql
 -- Remove added columns
 ALTER TABLE stations DROP COLUMN latitude;
 ALTER TABLE stations DROP COLUMN longitude;
 ALTER TABLE stations DROP COLUMN geohash;
 ```
 ## 🎯 **Next Steps**
 ### **Immediate Actions**
 1. **Gather Coordinates**: Collect GPS coordinates for all 16 stations
 2. **Update Database**: Add coordinates to remaining stations
 3. **Create Dashboards**: Build Grafana geomap visualizations
 ### **Future Enhancements**
 1. **Automatic Geocoding**: API integration for address-to-coordinate conversion
 2. **Mobile GPS**: Mobile app for field coordinate collection
 3. **Satellite Integration**: Satellite imagery overlay in Grafana
 4. **Geofencing**: Alert zones based on geographic boundaries
 The geolocation functionality is now fully implemented and ready for use with Grafana's geomap visualization. Station P.1 (Nawarat Bridge) serves as a working example with complete coordinate data.
@@ -286,8 +286,6 @@ make validate-workflows
 ### **Project-Specific Resources**
 - [Contributing Guide](../CONTRIBUTING.md)
 - [Deployment Checklist](../DEPLOYMENT_CHECKLIST.md)
 - [Project Structure](PROJECT_STRUCTURE.md)
 ### **Monitoring and Alerts**
 - Workflow status badges in README
@@ -1,168 +0,0 @@
 # Grafana Matrix Alerting Setup
 ## Overview
 Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.
 ## Prerequisites
 - Grafana instance with your PostgreSQL data source
 - Matrix account and access token
 - Matrix room for alerts
 ## Step 1: Configure Matrix Contact Point
 1. **In Grafana, go to Alerting → Contact Points**
 2. **Add new contact point:**
   ```
   Name: matrix-water-alerts
   Integration: Webhook
   URL: https://matrix.org/_matrix/client/v3/rooms/!ROOM_ID:matrix.org/send/m.room.message
   HTTP Method: POST
   ```
 3. **Add Headers:**
   ```
   Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
   Content-Type: application/json
   ```
 4. **Message Template:**
   ```json
   {
     "msgtype": "m.text",
     "body": "🌊 WATER ALERT: {{ .CommonLabels.alertname }}\n\nStation: {{ .CommonLabels.station_code }}\nLevel: {{ .CommonAnnotations.water_level }}m\nStatus: {{ .CommonLabels.severity }}\n\nTime: {{ .CommonAnnotations.time }}"
   }
   ```
 ## Step 2: Create Alert Rules
 ### High Water Level Alert
 ```yaml
 Rule Name: high-water-level
 Query: water_level > 6.0
 Condition: IS ABOVE 6.0 FOR 5m
 Labels:
  - severity: critical
  - station_code: {{ .station_code }}
 Annotations:
  - water_level: {{ .water_level }}
  - summary: "Critical water level at {{ .station_code }}"
 ```
 ### Low Water Level Alert
 ```yaml
 Rule Name: low-water-level
 Query: water_level < 1.0
 Condition: IS BELOW 1.0 FOR 10m
 Labels:
  - severity: warning
  - station_code: {{ .station_code }}
 ```
 ### Data Gap Alert
 ```yaml
 Rule Name: data-gap
 Query: increase(measurements_total[1h]) == 0
 Condition: IS EQUAL TO 0 FOR 30m
 Labels:
  - severity: warning
  - issue: data-gap
 ```
 ## Step 3: Matrix Setup
 ### Get Matrix Access Token
 ```bash
 curl -X POST https://matrix.org/_matrix/client/v3/login \
  -H "Content-Type: application/json" \
  -d '{
    "type": "m.login.password",
    "user": "your_username",
    "password": "your_password"
  }'
 ```
 ### Create Alert Room
 ```bash
 curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Water Level Alerts - Northern Thailand",
    "topic": "Automated alerts for Ping River water monitoring",
    "preset": "trusted_private_chat"
  }'
 ```
 ## Example Alert Queries
 ### Critical Water Levels
 ```promql
 # High water alert
 water_level{station_code=~"P.1|P.4A|P.20"} > 6.0
 # Dangerous discharge
 discharge{station_code=~".*"} > 500
 # Rapid level change
 increase(water_level[15m]) > 0.5
 ```
 ### System Health
 ```promql
 # No data received
 up{job="water-monitor"} == 0
 # Old data
 (time() - timestamp) > 7200
 ```
 ## Alert Notification Format
 Your Matrix messages will look like:
 ```
 🌊 WATER ALERT: High Water Level
 Station: P.1 (Chiang Mai)
 Level: 6.2m (CRITICAL)
 Discharge: 450 cms
 Status: DANGER
 Time: 2025-09-26 14:30:00
 Trend: Rising (+0.3m in 30min)
 📍 Location: 18.7883°N, 98.9853°E
 ```
 ## Advanced Features
 ### Escalation Rules
 ```yaml
 # Send to different rooms based on severity
 - if: severity == "critical"
  receiver: matrix-emergency
 - if: severity == "warning"
  receiver: matrix-alerts
 - if: time_of_day() outside "08:00-20:00"
  receiver: matrix-night-duty
 ```
 ### Rate Limiting
 ```yaml
 group_wait: 5m
 group_interval: 10m
 repeat_interval: 30m
 ```
 ## Testing Alerts
 1. **Test Contact Point** - Use Grafana's test button
 2. **Simulate Alert** - Manually trigger with test data
 3. **Verify Matrix** - Check message formatting and delivery
 ## Troubleshooting
 ### Common Issues
 - **403 Forbidden**: Check Matrix access token
 - **Room not found**: Verify room ID format
 - **No alerts**: Check query syntax and thresholds
 - **Spam**: Configure proper grouping and intervals
@@ -1,351 +0,0 @@
 # Complete Grafana Matrix Alerting Setup Guide
 ## Overview
 Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.
 ## Prerequisites
 - Grafana instance running (v8.0+)
 - PostgreSQL data source configured in Grafana
 - Matrix account
 - Matrix room for alerts
 ## Step 1: Get Matrix Access Token
 ### Method 1: Using curl
 ```bash
 curl -X POST https://matrix.org/_matrix/client/v3/login \
  -H "Content-Type: application/json" \
  -d '{
    "type": "m.login.password",
    "user": "your_username",
    "password": "your_password"
  }'
 ```
 ### Method 2: Using Element Web Client
 1. Open Element in browser: https://app.element.io
 2. Login to your account
 3. Go to Settings → Help & About → Advanced
 4. Copy your Access Token
 ### Method 3: Using Matrix Admin Panel
 - If you have admin access to your homeserver, generate token via admin API
 ## Step 2: Create Alert Room
 ```bash
 curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Water Level Alerts - Northern Thailand",
    "topic": "Automated alerts for Ping River water monitoring",
    "preset": "private_chat"
  }'
 ```
 Save the `room_id` from the response (format: !roomid:homeserver.com)
 ## Step 3: Configure Grafana Contact Point
 ### Navigate to Alerting
 1. In Grafana, go to **Alerting → Contact Points**
 2. Click **Add contact point**
 ### Contact Point Settings
 ```
 Name: matrix-water-alerts
 Integration: Webhook
 URL: https://matrix.org/_matrix/client/v3/rooms/!YOUR_ROOM_ID:matrix.org/send/m.room.message/{{ .GroupLabels.alertname }}_{{ .GroupLabels.severity }}_{{ now.Unix }}
 HTTP Method: POST
 ```
 ### Headers
 ```
 Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
 Content-Type: application/json
 ```
 ### Message Template (JSON Body)
 ```json
 {
  "msgtype": "m.text",
  "body": "🌊 **PING RIVER WATER ALERT**\n\n**Alert:** {{ .GroupLabels.alertname }}\n**Severity:** {{ .GroupLabels.severity | toUpper }}\n**Station:** {{ .GroupLabels.station_code }} ({{ .GroupLabels.station_name }})\n\n{{ range .Alerts }}**Status:** {{ .Status | toUpper }}\n**Water Level:** {{ .Annotations.water_level }}m\n**Threshold:** {{ .Annotations.threshold }}m\n**Time:** {{ .StartsAt.Format \"2006-01-02 15:04:05\" }}\n{{ if .Annotations.discharge }}**Discharge:** {{ .Annotations.discharge }} cms\n{{ end }}{{ if .Annotations.message }}**Details:** {{ .Annotations.message }}\n{{ end }}{{ end }}\n📈 **Dashboard:** {{ .ExternalURL }}\n📍 **Location:** Northern Thailand Ping River"
 }
 ```
 ## Step 4: Create Alert Rules
 ### High Water Level Alert
 ```yaml
 # Rule Configuration
 Rule Name: high-water-level
 Evaluation Group: water-level-alerts
 Folder: Water Monitoring
 # Query A
 SELECT
  station_code,
  station_name_th as station_name,
  water_level,
  discharge,
  timestamp
 FROM water_measurements
 WHERE
  timestamp > now() - interval '5 minutes'
  AND water_level > 6.0
 # Condition
 IS ABOVE 6.0 FOR 5 minutes
 # Labels
 severity: critical
 alertname: High Water Level
 station_code: {{ $labels.station_code }}
 station_name: {{ $labels.station_name }}
 # Annotations
 water_level: {{ $values.water_level }}
 threshold: 6.0
 discharge: {{ $values.discharge }}
 summary: Critical water level detected at {{ $labels.station_code }}
 ```
 ### Emergency Water Level Alert
 ```yaml
 Rule Name: emergency-water-level
 Query: water_level > 8.0
 Condition: IS ABOVE 8.0 FOR 2 minutes
 Labels:
  severity: emergency
  alertname: Emergency Water Level
 Annotations:
  threshold: 8.0
  message: IMMEDIATE ACTION REQUIRED - Flood risk imminent
 ```
 ### Low Water Level Alert
 ```yaml
 Rule Name: low-water-level
 Query: water_level < 1.0
 Condition: IS BELOW 1.0 FOR 15 minutes
 Labels:
  severity: warning
  alertname: Low Water Level
 Annotations:
  threshold: 1.0
  message: Drought conditions detected
 ```
 ### Data Gap Alert
 ```yaml
 Rule Name: data-gap
 Query:
  SELECT
    station_code,
    MAX(timestamp) as last_seen
  FROM water_measurements
  GROUP BY station_code
  HAVING MAX(timestamp) < now() - interval '2 hours'
 Condition: HAS NO DATA FOR 30 minutes
 Labels:
  severity: warning
  alertname: Data Gap
  issue: missing-data
 ```
 ### Rapid Level Change Alert
 ```yaml
 Rule Name: rapid-level-change
 Query:
  SELECT
    station_code,
    water_level,
    LAG(water_level, 1) OVER (PARTITION BY station_code ORDER BY timestamp) as prev_level
  FROM water_measurements
  WHERE timestamp > now() - interval '15 minutes'
  HAVING ABS(water_level - prev_level) > 0.5
 Condition: CHANGE > 0.5m FOR 1 minute
 Labels:
  severity: warning
  alertname: Rapid Water Level Change
 ```
 ## Step 5: Configure Notification Policy
 ### Create Notification Policy
 ```yaml
 # Policy Tree
 - receiver: matrix-water-alerts
  match:
    severity: emergency|critical
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 30m
 - receiver: matrix-water-alerts
  match:
    severity: warning
  group_wait: 30s
  group_interval: 10m
  repeat_interval: 2h
 ```
 ### Grouping Rules
 ```yaml
 group_by: [alertname, station_code]
 group_wait: 10s
 group_interval: 5m
 repeat_interval: 1h
 ```
 ## Step 6: Station-Specific Thresholds
 Create separate rules for each station with appropriate thresholds:
 ```sql
 -- P.1 (Chiang Mai) - Urban area, higher thresholds
 SELECT * FROM water_measurements
 WHERE station_code = 'P.1' AND water_level > 6.5
 -- P.4A (Mae Ping) - Agricultural area
 SELECT * FROM water_measurements
 WHERE station_code = 'P.4A' AND water_level > 5.0
 -- P.20 (Downstream) - Lower threshold
 SELECT * FROM water_measurements
 WHERE station_code = 'P.20' AND water_level > 4.0
 ```
 ## Step 7: Advanced Features
 ### Time-Based Routing
 ```yaml
 # Different receivers for day/night
 time_intervals:
  - name: working_hours
    time_intervals:
      - times:
        - start_time: '08:00'
          end_time: '20:00'
        weekdays: ['monday:friday']
 routes:
  - receiver: matrix-alerts-day
    match:
      severity: warning
    active_time_intervals: [working_hours]
  - receiver: matrix-alerts-night
    match:
      severity: warning
    active_time_intervals: ['!working_hours']
 ```
 ### Multi-Channel Alerts
 ```yaml
 # Send critical alerts to multiple rooms
 - receiver: matrix-emergency
  webhook_configs:
    - url: https://matrix.org/_matrix/client/v3/rooms/!emergency:matrix.org/send/m.room.message
      http_config:
        authorization:
          credentials: "Bearer EMERGENCY_TOKEN"
    - url: https://matrix.org/_matrix/client/v3/rooms/!general:matrix.org/send/m.room.message
      http_config:
        authorization:
          credentials: "Bearer GENERAL_TOKEN"
 ```
 ## Step 8: Testing
 ### Test Contact Point
 1. Go to Contact Points in Grafana
 2. Select your Matrix contact point
 3. Click "Test" button
 4. Check Matrix room for test message
 ### Test Alert Rules
 1. Temporarily lower thresholds
 2. Wait for condition to trigger
 3. Verify alert appears in Grafana
 4. Verify Matrix message received
 5. Reset thresholds
 ### Manual Alert Trigger
 ```bash
 # Simulate high water level in database
 INSERT INTO water_measurements (station_code, water_level, timestamp)
 VALUES ('P.1', 7.5, NOW());
 ```
 ## Troubleshooting
 ### Common Issues
 #### 403 Forbidden
 - **Cause**: Invalid Matrix access token
 - **Fix**: Regenerate token or check permissions
 #### Room Not Found
 - **Cause**: Incorrect room ID format
 - **Fix**: Ensure room ID starts with ! and includes homeserver
 #### No Alerts Firing
 - **Cause**: Query returns no results
 - **Fix**: Test queries in Grafana Explore, check data availability
 #### Alert Spam
 - **Cause**: No grouping configured
 - **Fix**: Configure proper group_by and intervals
 #### Messages Not Formatted
 - **Cause**: Template syntax errors
 - **Fix**: Validate JSON template, check Grafana template docs
 ### Debug Steps
 1. Check Grafana alert rule status
 2. Verify contact point test succeeds
 3. Check Grafana logs: `/var/log/grafana/grafana.log`
 4. Test Matrix API directly with curl
 5. Verify database connectivity and query results
 ## Environment Variables
 Add to your `.env`:
 ```bash
 MATRIX_HOMESERVER=https://matrix.org
 MATRIX_ACCESS_TOKEN=your_access_token_here
 MATRIX_ROOM_ID=!your_room_id:matrix.org
 GRAFANA_URL=http://your-grafana-host:3000
 ```
 ## Example Alert Message
 Your Matrix messages will appear as:
 ```
 🌊 **PING RIVER WATER ALERT**
 **Alert:** High Water Level
 **Severity:** CRITICAL
 **Station:** P.1 (สถานีเชียงใหม่)
 **Status:** FIRING
 **Water Level:** 6.75m
 **Threshold:** 6.0m
 **Time:** 2025-09-26 14:30:00
 **Discharge:** 450.2 cms
 📈 **Dashboard:** http://grafana:3000
 📍 **Location:** Northern Thailand Ping River
 ```
 ## Security Notes
 - Store Matrix tokens securely (environment variables)
 - Use room-specific tokens when possible
 - Enable rate limiting to prevent spam
 - Consider using dedicated alerting user account
 - Regularly rotate access tokens
 This setup provides comprehensive water level monitoring with immediate Matrix notifications when thresholds are exceeded.
@@ -1,389 +0,0 @@
 # HTTPS VictoriaMetrics Configuration Guide
 This guide explains how to configure the Thailand Water Monitor to connect to VictoriaMetrics through HTTPS and reverse proxies.
 ## Configuration Options
 ### 1. Environment Variables for HTTPS
 ```bash
 # Option 1: Full HTTPS URL (Recommended)
 export DB_TYPE=victoriametrics
 export VM_HOST=https://vm.example.com
 export VM_PORT=443
 # Option 2: Host and port separately
 export DB_TYPE=victoriametrics
 export VM_HOST=vm.example.com
 export VM_PORT=443
 # Option 3: Custom port with HTTPS
 export DB_TYPE=victoriametrics
 export VM_HOST=https://vm.example.com
 export VM_PORT=8443
 ```
 ### 2. Windows PowerShell Configuration
 ```powershell
 # Set environment variables for HTTPS
 $env:DB_TYPE="victoriametrics"
 $env:VM_HOST="https://vm.example.com"
 $env:VM_PORT="443"
 # Run the water monitor
 python water_scraper_v3.py
 ```
 ### 3. Linux/Mac Configuration
 ```bash
 # Set environment variables for HTTPS
 export DB_TYPE=victoriametrics
 export VM_HOST=https://vm.example.com
 export VM_PORT=443
 # Run the water monitor
 python water_scraper_v3.py
 ```
 ## Reverse Proxy Examples
 ### 1. Nginx Reverse Proxy
 ```nginx
 server {
    listen 443 ssl http2;
    server_name vm.example.com;
    # SSL Configuration
    ssl_certificate /path/to/certificate.crt;
    ssl_certificate_key /path/to/private.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;
    # Optional: Basic authentication
    # auth_basic "VictoriaMetrics";
    # auth_basic_user_file /etc/nginx/.htpasswd;
    location / {
        proxy_pass http://localhost:8428;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # WebSocket support (if needed)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
 }
 # Redirect HTTP to HTTPS
 server {
    listen 80;
    server_name vm.example.com;
    return 301 https://$server_name$request_uri;
 }
 ```
 ### 2. Apache Reverse Proxy
 ```apache
 <VirtualHost *:443>
    ServerName vm.example.com
    # SSL Configuration
    SSLEngine on
    SSLCertificateFile /path/to/certificate.crt
    SSLCertificateKeyFile /path/to/private.key
    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    # Security headers
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
    Header always set X-Frame-Options DENY
    Header always set X-Content-Type-Options nosniff
    # Reverse proxy configuration
    ProxyPreserveHost On
    ProxyPass / http://localhost:8428/
    ProxyPassReverse / http://localhost:8428/
    # Optional: Basic authentication
    # AuthType Basic
    # AuthName "VictoriaMetrics"
    # AuthUserFile /etc/apache2/.htpasswd
    # Require valid-user
 </VirtualHost>
 <VirtualHost *:80>
    ServerName vm.example.com
    Redirect permanent / https://vm.example.com/
 </VirtualHost>
 ```
 ### 3. Traefik Reverse Proxy
 ```yaml
 # docker-compose.yml with Traefik
 version: '3.8'
 services:
  traefik:
    image: traefik:v2.10
    command:
      - --api.dashboard=true
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --providers.docker=true
      - --certificatesresolvers.letsencrypt.acme.tlschallenge=true
      - --certificatesresolvers.letsencrypt.acme.email=admin@example.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - letsencrypt:/letsencrypt
    labels:
      - traefik.http.routers.api.rule=Host(`traefik.example.com`)
      - traefik.http.routers.api.tls.certresolver=letsencrypt
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    command:
      - '--storageDataPath=/victoria-metrics-data'
      - '--retentionPeriod=2y'
      - '--httpListenAddr=:8428'
    volumes:
      - vm_data:/victoria-metrics-data
    labels:
      - traefik.enable=true
      - traefik.http.routers.vm.rule=Host(`vm.example.com`)
      - traefik.http.routers.vm.tls.certresolver=letsencrypt
      - traefik.http.services.vm.loadbalancer.server.port=8428
 volumes:
  vm_data:
  letsencrypt:
 ```
 ## Testing HTTPS Configuration
 ### 1. Test Connection
 ```bash
 # Test HTTPS connection
 curl -k https://vm.example.com/health
 # Test with specific port
 curl -k https://vm.example.com:8443/health
 # Test API endpoint
 curl -k "https://vm.example.com/api/v1/query?query=up"
 ```
 ### 2. Test with Water Monitor
 ```bash
 # Set environment variables
 export DB_TYPE=victoriametrics
 export VM_HOST=https://vm.example.com
 export VM_PORT=443
 # Test with demo script
 python demo_databases.py victoriametrics
 # Run full water monitor
 python water_scraper_v3.py
 ```
 ### 3. Verify SSL Certificate
 ```bash
 # Check SSL certificate
 openssl s_client -connect vm.example.com:443 -servername vm.example.com
 # Check certificate expiration
 echo | openssl s_client -connect vm.example.com:443 2>/dev/null | openssl x509 -noout -dates
 ```
 ## Configuration Examples
 ### 1. Production HTTPS Setup
 ```bash
 # Environment variables for production
 export DB_TYPE=victoriametrics
 export VM_HOST=https://metrics.company.com
 export VM_PORT=443
 export LOG_LEVEL=INFO
 export SCRAPING_INTERVAL_HOURS=1
 # Run water monitor
 python water_scraper_v3.py
 ```
 ### 2. Development with Self-Signed Certificate
 ```bash
 # For development with self-signed certificates
 export DB_TYPE=victoriametrics
 export VM_HOST=https://dev-vm.local
 export VM_PORT=443
 export PYTHONHTTPSVERIFY=0  # Disable SSL verification (dev only)
 python water_scraper_v3.py
 ```
 ### 3. Custom Port Configuration
 ```bash
 # Custom HTTPS port
 export DB_TYPE=victoriametrics
 export VM_HOST=https://vm.example.com
 export VM_PORT=8443
 python water_scraper_v3.py
 ```
 ## Troubleshooting HTTPS Issues
 ### 1. SSL Certificate Errors
 ```bash
 # Error: SSL certificate verify failed
 # Solution: Check certificate validity
 openssl x509 -in certificate.crt -text -noout
 # Temporary workaround (not recommended for production)
 export PYTHONHTTPSVERIFY=0
 ```
 ### 2. Connection Timeout
 ```bash
 # Error: Connection timeout
 # Check firewall and network connectivity
 telnet vm.example.com 443
 nc -zv vm.example.com 443
 ```
 ### 3. DNS Resolution Issues
 ```bash
 # Error: Name resolution failed
 # Check DNS resolution
 nslookup vm.example.com
 dig vm.example.com
 ```
 ### 4. Proxy Configuration Issues
 ```bash
 # Check proxy logs
 # Nginx
 tail -f /var/log/nginx/error.log
 # Apache
 tail -f /var/log/apache2/error.log
 # Test direct connection to backend
 curl http://localhost:8428/health
 ```
 ## Security Best Practices
 ### 1. SSL/TLS Configuration
 - Use TLS 1.2 or higher
 - Disable weak ciphers
 - Enable HSTS headers
 - Use strong SSL certificates
 ### 2. Authentication
 ```nginx
 # Basic authentication in Nginx
 auth_basic "VictoriaMetrics Access";
 auth_basic_user_file /etc/nginx/.htpasswd;
 # Create password file
 htpasswd -c /etc/nginx/.htpasswd username
 ```
 ### 3. Network Security
 - Use firewall rules to restrict access
 - Consider VPN for internal access
 - Implement rate limiting
 - Monitor access logs
 ### 4. Certificate Management
 ```bash
 # Auto-renewal with Let's Encrypt
 certbot renew --dry-run
 # Certificate monitoring
 echo | openssl s_client -connect vm.example.com:443 2>/dev/null | \
 openssl x509 -noout -dates | grep notAfter
 ```
 ## Docker Configuration for HTTPS
 ### 1. Docker Compose with HTTPS
 ```yaml
 version: '3.8'
 services:
  water-monitor:
    build: .
    environment:
      - DB_TYPE=victoriametrics
      - VM_HOST=https://vm.example.com
      - VM_PORT=443
    restart: unless-stopped
    depends_on:
      - victoriametrics
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    ports:
      - "8428:8428"
    volumes:
      - vm_data:/victoria-metrics-data
    command:
      - '--storageDataPath=/victoria-metrics-data'
      - '--retentionPeriod=2y'
      - '--httpListenAddr=:8428'
 volumes:
  vm_data:
 ```
 ### 2. Environment File (.env)
 ```bash
 # .env file
 DB_TYPE=victoriametrics
 VM_HOST=https://vm.example.com
 VM_PORT=443
 LOG_LEVEL=INFO
 SCRAPING_INTERVAL_HOURS=1
 ```
 This configuration guide provides comprehensive instructions for setting up HTTPS connectivity to VictoriaMetrics through reverse proxies, ensuring secure and reliable data transmission for the Thailand Water Monitor.
@@ -1,136 +0,0 @@
 # Geolocation Migration Quick Start
 This is a quick reference guide for updating a running Thailand Water Monitor system to add geolocation support for Grafana geomap.
 ## 🚀 **Quick Migration (5 minutes)**
 ### **Step 1: Stop Application**
 ```bash
 # Stop the service (choose your method)
 sudo systemctl stop water-monitor
 # OR
 docker stop water-monitor
 # OR use Ctrl+C if running manually
 ```
 ### **Step 2: Backup Database**
 ```bash
 # SQLite backup
 cp water_monitoring.db water_monitoring.db.backup
 # PostgreSQL backup
 pg_dump water_monitoring > backup.sql
 # MySQL backup
 mysqldump water_monitoring > backup.sql
 ```
 ### **Step 3: Run Migration**
 ```bash
 # Run the automated migration script
 python migrate_geolocation.py
 ```
 ### **Step 4: Restart Application**
 ```bash
 # Restart the service
 sudo systemctl start water-monitor
 # OR
 docker start water-monitor
 # OR
 python water_scraper_v3.py
 ```
 ## ✅ **Expected Output**
 ```
 2025-07-28 17:30:00,123 - INFO - Starting geolocation column migration...
 2025-07-28 17:30:00,124 - INFO - Detected database type: SQLITE
 2025-07-28 17:30:00,127 - INFO - Added latitude column
 2025-07-28 17:30:00,128 - INFO - Added longitude column
 2025-07-28 17:30:00,129 - INFO - Added geohash column
 2025-07-28 17:30:00,133 - INFO - ✅ Migration completed successfully!
 ```
 ## 🗺️ **Verify Geolocation Works**
 ### **Check Database**
 ```bash
 # SQLite
 sqlite3 water_monitoring.db "SELECT station_code, latitude, longitude, geohash FROM stations WHERE station_code = 'P.1';"
 # Expected output: P.1|15.6944|100.2028|w5q6uuhvfcfp25
 ```
 ### **Test Application**
 ```bash
 # Run a test cycle
 python water_scraper_v3.py --test
 # Should complete without errors
 ```
 ## 🔧 **Grafana Setup**
 ### **Query for Geomap**
 ```sql
 SELECT 
    s.latitude, s.longitude, s.station_code, s.english_name,
    m.water_level, m.discharge_percent
 FROM stations s 
 JOIN water_measurements m ON s.id = m.station_id
 WHERE s.latitude IS NOT NULL 
  AND m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id)
 ```
 ### **Geomap Configuration**
 1. Create new panel → Select "Geomap"
 2. Set **Latitude field**: `latitude`
 3. Set **Longitude field**: `longitude`
 4. Set **Color field**: `water_level`
 5. Set **Size field**: `discharge_percent`
 ## 🚨 **Troubleshooting**
 ### **Database Locked**
 ```bash
 sudo systemctl stop water-monitor
 pkill -f water_scraper
 sleep 5
 python migrate_geolocation.py
 ```
 ### **Permission Error**
 ```bash
 sudo chown $USER:$USER water_monitoring.db
 chmod 664 water_monitoring.db
 ```
 ### **Missing Dependencies**
 ```bash
 pip install psycopg2-binary pymysql
 ```
 ## 🔄 **Rollback (if needed)**
 ```bash
 # Stop application
 sudo systemctl stop water-monitor
 # Restore backup
 cp water_monitoring.db.backup water_monitoring.db
 # Restart
 sudo systemctl start water-monitor
 ```
 ## 📚 **More Information**
 - **Full Guide**: See `GEOLOCATION_GUIDE.md`
 - **Migration Script**: `migrate_geolocation.py`
 - **Database Schema**: Updated with latitude, longitude, geohash columns
 ## 🎯 **What You Get**
 - ✅ **P.1 Station** ready for geomap (Nawarat Bridge)
 - ✅ **Database Schema** updated for all 16 stations
 - ✅ **Grafana Compatible** data structure
 - ✅ **Backward Compatible** - existing data preserved
 **Total Time**: ~5 minutes for complete migration
@@ -0,0 +1,132 @@
 # Flood notifications (ntfy)
 Public push notifications for threshold crossings, without accounts, mailing
 lists or app-store review: the monitor publishes to a self-hosted
 [ntfy](https://ntfy.sh) server, and anyone subscribes to the topics they care
 about from the free ntfy app (iOS, Android, F-Droid) or a browser tab.
 ntfy is one Go binary with a sqlite cache: ~30 MB RSS idle, negligible CPU. It
 runs on the same VPS as the monitor.
 ## What subscribers get
 Every message is a **transition**, never a state. Crossing up into a level sends
 one message; dropping back below it (with 0.10 m hysteresis) sends one
 all-clear. A river that sits at 3.9 m for three days produces two messages, not
 seventy-two. In a quiet season a subscriber hears nothing.
 | Topic | Trigger | Priority |
 |---|---|---|
 | `ping-warning` | any gauge crosses its warning threshold; levels falling back | 4 (high) / 2 |
 | `ping-danger` | any gauge crosses its danger threshold | 5 (max, breaks Do-Not-Disturb) |
 | `ping-<station>-warning` | that gauge crosses warning; back to normal | 4 / 2 |
 | `ping-<station>-danger` | that gauge crosses danger; back below danger | 5 / 3 |
 | `ping-p1-outlook` | model P(warning within 24 h) at P.1 rises through 50 % (clears below 25 %) | 4 / 2 |
 | `ping-status` | gauge feed stale ≥ 3 h; feed recovered | 3 / 2 |
 Station slugs are the code lowercased without the dot: `p1`, `p103`, `p67`.
 Thresholds are the ones in `src/ml/features.py` (`THRESHOLDS`): P.1 3.70 /
 4.20 m, P.103 5.95 / 6.75 m, and so on.
 The outlook topic is opt-in for a reason: it is model output, and the message
 says so. Observed-crossing topics only ever report a gauge reading.
 Each message carries a click-through and an "Open dashboard" action button to
 the public dashboard.
 ## How it runs
 `src/notify.py` is called once per collection cycle inside the API process
 (leader only), right after the forecast precompute, so it sees exactly the
 readings and forecasts the dashboard shows. Per-key last-sent state is stored
 in the `notification_state` table of the monitor's own database, so a restart
 or redeploy never re-sends and never misses a crossing that happened while
 the service was down (the next cycle compares against the persisted state).
 If ntfy is unreachable the transition is **not** recorded, so it is retried
 on the next cycle rather than silently lost. Any other failure in the notify
 step is logged and never reaches the collection loop.
 The dashboard's "🔔 Get alerts" button appears only when `NTFY_SERVER` is
 set; it reads `GET /api/notifications` and renders subscribe links
 (`ntfy://` deep links for the app, https links for the web UI).
 ## Deployment
 On the monitor VPS, as root:
 ```bash
 cd /opt/thailand-water-monitor
 NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
 ```
 This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
 host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
 message cache, signup/login/metrics off, tight visitor limits), enables the
 systemd unit,
 creates the `monitor` user with **write-only access to `ping-*`**, mints a
 token, and appends `NTFY_SERVER` (public URL for subscribers),
 `NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
 and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
 ```bash
 systemctl restart water-monitor
 journalctl -u water-monitor -n 20 | grep ntfy      # "ntfy notifications: https://... topics ping-*"
 curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1'   # anonymous read works
 ```
 The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
 the monitor host's Tailscale address and nothing is exposed on a public
 interface. On the Caddy machine:
 ```caddyfile
 ntfy.buildfor.life {
    reverse_proxy <monitor tailscale ip>:2586
 }
 ```
 Caddy proxies websockets and keeps long-poll connections open by default;
 subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
 `X-Forwarded-For` rather than treating every subscriber as the proxy.
 Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
 monitor at the Tailscale address directly, so a DNS or proxy problem never
 holds back an alert. Test the pipeline before the domain is live with
 `curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
 ## Configuration
 | Variable | Default | Meaning |
 |---|---|---|
 | `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
 | `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
 | `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
 | `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
 | `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
 Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
 (0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
 ## Testing
 `tests/test_notify.py` covers the state machine: quiet river sends nothing;
 crossing once, then silence while above, then all-clear; hysteresis on the way
 down; escalation to danger and back; basin digest grouping; outlook on/off;
 heuristic forecasts ignored; stale feed and recovery; state survives a restart
 through sqlite; a failed publish is retried next cycle.
 To exercise the real path against a real ntfy locally: run `ntfy serve` (any
 platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
 poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
 template.
 ## Why ntfy and not …
 - **Matrix** (`src/alerting.py`, still there): needs a homeserver account per
  subscriber and a room invite; fine for a team, wrong for the public.
 - **Gotify**: also self-hosted and light, but Android-only client and one
  account per subscriber.
 - **Email / SMS**: deliverability work, cost per message, no priority
  semantics; ntfy can forward to email per subscription if someone wants it.
 - **Telegram / LINE bots**: platform lock-in and a bot token in the loop; can be
  added later as ntfy→webhook fan-out without touching the monitor.
@@ -1,206 +0,0 @@
 # Thailand Water Monitor - Current Project Status
 ## 📁 **Clean Project Structure**
 The project has been cleaned up and organized with the following structure:
 ```
 water_level_monitor/
 ├── 📄 .gitignore                             # Git ignore rules
 ├── 📄 README.md                              # Main project documentation
 ├── 📄 requirements.txt                       # Python dependencies
 ├── 📄 config.py                              # Configuration management
 ├── 📄 water_scraper_v3.py                    # Main application (15-min scheduler)
 ├── 📄 database_adapters.py                   # Multi-database support
 ├── 📄 demo_databases.py                      # Database demonstration
 ├── 📄 Dockerfile                             # Container configuration
 ├── 📄 docker-compose.victoriametrics.yml     # VictoriaMetrics stack
 ├── 📚 Documentation/
 │   ├── 📄 DATABASE_DEPLOYMENT_GUIDE.md       # Multi-database setup guide
 │   ├── 📄 DEBIAN_TROUBLESHOOTING.md          # Linux deployment guide
 │   ├── 📄 ENHANCED_SCHEDULER_GUIDE.md        # 15-minute scheduler guide
 │   ├── 📄 GAP_FILLING_GUIDE.md              # Data gap filling guide
 │   ├── 📄 HTTPS_CONFIGURATION.md            # HTTPS setup guide
 │   └── 📄 VICTORIAMETRICS_SETUP.md          # VictoriaMetrics guide
 └── 📁 grafana/                              # Grafana configuration
    ├── 📁 provisioning/
    │   ├── 📁 datasources/
    │   │   └── 📄 victoriametrics.yml        # VictoriaMetrics data source
    │   └── 📁 dashboards/
    │       └── 📄 dashboard.yml              # Dashboard provider config
    └── 📁 dashboards/
        └── 📄 water-monitoring-dashboard.json # Pre-built dashboard
 ```
 ## 🧹 **Files Removed During Cleanup**
 ### **Old Data Files**
 - ❌ `thailand_water_data_v2.csv` - Old CSV export
 - ❌ `water_monitor.log` - Log file (regenerated automatically)
 - ❌ `water_monitoring.db` - SQLite database (recreated automatically)
 ### **Outdated Documentation**
 - ❌ `FINAL_SUMMARY.md` - Contained references to non-existent v2 files
 - ❌ `PROJECT_SUMMARY.md` - Outdated project information
 ### **System Files**
 - ❌ `__pycache__/` - Python compiled files directory
 ## ✅ **Current Features**
 ### **Enhanced 15-Minute Scheduler**
 - **Timing**: Runs every 15 minutes (1:00, 1:15, 1:30, 1:45, 2:00, etc.)
 - **Full Checks**: At :00 minutes (gap filling + data updates)
 - **Quick Checks**: At :15, :30, :45 minutes (data fetch only)
 - **Gap Filling**: Automatically fills missing historical data
 - **Data Updates**: Updates existing records when values change
 ### **Multi-Database Support**
 - **VictoriaMetrics** (Recommended) - High-performance time-series
 - **InfluxDB** - Purpose-built time-series database
 - **PostgreSQL + TimescaleDB** - Relational with time-series optimization
 - **MySQL** - Traditional relational database
 - **SQLite** - Local development and testing
 ### **Production Features**
 - **Docker Support**: Complete containerization
 - **Grafana Integration**: Pre-built dashboards
 - **HTTPS Configuration**: Secure deployment options
 - **Health Monitoring**: Comprehensive logging and error handling
 - **Gap Detection**: Automatic identification of missing data
 - **Retry Logic**: Database lock handling and network error recovery
 ## 🚀 **Quick Start**
 ### **1. Basic Setup (SQLite)**
 ```bash
 cd water_level_monitor
 pip install -r requirements.txt
 python water_scraper_v3.py
 ```
 ### **2. VictoriaMetrics Setup**
 ```bash
 # Start VictoriaMetrics + Grafana
 docker-compose -f docker-compose.victoriametrics.yml up -d
 # Configure environment
 export DB_TYPE=victoriametrics
 export VM_HOST=localhost
 export VM_PORT=8428
 # Run monitor
 python water_scraper_v3.py
 ```
 ### **3. Test Different Databases**
 ```bash
 # Test all supported databases
 python demo_databases.py all
 # Test specific database
 python demo_databases.py victoriametrics
 ```
 ## 📊 **Data Collection**
 ### **Station Coverage**
 - **16 Water Monitoring Stations** across Thailand
 - **Accurate Station Codes**: P.1, P.20, P.21, P.4A, P.5, P.67, P.75, P.76, P.77, P.81, P.82, P.84, P.85, P.87, P.92, P.103
 - **Bilingual Names**: Thai and English station identification
 ### **Metrics Collected**
 - 🌊 **Water Level**: Measured in meters (m)
 - 💧 **Discharge**: Measured in cubic meters per second (cms)
 - 📊 **Discharge Percentage**: Relative to station capacity
 - ⏰ **Timestamp**: Hour 24 handling (midnight = 00:00 next day)
 ### **Data Frequency**
 - **Every 15 Minutes**: Continuous monitoring
 - **~300+ Data Points**: Per collection cycle
 - **Automatic Gap Filling**: Historical data recovery
 - **Data Updates**: Changed values detection and correction
 ## 🔧 **Command Line Tools**
 ### **Main Application**
 ```bash
 python water_scraper_v3.py                    # Run continuous monitoring
 python water_scraper_v3.py --test             # Single test cycle
 python water_scraper_v3.py --help             # Show help
 ```
 ### **Gap Management**
 ```bash
 python water_scraper_v3.py --check-gaps [days]  # Check for missing data
 python water_scraper_v3.py --fill-gaps [days]   # Fill missing data gaps
 python water_scraper_v3.py --update-data [days] # Update existing data
 ```
 ### **Database Testing**
 ```bash
 python demo_databases.py                      # SQLite demo
 python demo_databases.py victoriametrics      # VictoriaMetrics demo
 python demo_databases.py all                  # Test all databases
 ```
 ## 📈 **Monitoring & Visualization**
 ### **Grafana Dashboard**
 - **URL**: http://localhost:3000 (when using docker-compose)
 - **Username**: admin
 - **Password**: admin_password
 - **Features**: Time series charts, status tables, gauges, alerts
 ### **VictoriaMetrics API**
 - **URL**: http://localhost:8428
 - **Health**: http://localhost:8428/health
 - **Metrics**: http://localhost:8428/metrics
 - **Query API**: http://localhost:8428/api/v1/query
 ## 🛡️ **Security & Production**
 ### **HTTPS Configuration**
 - Complete guide in `HTTPS_CONFIGURATION.md`
 - SSL certificate setup
 - Reverse proxy configuration
 - Security best practices
 ### **Deployment Options**
 - **Docker**: Containerized deployment
 - **Systemd**: Linux service configuration
 - **Cloud**: AWS, GCP, Azure deployment guides
 - **Monitoring**: Health checks and alerting
 ## 📚 **Documentation**
 ### **Available Guides**
 1. **README.md** - Main project documentation
 2. **DATABASE_DEPLOYMENT_GUIDE.md** - Multi-database setup
 3. **ENHANCED_SCHEDULER_GUIDE.md** - 15-minute scheduler details
 4. **GAP_FILLING_GUIDE.md** - Data integrity and gap filling
 5. **DEBIAN_TROUBLESHOOTING.md** - Linux deployment troubleshooting
 6. **VICTORIAMETRICS_SETUP.md** - VictoriaMetrics configuration
 7. **HTTPS_CONFIGURATION.md** - Secure deployment setup
 ### **Key Features Documented**
 - ✅ Installation and configuration
 - ✅ Multi-database support
 - ✅ 15-minute scheduling system
 - ✅ Gap filling and data integrity
 - ✅ Production deployment
 - ✅ Monitoring and troubleshooting
 - ✅ Security configuration
 ## 🎯 **Project Status: PRODUCTION READY**
 The Thailand Water Monitor is now:
 - ✅ **Clean**: All old and redundant files removed
 - ✅ **Organized**: Clear project structure with proper documentation
 - ✅ **Enhanced**: 15-minute scheduling with gap filling
 - ✅ **Scalable**: Multi-database support with VictoriaMetrics
 - ✅ **Secure**: HTTPS configuration and security best practices
 - ✅ **Monitored**: Comprehensive logging and Grafana dashboards
 - ✅ **Documented**: Complete guides for all features and deployment options
 The project is ready for production deployment with professional-grade monitoring capabilities.
@@ -1,272 +0,0 @@
 # 🏗️ Project Structure - Northern Thailand Ping River Monitor
 ## 📁 Directory Layout
 ```
 Northern-Thailand-Ping-River-Monitor/
 ├── 📁 src/                          # Main application source code
 │   ├── __init__.py                  # Package initialization
 │   ├── main.py                      # CLI entry point and main application
 │   ├── water_scraper_v3.py          # Core data collection engine
 │   ├── web_api.py                   # FastAPI web interface
 │   ├── config.py                    # Configuration management
 │   ├── database_adapters.py         # Database abstraction layer
 │   ├── models.py                    # Data models and type definitions
 │   ├── exceptions.py                # Custom exception classes
 │   ├── validators.py                # Data validation layer
 │   ├── metrics.py                   # Metrics collection system
 │   ├── health_check.py              # Health monitoring system
 │   ├── rate_limiter.py              # Rate limiting and request tracking
 │   └── logging_config.py            # Enhanced logging configuration
 ├── 📁 docs/                         # Documentation files
 │   ├── STATION_MANAGEMENT_GUIDE.md  # Station management documentation
 │   ├── ENHANCEMENT_SUMMARY.md       # Feature enhancement summary
 │   └── PROJECT_STRUCTURE.md         # This file
 ├── 📁 scripts/                      # Utility scripts
 │   └── migrate_geolocation.py       # Database migration script
 ├── 📁 grafana/                      # Grafana configuration
 │   ├── dashboards/                  # Dashboard definitions
 │   └── provisioning/                # Grafana provisioning config
 ├── 📁 tests/                        # Test files
 │   ├── test_integration.py          # Integration test suite
 │   ├── test_station_management.py   # Station management tests
 │   └── test_api.py                  # API endpoint tests
 ├── 📄 run.py                        # Simple startup script
 ├── 📄 requirements.txt              # Production dependencies
 ├── 📄 requirements-dev.txt          # Development dependencies
 ├── 📄 setup.py                      # Package installation script
 ├── 📄 Dockerfile                    # Docker container definition
 ├── 📄 docker-compose.victoriametrics.yml  # Complete stack deployment
 ├── 📄 Makefile                      # Common development tasks
 ├── 📄 .env.example                  # Environment configuration template
 ├── 📄 .gitignore                    # Git ignore patterns
 ├── 📄 .gitlab-ci.yml                # CI/CD pipeline configuration
 ├── 📄 LICENSE                       # MIT license
 ├── 📄 README.md                     # Main project documentation
 └── 📄 CONTRIBUTING.md               # Contribution guidelines
 ```
 ## 🔧 Core Components
 ### **Application Layer**
 - **`src/main.py`** - Command-line interface and application orchestration
 - **`src/web_api.py`** - FastAPI web interface with REST endpoints
 - **`src/water_scraper_v3.py`** - Core data collection and processing engine
 ### **Data Layer**
 - **`src/database_adapters.py`** - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
 - **`src/models.py`** - Pydantic data models and type definitions
 - **`src/validators.py`** - Data validation and sanitization
 ### **Infrastructure Layer**
 - **`src/config.py`** - Configuration management with environment variable support
 - **`src/logging_config.py`** - Structured logging with rotation and colors
 - **`src/metrics.py`** - Application metrics collection (counters, gauges, histograms)
 - **`src/health_check.py`** - System health monitoring and status checks
 ### **Utility Layer**
 - **`src/exceptions.py`** - Custom exception hierarchy
 - **`src/rate_limiter.py`** - API rate limiting and request tracking
 ## 🌐 Web API Structure
 ### **Endpoints Organization**
 ```
 /                           # Dashboard homepage
 ├── /health                 # System health status
 ├── /metrics               # Application metrics
 ├── /config                # Configuration (masked)
 ├── /stations              # Station management
 │   ├── GET /              # List all stations
 │   ├── POST /             # Create new station
 │   ├── GET /{id}          # Get specific station
 │   ├── PUT /{id}          # Update station
 │   └── DELETE /{id}       # Delete station
 ├── /measurements          # Data access
 │   ├── /latest            # Latest measurements
 │   └── /station/{code}    # Station-specific data
 └── /scraping              # Data collection control
    ├── /trigger           # Manual data collection
    └── /status            # Scraping status
 ```
 ### **API Models**
 - **Request Models**: Station creation/update, query parameters
 - **Response Models**: Station info, measurements, health status
 - **Error Models**: Standardized error responses
 ## 🗄️ Database Architecture
 ### **Supported Databases**
 1. **SQLite** - Local development and testing
 2. **MySQL** - Traditional relational database
 3. **PostgreSQL** - Advanced relational with TimescaleDB support
 4. **InfluxDB** - Purpose-built time-series database
 5. **VictoriaMetrics** - High-performance metrics storage
 ### **Schema Design**
 ```sql
 -- Stations table
 stations (
    id INTEGER PRIMARY KEY,
    station_code VARCHAR(10) UNIQUE,
    thai_name VARCHAR(255),
    english_name VARCHAR(255),
    latitude DECIMAL(10,8),
    longitude DECIMAL(11,8),
    geohash VARCHAR(20),
    status VARCHAR(20),
    created_at TIMESTAMP,
    updated_at TIMESTAMP
 )
 -- Measurements table
 water_measurements (
    id BIGINT PRIMARY KEY,
    timestamp DATETIME,
    station_id INTEGER,
    water_level DECIMAL(10,3),
    discharge DECIMAL(10,2),
    discharge_percent DECIMAL(5,2),
    status VARCHAR(20),
    created_at TIMESTAMP,
    FOREIGN KEY (station_id) REFERENCES stations(id),
    UNIQUE(timestamp, station_id)
 )
 ```
 ## 🐳 Docker Architecture
 ### **Multi-Stage Build**
 1. **Builder Stage** - Compile dependencies and build artifacts
 2. **Production Stage** - Minimal runtime environment
 ### **Service Composition**
 - **ping-river-monitor** - Data collection service
 - **ping-river-api** - Web API service
 - **victoriametrics** - Time-series database
 - **grafana** - Visualization dashboard
 ## 📊 Monitoring Architecture
 ### **Metrics Collection**
 - **Counters** - API requests, database operations, scraping cycles
 - **Gauges** - Current values, connection status, resource usage
 - **Histograms** - Response times, processing durations
 ### **Health Checks**
 - **Database Health** - Connection status, data freshness
 - **API Health** - External API availability, response times
 - **System Health** - Memory usage, disk space, CPU load
 ### **Logging Levels**
 - **DEBUG** - Detailed execution information
 - **INFO** - General operational messages
 - **WARNING** - Potential issues and recoverable errors
 - **ERROR** - Serious problems requiring attention
 - **CRITICAL** - System-threatening issues
 ## 🔧 Configuration Management
 ### **Environment Variables**
 ```bash
 # Database
 DB_TYPE=victoriametrics
 VM_HOST=localhost
 VM_PORT=8428
 # Application
 SCRAPING_INTERVAL_HOURS=1
 LOG_LEVEL=INFO
 DATA_RETENTION_DAYS=365
 # Security
 SECRET_KEY=your-secret-key
 API_KEY=your-api-key
 ```
 ### **Configuration Hierarchy**
 1. Environment variables (highest priority)
 2. .env file
 3. Default values in config.py (lowest priority)
 ## 🧪 Testing Architecture
 ### **Test Categories**
 - **Unit Tests** - Individual component testing
 - **Integration Tests** - System component interaction
 - **API Tests** - Endpoint functionality and responses
 - **Performance Tests** - Load and stress testing
 ### **Test Data**
 - **Mock Data** - Simulated API responses
 - **Test Database** - Isolated test environment
 - **Fixtures** - Reusable test data sets
 ## 📦 Deployment Architecture
 ### **Development**
 ```bash
 python run.py --web-api    # Local development server
 ```
 ### **Production**
 ```bash
 docker-compose up -d       # Full stack deployment
 ```
 ### **CI/CD Pipeline**
 1. **Test Stage** - Run all tests and quality checks
 2. **Build Stage** - Create Docker images
 3. **Deploy Stage** - Deploy to staging/production
 4. **Health Check** - Verify deployment success
 ## 🔒 Security Architecture
 ### **Input Validation**
 - Pydantic models for API requests
 - Data range validation for measurements
 - SQL injection prevention through ORM
 ### **Authentication** (Future)
 - API key authentication
 - JWT token support
 - Role-based access control
 ### **Data Protection**
 - Environment variable configuration
 - Sensitive data masking in logs
 - HTTPS support for production
 ## 📈 Performance Architecture
 ### **Optimization Strategies**
 - Database connection pooling
 - Query optimization and indexing
 - Response caching for static data
 - Async processing for I/O operations
 ### **Scalability Considerations**
 - Horizontal scaling with load balancers
 - Database read replicas
 - Microservice architecture readiness
 - Container orchestration support
 ## 🔄 Data Flow Architecture
 ### **Collection Flow**
 ```
 External API → Rate Limiter → Data Validator → Database Adapter → Database
 ```
 ### **API Flow**
 ```
 HTTP Request → FastAPI → Business Logic → Database Adapter → HTTP Response
 ```
 ### **Monitoring Flow**
 ```
 Application Events → Metrics Collector → Health Checks → Monitoring Dashboard
 ```
 This architecture provides a solid foundation for a production-ready water monitoring system with excellent maintainability, scalability, and observability.
@@ -34,6 +34,20 @@ This document contains important references and external resources related to th
 - **Usage**: Reference for individual station characteristics and historical data patterns
 - **Station**: P.76 - บ้านแม่อีไฮ (Banb Mae I Hai)
 ### **ThaiWater / HII (Hydro-Informatics Institute) Resources**
 #### **4. ThaiWater Portal**
 - **URL**: https://twa.thaiwater.net
 - **Description**: National water situation portal (rainfall, water level, dams, warnings)
 - **Language**: Thai/English
 - **Usage**: Backed by open and auth-gated APIs — full endpoint catalog in [DATA_SOURCES.md](../DATA_SOURCES.md)
 #### **5. ThaiWater Data Standard**
 - **URL**: https://standard.thaiwater.net
 - **Description**: Official water-data standard for exchange and warning — canonical station/basin/province code registries, data formats, warning-level definitions
 - **Language**: Thai
 - **Usage**: Reference for station metadata and warning-level semantics
 ## 📊 **Data Sources and APIs**
 ### **Primary Data Source**
@@ -0,0 +1,723 @@
 [
 {
  "station": "P.1",
  "warn_thr": 3.7,
  "folds": [
   {
    "year": 2021,
    "n_train": 20024,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07375926701460789,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.07332598842242062,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.07383022350644125,
      "mae_above_2p5": null,
      "brier_warn": 1.0850721383440065e-12,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.06977345537342049,
      "mae_above_2p5": null,
      "brier_warn": 7.128994064266462e-16,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28784,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-10-02T19:00:00",
      "peak_ts": "2022-10-03T15:00:00",
      "peak_level": 4.65
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.08452847354970178,
      "mae_above_2p5": 0.25082252888260664,
      "brier_warn": 0.004300908725927739,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8173954245046406
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.08369773912557228,
      "mae_above_2p5": 0.24702357745371217,
      "brier_warn": 0.004325741658698973,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8114190118860223
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.08230165237819607,
      "mae_above_2p5": 0.2599241552494541,
      "brier_warn": 0.004191550822623699,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 3.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.693734826616603
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.08181109208140971,
      "mae_above_2p5": 0.2639358812546371,
      "brier_warn": 0.004581181754314636,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 2.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.620470606696475
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37539,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07696637416933236,
      "mae_above_2p5": 0.10643655855133666,
      "brier_warn": 1.919860722404625e-15,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.07708101918232377,
      "mae_above_2p5": 0.09974894450126857,
      "brier_warn": 1.7028444765622288e-15,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.06968496247786034,
      "mae_above_2p5": 0.10210094973854984,
      "brier_warn": 1.0987601008721297e-09,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.06726603345661021,
      "mae_above_2p5": 0.09716644572204487,
      "brier_warn": 2.7085590803510675e-09,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46323,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2024-09-24T17:00:00",
      "peak_ts": "2024-09-26T02:00:00",
      "peak_level": 4.93
     },
     {
      "crossing": "2024-10-03T09:00:00",
      "peak_ts": "2024-10-05T12:00:00",
      "peak_level": 5.3
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.0890019157543212,
      "mae_above_2p5": 0.2093245927883516,
      "brier_warn": 0.005826169840715695,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 10.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.817519939833057
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.546588884631041
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.08875916089292855,
      "mae_above_2p5": 0.20975114218621593,
      "brier_warn": 0.006061154593275248,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 11.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.800924141216692
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 72.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.577164984770105
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.08341835741803395,
      "mae_above_2p5": 0.18688839374651373,
      "brier_warn": 0.003188229521816099,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 17.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 5.058029430632501
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.3311787370709975
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.08412336465267245,
      "mae_above_2p5": 0.19173218504592401,
      "brier_warn": 0.0034722638888286116,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 15.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 5.0118284217314
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.3520431553190155
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 55083,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-27T18:00:00",
      "peak_ts": "2025-09-27T22:00:00",
      "peak_level": 3.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.11202835461695825,
      "mae_above_2p5": 0.2456782496767171,
      "brier_warn": 0.005178356039745929,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.11083068059655521,
      "mae_above_2p5": 0.23787013498709667,
      "brier_warn": 0.005228043825289021,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.10372526043263834,
      "mae_above_2p5": 0.23846363852091013,
      "brier_warn": 0.005898259026876986,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 1.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_rain_quantile_uw": {
      "mae": 0.10267267834487567,
      "mae_above_2p5": 0.2507951319537275,
      "brier_warn": 0.005020467408392599,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.2417205711942434
       }
      ],
      "false_alarm_episodes": 1
     }
    }
   }
  ]
 },
 {
  "station": "P.103",
  "warn_thr": 5.95,
  "folds": [
   {
    "year": 2021,
    "n_train": 20009,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.14438683486790602,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.14305613024043953,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.13512418754716052,
      "mae_above_2p5": null,
      "brier_warn": 6.937198832251013e-10,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.10270057685061172,
      "mae_above_2p5": null,
      "brier_warn": 6.565945930999852e-14,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28769,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-08-14T04:00:00",
      "peak_ts": "2022-08-14T08:00:00",
      "peak_level": 6.09
     },
     {
      "crossing": "2022-10-02T16:00:00",
      "peak_ts": "2022-10-03T16:00:00",
      "peak_level": 7.54
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.14392334606606189,
      "mae_above_2p5": 0.41075382386412596,
      "brier_warn": 0.00764679988213082,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.825270553204425
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194064117976157
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.14284722696431765,
      "mae_above_2p5": 0.39401132538096667,
      "brier_warn": 0.007427375799450148,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.845896993132048
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194661123502819
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.1458954690776336,
      "mae_above_2p5": 0.4674051188369517,
      "brier_warn": 0.00899991317044724,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 1.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.82714042795344
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 5.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 6.59871451008115
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.13203958422483053,
      "mae_above_2p5": 0.4902773906613983,
      "brier_warn": 0.008514527559601331,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 4.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.895369771408423
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 3.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 6.294082735853337
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37524,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.16195301512514512,
      "mae_above_2p5": null,
      "brier_warn": 3.0208441147560167e-07,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.16334224973870387,
      "mae_above_2p5": null,
      "brier_warn": 1.1626187140381066e-08,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.14755024879522577,
      "mae_above_2p5": null,
      "brier_warn": 3.850104714388072e-05,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.10589805327026759,
      "mae_above_2p5": null,
      "brier_warn": 1.8896757611081001e-07,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46308,
    "n_test": 4058,
    "events": [
     {
      "crossing": "2024-09-24T10:00:00",
      "peak_ts": "2024-09-26T00:00:00",
      "peak_level": 8.27
     },
     {
      "crossing": "2024-09-30T03:00:00",
      "peak_ts": "2024-09-30T06:00:00",
      "peak_level": 5.99
     },
     {
      "crossing": "2024-10-03T06:00:00",
      "peak_ts": "2024-10-05T07:00:00",
      "peak_level": 9.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.13901842325128816,
      "mae_above_2p5": 0.2887416310966684,
      "brier_warn": 0.0072040857753137046,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.212734363860193
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.47565489914091
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.780278464915938
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_fc48": {
      "mae": 0.13850733053801131,
      "mae_above_2p5": 0.29817392926900865,
      "brier_warn": 0.007928846574280278,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.166020251020889
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 10.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.524584037259288
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.595120917488185
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile": {
      "mae": 0.12640546558686208,
      "mae_above_2p5": 0.27944115421093924,
      "brier_warn": 0.007730268539879654,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 20.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.336487732683672
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 4.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.45284915024346
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.575915226697406
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_quantile_uw": {
      "mae": 0.12480975852168202,
      "mae_above_2p5": 0.2857575557415695,
      "brier_warn": 0.005894928998647136,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 21.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.444720326882825
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.42
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 32.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.781001847167515
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 54577,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-26T06:00:00",
      "peak_ts": "2025-09-27T21:00:00",
      "peak_level": 6.64
     },
     {
      "crossing": "2025-10-03T06:00:00",
      "peak_ts": "2025-10-03T12:00:00",
      "peak_level": 6.14
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.17206685418009837,
      "mae_above_2p5": 0.3840520085986099,
      "brier_warn": 0.015824852891785323,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73043665401637
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.609919760117381
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_fc48": {
      "mae": 0.1722274451362234,
      "mae_above_2p5": 0.3872216974630654,
      "brier_warn": 0.016245727130063177,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 4.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.7312263707793685
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.685511824758637
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_quantile": {
      "mae": 0.15912275848686555,
      "mae_above_2p5": 0.3845695612674703,
      "brier_warn": 0.014797606329811499,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 11.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 10.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.58133012298031
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_quantile_uw": {
      "mae": 0.15385355845103685,
      "mae_above_2p5": 0.41314645854725585,
      "brier_warn": 0.016528116336109958,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 9.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.741254234340573
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.418105405306207
       }
      ],
      "false_alarm_episodes": 2
     }
    }
   }
  ]
 }
 ]
@@ -0,0 +1,439 @@
 [
 {
  "station": "P.1",
  "warn_thr": 3.7,
  "folds": [
   {
    "year": 2021,
    "n_train": 20024,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07375926701460789,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.07375926701460789,
      "mae_above_2p5": null,
      "brier_warn": 7.852802408474157e-14,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28784,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-10-02T19:00:00",
      "peak_ts": "2022-10-03T15:00:00",
      "peak_level": 4.65
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.08452847354970178,
      "mae_above_2p5": 0.25082252888260664,
      "brier_warn": 0.004300908725927739,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8173954245046406
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.08452847354970178,
      "mae_above_2p5": 0.25082252888260664,
      "brier_warn": 0.0039815091186836665,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.8173954245046406
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37539,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.07696637416933236,
      "mae_above_2p5": 0.10643655855133666,
      "brier_warn": 1.919860722404625e-15,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.07696637416933236,
      "mae_above_2p5": 0.10643655855133666,
      "brier_warn": 4.0844243581898366e-08,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46323,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2024-09-24T17:00:00",
      "peak_ts": "2024-09-26T02:00:00",
      "peak_level": 4.93
     },
     {
      "crossing": "2024-10-03T09:00:00",
      "peak_ts": "2024-10-05T12:00:00",
      "peak_level": 5.3
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.0890019157543212,
      "mae_above_2p5": 0.2093245927883516,
      "brier_warn": 0.005826169840715695,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 10.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.817519939833057
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.546588884631041
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.0890019157543212,
      "mae_above_2p5": 0.2093245927883516,
      "brier_warn": 0.005475787026695418,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 10.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.817519939833057
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.546588884631041
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 55083,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-27T18:00:00",
      "peak_ts": "2025-09-27T22:00:00",
      "peak_level": 3.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.11202835461695825,
      "mae_above_2p5": 0.2456782496767171,
      "brier_warn": 0.005178356039745929,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.11202835461695825,
      "mae_above_2p5": 0.2456782496767171,
      "brier_warn": 0.00491346243876589,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   }
  ]
 },
 {
  "station": "P.103",
  "warn_thr": 5.95,
  "folds": [
   {
    "year": 2021,
    "n_train": 20009,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.14438683486790602,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.14438683486790602,
      "mae_above_2p5": null,
      "brier_warn": 3.5349670949872053e-13,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28769,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-08-14T04:00:00",
      "peak_ts": "2022-08-14T08:00:00",
      "peak_level": 6.09
     },
     {
      "crossing": "2022-10-02T16:00:00",
      "peak_ts": "2022-10-03T16:00:00",
      "peak_level": 7.54
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.14392334606606189,
      "mae_above_2p5": 0.41075382386412596,
      "brier_warn": 0.00764679988213082,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.825270553204425
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194064117976157
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.14392334606606189,
      "mae_above_2p5": 0.41075382386412596,
      "brier_warn": 0.006975493312169236,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.825270553204425
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.194064117976157
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37524,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise_rain": {
      "mae": 0.16195301512514512,
      "mae_above_2p5": null,
      "brier_warn": 3.0208441147560167e-07,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.16195301512514512,
      "mae_above_2p5": null,
      "brier_warn": 3.2176039819636275e-05,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46308,
    "n_test": 4058,
    "events": [
     {
      "crossing": "2024-09-24T10:00:00",
      "peak_ts": "2024-09-26T00:00:00",
      "peak_level": 8.27
     },
     {
      "crossing": "2024-09-30T03:00:00",
      "peak_ts": "2024-09-30T06:00:00",
      "peak_level": 5.99
     },
     {
      "crossing": "2024-10-03T06:00:00",
      "peak_ts": "2024-10-05T07:00:00",
      "peak_level": 9.93
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.13901842325128816,
      "mae_above_2p5": 0.2887416310966684,
      "brier_warn": 0.0072040857753137046,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.212734363860193
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.47565489914091
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.780278464915938
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain_qsigma": {
      "mae": 0.13901842325128816,
      "mae_above_2p5": 0.2887416310966684,
      "brier_warn": 0.006910847607098417,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.212734363860193
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 9.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.47565489914091
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.780278464915938
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 54577,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-26T06:00:00",
      "peak_ts": "2025-09-27T21:00:00",
      "peak_level": 6.64
     },
     {
      "crossing": "2025-10-03T06:00:00",
      "peak_ts": "2025-10-03T12:00:00",
      "peak_level": 6.14
     }
    ],
    "variants": {
     "rise_rain": {
      "mae": 0.17206685418009837,
      "mae_above_2p5": 0.3840520085986099,
      "brier_warn": 0.015824852891785323,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73043665401637
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.609919760117381
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise_rain_qsigma": {
      "mae": 0.17206685418009837,
      "mae_above_2p5": 0.3840520085986099,
      "brier_warn": 0.015889933586185904,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73043665401637
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.609919760117381
       }
      ],
      "false_alarm_episodes": 2
     }
    }
   }
  ]
 }
 ]
@@ -0,0 +1,439 @@
 [
 {
  "station": "P.1",
  "warn_thr": 3.7,
  "folds": [
   {
    "year": 2021,
    "n_train": 20024,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise": {
      "mae": 0.070970681677648,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.07247641391827915,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28782,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-10-02T19:00:00",
      "peak_ts": "2022-10-03T15:00:00",
      "peak_level": 4.65
     }
    ],
    "variants": {
     "rise": {
      "mae": 0.08489734638010188,
      "mae_above_2p5": 0.24517428534843191,
      "brier_warn": 0.004136576477574926,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 6.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.870617057762467
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.08491621780630447,
      "mae_above_2p5": 0.24539225527181602,
      "brier_warn": 0.004209435091817237,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 5.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.844081593978701
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37537,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise": {
      "mae": 0.07647507344443248,
      "mae_above_2p5": 0.12593127745781565,
      "brier_warn": 3.19529506149809e-11,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.07491688056430452,
      "mae_above_2p5": 0.09506457784884237,
      "brier_warn": 6.639993512164624e-14,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46321,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2024-09-24T17:00:00",
      "peak_ts": "2024-09-26T02:00:00",
      "peak_level": 4.93
     },
     {
      "crossing": "2024-10-03T09:00:00",
      "peak_ts": "2024-10-05T12:00:00",
      "peak_level": 5.3
     }
    ],
    "variants": {
     "rise": {
      "mae": 0.10075626719251345,
      "mae_above_2p5": 0.26384939692286363,
      "brier_warn": 0.011608221543810462,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 6.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.534709676862131
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 3.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.056619694027486
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.08874847709708966,
      "mae_above_2p5": 0.20967242138010514,
      "brier_warn": 0.005862181747890141,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 11.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.863137825109792
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 21.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.557547530221961
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 55081,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-27T18:00:00",
      "peak_ts": "2025-09-27T22:00:00",
      "peak_level": 3.93
     }
    ],
    "variants": {
     "rise": {
      "mae": 0.13102686839694494,
      "mae_above_2p5": 0.2729944665011028,
      "brier_warn": 0.010889835530782944,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 46.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.232029710676319
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_rain": {
      "mae": 0.11160288394028639,
      "mae_above_2p5": 0.24838881498904114,
      "brier_warn": 0.005150122823202273,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 2.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.239754736790134
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   }
  ]
 },
 {
  "station": "P.103",
  "warn_thr": 5.95,
  "folds": [
   {
    "year": 2021,
    "n_train": 20009,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise": {
      "mae": 0.15279228710793116,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.15114584578006643,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28769,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-08-14T04:00:00",
      "peak_ts": "2022-08-14T08:00:00",
      "peak_level": 6.09
     },
     {
      "crossing": "2022-10-02T16:00:00",
      "peak_ts": "2022-10-03T16:00:00",
      "peak_level": 7.54
     }
    ],
    "variants": {
     "rise": {
      "mae": 0.15389446229226025,
      "mae_above_2p5": 0.41887094569113953,
      "brier_warn": 0.007740077230532647,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.837975953559253
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.168225721504108
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.15303667736579823,
      "mae_above_2p5": 0.38833713105646916,
      "brier_warn": 0.0074511589478895475,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.8804746828604815
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 10.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.226685294045882
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37524,
    "n_test": 4392,
    "events": [],
    "variants": {
     "rise": {
      "mae": 0.16106550376855028,
      "mae_above_2p5": null,
      "brier_warn": 0.0001779564366636896,
      "events": [],
      "false_alarm_episodes": 1
     },
     "rise_rain": {
      "mae": 0.15541158498896382,
      "mae_above_2p5": null,
      "brier_warn": 4.103077380009017e-09,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46308,
    "n_test": 4058,
    "events": [
     {
      "crossing": "2024-09-24T10:00:00",
      "peak_ts": "2024-09-26T00:00:00",
      "peak_level": 8.27
     },
     {
      "crossing": "2024-09-30T03:00:00",
      "peak_ts": "2024-09-30T06:00:00",
      "peak_level": 5.99
     },
     {
      "crossing": "2024-10-03T06:00:00",
      "peak_ts": "2024-10-05T07:00:00",
      "peak_level": 9.93
     }
    ],
    "variants": {
     "rise": {
      "mae": 0.17314177863843333,
      "mae_above_2p5": 0.48294427141283425,
      "brier_warn": 0.016091510788709233,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 10.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 7.167139790234238
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 10.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.618725525893519
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 7.889819144742388
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_rain": {
      "mae": 0.1400555603272253,
      "mae_above_2p5": 0.2920105854094151,
      "brier_warn": 0.007294262727198402,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 19.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 8.04142923647258
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 10.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.42
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 55.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 8.655375706947945
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 54577,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-26T06:00:00",
      "peak_ts": "2025-09-27T21:00:00",
      "peak_level": 6.64
     },
     {
      "crossing": "2025-10-03T06:00:00",
      "peak_ts": "2025-10-03T12:00:00",
      "peak_level": 6.14
     }
    ],
    "variants": {
     "rise": {
      "mae": 0.21617454799997243,
      "mae_above_2p5": 0.406245211140301,
      "brier_warn": 0.012587550711857222,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 14.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 47.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.953951787788988
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_rain": {
      "mae": 0.17651730883692462,
      "mae_above_2p5": 0.38926658786456825,
      "brier_warn": 0.015873258461498164,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 6.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.760150202082536
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 8.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.605274163405912
       }
      ],
      "false_alarm_episodes": 2
     }
    }
   }
  ]
 }
 ]
@@ -0,0 +1,723 @@
 [
 {
  "station": "P.1",
  "warn_thr": 3.7,
  "folds": [
   {
    "year": 2021,
    "n_train": 20024,
    "n_test": 4392,
    "events": [],
    "variants": {
     "baseline_abs": {
      "mae": 0.073069548799388,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.070970681677648,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.07394501467643348,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.07289422294875283,
      "mae_above_2p5": null,
      "brier_warn": 6.870660393452143e-17,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28782,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-10-02T19:00:00",
      "peak_ts": "2022-10-03T15:00:00",
      "peak_level": 4.65
     }
    ],
    "variants": {
     "baseline_abs": {
      "mae": 0.09128484356553154,
      "mae_above_2p5": 0.2856025611084259,
      "brier_warn": 0.005440790049399641,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 0.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.4
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.08489734638010188,
      "mae_above_2p5": 0.24517428534843191,
      "brier_warn": 0.004136576477574926,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 6.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.870617057762467
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.08451609397908402,
      "mae_above_2p5": 0.24677443475763222,
      "brier_warn": 0.0041231071254735656,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 6.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.9060484870307466
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.08412283956084679,
      "mae_above_2p5": 0.2653552568120815,
      "brier_warn": 0.004349073127074364,
      "events": [
       {
        "crossing": "2022-10-02T19:00:00",
        "lead_h": 3.0,
        "peak_level": 4.65,
        "peak_pred_24h_before": 3.641048749261635
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37537,
    "n_test": 4392,
    "events": [],
    "variants": {
     "baseline_abs": {
      "mae": 0.08161389876263064,
      "mae_above_2p5": 0.1335546690803318,
      "brier_warn": 3.284466170848282e-09,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.07647507344443248,
      "mae_above_2p5": 0.12593127745781565,
      "brier_warn": 3.19529506149809e-11,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.07633405851952908,
      "mae_above_2p5": 0.12234064428755373,
      "brier_warn": 9.093694748477163e-10,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.07460579730428984,
      "mae_above_2p5": 0.11243109915383911,
      "brier_warn": 8.148092046865233e-08,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46321,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2024-09-24T17:00:00",
      "peak_ts": "2024-09-26T02:00:00",
      "peak_level": 4.93
     },
     {
      "crossing": "2024-10-03T09:00:00",
      "peak_ts": "2024-10-05T12:00:00",
      "peak_level": 5.3
     }
    ],
    "variants": {
     "baseline_abs": {
      "mae": 0.11011679980584997,
      "mae_above_2p5": 0.28849258735632527,
      "brier_warn": 0.010963910189916666,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 0.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.39
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 0.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 4.9
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.10075626719251345,
      "mae_above_2p5": 0.26384939692286363,
      "brier_warn": 0.011608221543810462,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 6.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.534709676862131
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 3.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.056619694027486
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.10644718990351541,
      "mae_above_2p5": 0.2848978622398853,
      "brier_warn": 0.013872775238121434,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 9.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.502016075573465
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 72.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 5.123356409745132
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.09617759284936986,
      "mae_above_2p5": 0.271753261978628,
      "brier_warn": 0.011660822672241142,
      "events": [
       {
        "crossing": "2024-09-24T17:00:00",
        "lead_h": 4.0,
        "peak_level": 4.93,
        "peak_pred_24h_before": 4.532042588397363
       },
       {
        "crossing": "2024-10-03T09:00:00",
        "lead_h": 2.0,
        "peak_level": 5.3,
        "peak_pred_24h_before": 4.956335200519836
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 55081,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-27T18:00:00",
      "peak_ts": "2025-09-27T22:00:00",
      "peak_level": 3.93
     }
    ],
    "variants": {
     "baseline_abs": {
      "mae": 0.15985415338980086,
      "mae_above_2p5": 0.277881996645181,
      "brier_warn": 0.009824391159106445,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 0.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.2561560254201525
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise": {
      "mae": 0.13102686839694494,
      "mae_above_2p5": 0.2729944665011028,
      "brier_warn": 0.010889835530782944,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 46.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.232029710676319
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_weighted": {
      "mae": 0.13121120517634968,
      "mae_above_2p5": 0.27796411411759725,
      "brier_warn": 0.009393099541680463,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 46.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 3
     },
     "rise_quantile": {
      "mae": 0.11678707793162903,
      "mae_above_2p5": 0.28528124941530186,
      "brier_warn": 0.013009605253899563,
      "events": [
       {
        "crossing": "2025-09-27T18:00:00",
        "lead_h": 45.0,
        "peak_level": 3.93,
        "peak_pred_24h_before": 3.23
       }
      ],
      "false_alarm_episodes": 3
     }
    }
   }
  ]
 },
 {
  "station": "P.103",
  "warn_thr": 5.95,
  "folds": [
   {
    "year": 2021,
    "n_train": 20009,
    "n_test": 4392,
    "events": [],
    "variants": {
     "baseline_abs": {
      "mae": 0.1517830652111977,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.15279228710793116,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.17632375724441426,
      "mae_above_2p5": null,
      "brier_warn": 0.0,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.13593873552532,
      "mae_above_2p5": null,
      "brier_warn": 1.6481640122270367e-09,
      "events": [],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2022,
    "n_train": 28769,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2022-08-14T04:00:00",
      "peak_ts": "2022-08-14T08:00:00",
      "peak_level": 6.09
     },
     {
      "crossing": "2022-10-02T16:00:00",
      "peak_ts": "2022-10-03T16:00:00",
      "peak_level": 7.54
     }
    ],
    "variants": {
     "baseline_abs": {
      "mae": 0.15960776318568,
      "mae_above_2p5": 0.5240325509357986,
      "brier_warn": 0.010134410144044625,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 0.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 5.01930531142127
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 0.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 6.01
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.15389446229226025,
      "mae_above_2p5": 0.41887094569113953,
      "brier_warn": 0.007740077230532647,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.837975953559253
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 9.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.168225721504108
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.1498671765711928,
      "mae_above_2p5": 0.40088103598066704,
      "brier_warn": 0.00801963186023152,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 6.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.81
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 8.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 7.002000385945215
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.13833442255794418,
      "mae_above_2p5": 0.4529809369770209,
      "brier_warn": 0.009090606171021184,
      "events": [
       {
        "crossing": "2022-08-14T04:00:00",
        "lead_h": 1.0,
        "peak_level": 6.09,
        "peak_pred_24h_before": 4.8698313890426705
       },
       {
        "crossing": "2022-10-02T16:00:00",
        "lead_h": 5.0,
        "peak_level": 7.54,
        "peak_pred_24h_before": 6.650326949981244
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2023,
    "n_train": 37524,
    "n_test": 4392,
    "events": [],
    "variants": {
     "baseline_abs": {
      "mae": 0.1489843074454942,
      "mae_above_2p5": null,
      "brier_warn": 1.882373500761472e-08,
      "events": [],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.16106550376855028,
      "mae_above_2p5": null,
      "brier_warn": 0.0001779564366636896,
      "events": [],
      "false_alarm_episodes": 1
     },
     "rise_weighted": {
      "mae": 0.1688589840007776,
      "mae_above_2p5": null,
      "brier_warn": 0.0012753245446952602,
      "events": [],
      "false_alarm_episodes": 1
     },
     "rise_quantile": {
      "mae": 0.1510624979748974,
      "mae_above_2p5": null,
      "brier_warn": 0.0005959001115900152,
      "events": [],
      "false_alarm_episodes": 1
     }
    }
   },
   {
    "year": 2024,
    "n_train": 46308,
    "n_test": 4058,
    "events": [
     {
      "crossing": "2024-09-24T10:00:00",
      "peak_ts": "2024-09-26T00:00:00",
      "peak_level": 8.27
     },
     {
      "crossing": "2024-09-30T03:00:00",
      "peak_ts": "2024-09-30T06:00:00",
      "peak_level": 5.99
     },
     {
      "crossing": "2024-10-03T06:00:00",
      "peak_ts": "2024-10-05T07:00:00",
      "peak_level": 9.93
     }
    ],
    "variants": {
     "baseline_abs": {
      "mae": 0.193584781859895,
      "mae_above_2p5": 0.5381754200484403,
      "brier_warn": 0.018292133172974026,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 5.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 7.11
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 0.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.42
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 0.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 7.86
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise": {
      "mae": 0.17314177863843333,
      "mae_above_2p5": 0.48294427141283425,
      "brier_warn": 0.016091510788709233,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 10.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 7.167139790234238
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 10.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.618725525893519
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 7.889819144742388
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_weighted": {
      "mae": 0.18833560689452924,
      "mae_above_2p5": 0.4948752445774323,
      "brier_warn": 0.01622076553986666,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 11.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 7.283961881370446
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 10.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.783590096006894
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 69.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 7.922098265471745
       }
      ],
      "false_alarm_episodes": 0
     },
     "rise_quantile": {
      "mae": 0.1644888402154109,
      "mae_above_2p5": 0.46461119464587947,
      "brier_warn": 0.013172046828124688,
      "events": [
       {
        "crossing": "2024-09-24T10:00:00",
        "lead_h": 11.0,
        "peak_level": 8.27,
        "peak_pred_24h_before": 7.348489352370365
       },
       {
        "crossing": "2024-09-30T03:00:00",
        "lead_h": 8.0,
        "peak_level": 5.99,
        "peak_pred_24h_before": 5.42
       },
       {
        "crossing": "2024-10-03T06:00:00",
        "lead_h": 2.0,
        "peak_level": 9.93,
        "peak_pred_24h_before": 7.91343423984052
       }
      ],
      "false_alarm_episodes": 0
     }
    }
   },
   {
    "year": 2025,
    "n_train": 54577,
    "n_test": 4392,
    "events": [
     {
      "crossing": "2025-09-26T06:00:00",
      "peak_ts": "2025-09-27T21:00:00",
      "peak_level": 6.64
     },
     {
      "crossing": "2025-10-03T06:00:00",
      "peak_ts": "2025-10-03T12:00:00",
      "peak_level": 6.14
     }
    ],
    "variants": {
     "baseline_abs": {
      "mae": 0.2036944006891808,
      "mae_above_2p5": 0.40787978637525985,
      "brier_warn": 0.014130122065259989,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 14.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 41.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.703778873833753
       }
      ],
      "false_alarm_episodes": 2
     },
     "rise": {
      "mae": 0.21617454799997243,
      "mae_above_2p5": 0.406245211140301,
      "brier_warn": 0.012587550711857222,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 14.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 47.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.953951787788988
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_weighted": {
      "mae": 0.21768864574559124,
      "mae_above_2p5": 0.41071503357135775,
      "brier_warn": 0.014605838473436519,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 13.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 49.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 5.606011827978066
       }
      ],
      "false_alarm_episodes": 1
     },
     "rise_quantile": {
      "mae": 0.18196250028633115,
      "mae_above_2p5": 0.3987262421172889,
      "brier_warn": 0.011560937993250782,
      "events": [
       {
        "crossing": "2025-09-26T06:00:00",
        "lead_h": 13.0,
        "peak_level": 6.64,
        "peak_pred_24h_before": 5.73
       },
       {
        "crossing": "2025-10-03T06:00:00",
        "lead_h": 42.0,
        "peak_level": 6.14,
        "peak_pred_24h_before": 6.00597561680636
       }
      ],
      "false_alarm_episodes": 1
     }
    }
   }
  ]
 }
 ]
@@ -1,38 +0,0 @@
 # -*- mode: python ; coding: utf-8 -*-
 a = Analysis(
    ['run.py'],
    pathex=[],
    binaries=[],
    datas=[('.env', '.'), ('sql', 'sql'), ('README.md', '.'), ('POSTGRESQL_SETUP.md', '.'), ('SQLITE_MIGRATION.md', '.')],
    hiddenimports=['psycopg2', 'sqlalchemy.dialects.postgresql', 'sqlalchemy.dialects.sqlite', 'dotenv', 'pydantic', 'fastapi', 'uvicorn', 'schedule', 'pandas'],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
    optimize=0,
 )
 pyz = PYZ(a.pure)
 exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.datas,
    [],
    name='ping-river-monitor',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
 )
@@ -34,23 +34,23 @@ classifiers = [
    "Environment :: Web Environment",
    "Framework :: FastAPI"
 ]
-requires-python = ">=3.11"
+requires-python = ">=3.11,<3.12"
 dependencies = [
    # Core dependencies
-    "requests==2.31.0",
+    "requests==2.34.2",
    "schedule==1.2.0",
    "pandas==2.0.3",
    "numpy>=1.24,<2",
    # Flood forecasting (ML)
    "scikit-learn==1.9.0",
    # Web API framework
-    "fastapi==0.104.1",
+    "fastapi==0.141.1",
-    "uvicorn[standard]==0.24.0",
+    "uvicorn[standard]==0.52.4",
-    "pydantic==2.5.0",
+    "pydantic==2.13.5",
    # Database adapters
    "sqlalchemy==2.0.23",
    "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
    "psycopg2-binary==2.9.9",
    # Monitoring and metrics
    "psutil==5.9.6"
@@ -59,11 +59,11 @@ dependencies = [
 [project.optional-dependencies]
 dev = [
    # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
    "pytest-cov==4.1.0",
    "pytest-asyncio==0.21.1",
    # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
    "flake8==6.1.0",
    "isort==5.12.0",
    "mypy==1.7.1",
@@ -73,7 +73,7 @@ dev = [
    "ipython==8.17.2",
    "jupyter==1.0.0",
    # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
    "types-python-dateutil==2.8.19.14"
 ]
 docs = [
@@ -83,7 +83,7 @@ docs = [
 ]
 all = [
    "influxdb==5.3.1",
-    "pymysql==1.1.0",
+    "pymysql==1.2.0",
    "psycopg2-binary==2.9.9"
 ]
@@ -100,11 +100,11 @@ Documentation = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/
 [dependency-groups]
 dev = [
    # Testing
-    "pytest==7.4.3",
+    "pytest==9.1.1",
    "pytest-cov==4.1.0",
    "pytest-asyncio==0.21.1",
    # Code formatting and linting
-    "black==23.11.0",
+    "black==26.5.1",
    "flake8==6.1.0",
    "isort==5.12.0",
    "mypy==1.7.1",
@@ -114,7 +114,7 @@ dev = [
    "ipython==8.17.2",
    "jupyter==1.0.0",
    # Type stubs
-    "types-requests==2.31.0.10",
+    "types-requests==2.33.0.20260906",
    "types-python-dateutil==2.8.19.14",
    # Documentation
    "sphinx==7.2.6",
@@ -128,3 +128,16 @@ where = ["src"]
 [tool.setuptools.package-dir]
 "" = "src"
 # One formatting contract for CI, pre-commit and editors. Black's default 88
 # columns; isort in black-compatible mode. Run `make format` before committing.
 [tool.black]
 line-length = 88
 target-version = ["py311"]
 extend-exclude = '/(\.venv|venv|models|\.claude-flow|\.swarm)/'
 [tool.isort]
 profile = "black"
 line_length = 88
 known_first_party = ["src"]
 skip_gitignore = true
@@ -2,12 +2,12 @@
 -r requirements.txt
 # Testing
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
 pytest-asyncio==0.21.1
 # Code formatting and linting
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 isort==5.12.0
 mypy==1.7.1
@@ -25,5 +25,5 @@ ipython==8.17.2
 jupyter==1.0.0
 # Type stubs
-types-requests==2.31.0.10
+types-requests==2.33.0.20260906
 types-python-dateutil==2.8.19.14
@@ -1,5 +1,5 @@
 # Core dependencies
-requests==2.31.0
+requests==2.34.2
 schedule==1.2.0
 pandas==2.0.3
 numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
@@ -8,23 +8,23 @@ numpy>=1.24,<2  # pandas 2.0.3 wheels are ABI-incompatible with numpy 2.x
 scikit-learn==1.9.0
 # Web API framework
-fastapi==0.104.1
+fastapi==0.141.1
-uvicorn[standard]==0.24.0
+uvicorn[standard]==0.52.4
-pydantic==2.5.0
+pydantic==2.13.5
 # Database adapters
 sqlalchemy==2.0.23
 influxdb==5.3.1
-pymysql==1.1.0
+pymysql==1.2.0
 psycopg2-binary==2.9.9
 # Monitoring and metrics
 psutil==5.9.6
 # Development dependencies (optional)
-pytest==7.4.3
+pytest==9.1.1
 pytest-cov==4.1.0
-black==23.11.0
+black==26.5.1
 flake8==6.1.0
 mypy==1.7.1
 pre-commit==3.5.0
@@ -0,0 +1,18 @@
 #!/usr/bin/env python3
 """CLI entry point for backfilling hii_waterlevel from the HII archive.
 Usage:
    python scripts/backfill_hii_waterlevel.py                      # key stations, 2019..today
    python scripts/backfill_hii_waterlevel.py --stations P.1,P.67
    python scripts/backfill_hii_waterlevel.py --start 2024-09-01 --end 2024-11-01 --all
 """
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
 from src.hii_backfill import main
 if __name__ == "__main__":
    sys.exit(0 if main() else 1)
@@ -0,0 +1,48 @@
 #!/usr/bin/env python3
 """Backfill the openmeteo_rain table with the full 2021+ catchment history.
 Usage:
    uv run scripts/backfill_rain_db.py                 # DB from Config/.env
    uv run scripts/backfill_rain_db.py --db-url postgresql://...
 """
 import argparse
 import logging
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
 from src.config import Config
 from src.ml.rain import backfill_db
 def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--db-url", default=None)
    args = parser.parse_args(argv)
    logging.basicConfig(
        level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
    )
    if args.db_url:
        # postgresql+psycopg2://... -> postgresql (driver suffix is not a dialect)
        connection_string = args.db_url
        db_type = args.db_url.split(":", 1)[0].split("+", 1)[0]
    else:
        cfg = Config.get_database_config()
        if cfg["type"] not in ("sqlite", "postgresql", "mysql"):
            print(f"requires a SQL database, got {cfg['type']}", file=sys.stderr)
            return 1
        connection_string, db_type = cfg["connection_string"], cfg["type"]
    from sqlalchemy import create_engine
    engine = create_engine(connection_string, pool_pre_ping=True)
    saved = backfill_db(engine, db_type)
    print(f"backfilled {saved} hourly rows into openmeteo_rain")
    return 0 if saved else 1
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,126 @@
 #!/usr/bin/env python3
 """Backfill rid_reservoir_daily with RID large-dam history (Mae Ngat et al.).
 Two paths, both idempotent and both skipping what is already stored, so a
 rerun repairs holes left by transient failures and is safe alongside the
 hourly live collector:
  --dam-id (default: Mae Ngat)  one dam, whole range, via api/dam — a handful
                                of requests for the entire 2009-today archive
  --all-dams                    all ~35 dams, one request per calendar day via
                                api/dams — thousands of requests, ~25 minutes
 Usage:
    uv run scripts/backfill_rid_reservoir.py                     # Mae Ngat since 2018-08-01
    uv run scripts/backfill_rid_reservoir.py --start 2009-01-01  # full archive
    uv run scripts/backfill_rid_reservoir.py --refresh           # rewrite stored days too
    uv run scripts/backfill_rid_reservoir.py --all-dams --start 2015-01-01
    uv run scripts/backfill_rid_reservoir.py --db-url postgresql://...
 """
 import argparse
 import datetime
 import logging
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
 from src.config import Config
 from src.rid_reservoir import (
    MAE_NGAT_DAM_ID,
    RidReservoirStore,
    backfill,
    backfill_dam,
 )
 DEFAULT_START = datetime.date(2018, 8, 1)  # start of the water_measurements grid
 def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument(
        "--start", type=datetime.date.fromisoformat, default=DEFAULT_START
    )
    parser.add_argument("--end", type=datetime.date.fromisoformat, default=None)
    parser.add_argument("--db-url", default=None)
    parser.add_argument("--throttle", type=float, default=0.4)
    parser.add_argument(
        "--dam-id",
        default=MAE_NGAT_DAM_ID,
        help="dam to backfill via the fast range endpoint (default Mae Ngat)",
    )
    parser.add_argument(
        "--all-dams",
        action="store_true",
        help="every dam, one request per calendar day (slow full-fleet path)",
    )
    parser.add_argument(
        "--chunk-days",
        type=int,
        default=1830,
        help="days per range request; the endpoint imposes no limit of its own",
    )
    parser.add_argument(
        "--refresh",
        action="store_true",
        help="re-fetch days already stored (adds level_msl to api/dams rows)",
    )
    args = parser.parse_args(argv)
    logging.basicConfig(
        level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
    )
    if args.db_url:
        # postgresql+psycopg2://... -> postgresql (driver suffix is not a dialect)
        connection_string = args.db_url
        db_type = args.db_url.split(":", 1)[0].split("+", 1)[0]
    else:
        cfg = Config.get_database_config()
        if cfg["type"] not in ("sqlite", "postgresql", "mysql"):
            print(f"requires a SQL database, got {cfg['type']}", file=sys.stderr)
            return 1
        connection_string, db_type = cfg["connection_string"], cfg["type"]
    store = RidReservoirStore(connection_string, db_type)
    if not store.connect():
        print(
            "database connection failed — check the connection string",
            file=sys.stderr,
        )
        return 1
    end = args.end or datetime.date.today()
    span_days = (end - args.start).days + 1
    dam_id = None if args.all_dams else args.dam_id
    missing = span_days - len(store.present_dates(args.start, end, dam_id=dam_id))
    if args.all_dams:
        saved = backfill(store, args.start, end, throttle_seconds=args.throttle)
        print(f"backfilled {saved} dam-day rows ({missing} days were missing)")
        return 0 if saved or missing == 0 else 1
    stats = {}
    saved = backfill_dam(
        store,
        dam_id=args.dam_id,
        start=args.start,
        end=end,
        chunk_days=args.chunk_days,
        throttle_seconds=max(args.throttle, 1.0),
        skip_present=not args.refresh,
        stats=stats,
    )
    still_missing = span_days - len(
        store.present_dates(args.start, end, dam_id=args.dam_id)
    )
    print(
        f"backfilled {saved} dam-day rows for dam {args.dam_id} "
        f"(requests: {stats.get('requests', 0)}, {missing} days were missing, "
        f"{still_missing} never published by the source)"
    )
    # A rerun saves nothing once the archive is complete — only a real
    # transport/database failure is an error here.
    return 1 if stats.get("aborted") or stats.get("failures") else 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,277 @@
 #!/usr/bin/env python3
 """Regenerate the documented P.1 flood-backtest charts in docs/img/.
 For each chart an eval-only model (regression 24 h peak + warning classifier)
 is trained on data STRICTLY BEFORE the event, then the event window is walked
 hour by hour exactly as the live system would have seen it:
  backtest-2024-p1.png         Oct 2024 record flood, trained < 1 Sep 2024
  backtest-2024-p1-detail.png  22-28 Sep 2024 zoom of the first crossing
  backtest-2025-p1.png         Sep 2025 flood, deployed config (trained <= 2024)
 This codifies the previously prose-only acceptance test: the run fails with a
 non-zero exit if the model gives less than 12 h of warning before the first
 3.70 m crossing of the 2024 event.
 Usage:
    python scripts/backtest_render.py                 # uses FLOOD_ML_DB_URL/Config
    python scripts/backtest_render.py --db-url postgresql://...
 """
 import argparse
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
 import matplotlib
 matplotlib.use("Agg")
 import matplotlib.dates as mdates
 import matplotlib.pyplot as plt
 import pandas as pd
 from src.ml import data, features
 from src.ml.train import _make_classifier, _make_regressor
 STATION = "P.1"
 STAGE1 = 3.70  # official Chiang Mai stage 1 - city flooding begins
 STAGE7 = 4.60  # stage 7 - widespread
 HORIZON = 24
 INK = "#132b35"
 BLUE = "#1c6ea4"
 AMBER = "#c07d10"
 RED = "#d9534f"
 def fit_backtest_model(df_long: pd.DataFrame, train_end: str, use_dam: bool = False):
    """Train the 24 h regression + warning heads on rows <= train_end only.
    Mirrors the deployed hgb-v3 pipeline: the regression head learns the RISE
    over the current level, with Open-Meteo catchment-rain features (trailing
    sums + the forward-24h forecast sum); label statistics are bounded to the
    training cutoff. use_dam=True adds the Mae Ngat reservoir columns — an
    ablation-only configuration (2026-08-13 result: costs 1-3 h of lead).
    """
    from src.ml import dam as dam_mod
    from src.ml import rain as rain_mod
    rain_series = rain_mod.catchment_mean(rain_mod.load_history())
    dam_frame = dam_mod.load_history() if use_dam else None
    X, Y, _meta = features.build_matrix(
        df_long, STATION, (HORIZON,), stats_end=train_end, rain=rain_series,
        dam=dam_frame,
    )
    train_mask = X.index <= pd.Timestamp(train_end)
    X_train, Y_train = X.loc[train_mask], Y.loc[train_mask]
    max_col, warn_col = f"max_level_{HORIZON}", f"exceed_warn_{HORIZON}"
    reg_rows = Y_train[max_col].notna()
    rise = Y_train.loc[reg_rows, max_col] - X_train.loc[reg_rows, "level"]
    reg = _make_regressor().fit(X_train.loc[reg_rows], rise)
    warn_rows = Y_train[warn_col].notna()
    clf = _make_classifier().fit(
        X_train.loc[warn_rows], Y_train.loc[warn_rows, warn_col].astype(int)
    )
    return X, reg, clf
 def event_series(df_long, X, reg, clf, window_start: str, window_end: str):
    """Observed level plus the forecasts the model would have issued hourly."""
    grid = features.make_hourly_grid(df_long)
    # observed has MultiIndex columns (station_code, field)
    observed = grid.observed[(STATION, "water_level")]
    observed = observed.loc[window_start:window_end].dropna().astype(float)
    Xw = X.loc[window_start:window_end]
    forecasts = pd.DataFrame(index=Xw.index)
    # reg predicts the rise; add the current level back (as serving does)
    forecasts["pred_max"] = reg.predict(Xw) + Xw["level"].to_numpy()
    # Belt-and-braces probability: the classifier OR the regression-sigmoid,
    # whichever is more alarmed. The classifier alone proved unreliable on
    # out-of-distribution extremes (silent on the 2024 record flood).
    import numpy as np
    p_clf = clf.predict_proba(Xw)[:, 1]
    p_sig = 1.0 / (1.0 + np.exp(-(forecasts["pred_max"] - STAGE1) / 0.15))
    forecasts["p_flood"] = np.maximum(p_clf, p_sig)
    flood_start = observed[observed >= STAGE1].index.min()
    alerts = forecasts[forecasts["p_flood"] >= 0.5].index
    first_alert = alerts.min() if len(alerts) else None
    return observed, forecasts, flood_start, first_alert
 def _style_axes(ax):
    ax.spines[["top", "right"]].set_visible(False)
    ax.tick_params(colors=INK, labelsize=11)
    ax.grid(axis="y", color="#dfe9e7", linewidth=0.8)
    ax.set_axisbelow(True)
 def render(observed, forecasts, flood_start, first_alert, out_path, *,
           title, subtitle, detail=False, show_stage7=False, peak_note=None):
    fig, (ax, axp) = plt.subplots(
        2, 1, figsize=(12.6, 7.6), sharex=True,
        gridspec_kw={"height_ratios": [2.2, 1], "hspace": 0.12},
    )
    fig.patch.set_facecolor("white")
    marker = dict(marker="o", markersize=3) if detail else {}
    ax.plot(observed.index, observed.values, color=BLUE, linewidth=2.2,
            label="Observed level" + (" (hourly)" if detail else ""), **marker)
    marker = dict(marker="s", markersize=3) if detail else {}
    ax.plot(forecasts.index, forecasts["pred_max"], color=AMBER, linewidth=2,
            linestyle="--", label="Predicted 24 h peak (issued at that hour)", **marker)
    ax.axhline(STAGE1, color=RED, linewidth=1, alpha=0.65)
    ax.annotate(f"{STAGE1:.2f} m · stage 1 · flooding begins", xy=(0.06, STAGE1),
                xycoords=("axes fraction", "data"), xytext=(0, 5),
                textcoords="offset points", color=RED, fontsize=10.5)
    if show_stage7:
        ax.axhline(STAGE7, color=RED, linewidth=1, alpha=0.65)
        ax.annotate(f"{STAGE7:.2f} m · stage 7 · widespread", xy=(0.06, STAGE7),
                    xycoords=("axes fraction", "data"), xytext=(0, 5),
                    textcoords="offset points", color=RED, fontsize=10.5)
    if peak_note:
        peak_ts = observed.idxmax()
        ax.annotate(peak_note, xy=(peak_ts, observed.max()),
                    xytext=(12, 10), textcoords="offset points",
                    color=BLUE, fontsize=11.5, fontweight="bold")
    ax.set_ylabel("P.1 water level (m)", color=INK, fontsize=11.5)
    ax.legend(loc="upper left", frameon=False, fontsize=10.5)
    _style_axes(ax)
    axp.plot(forecasts.index, forecasts["p_flood"], color=AMBER, linewidth=1.8)
    axp.fill_between(forecasts.index, 0, forecasts["p_flood"],
                     color=AMBER, alpha=0.28)
    axp.axhline(0.5, color=INK, linewidth=0.9, linestyle=":", alpha=0.6)
    axp.set_ylim(-0.02, 1.1)
    axp.set_ylabel(f"P(flooding within {HORIZON} h)", color=INK, fontsize=11.5)
    _style_axes(axp)
    if first_alert is not None:
        lead_h = None if flood_start is None else \
            int((flood_start - first_alert).total_seconds() // 3600)
        lead_txt = "" if lead_h is None else (
            f"\n({lead_h} h before flooding began)" if lead_h >= 0
            else f"\n({-lead_h} h after flooding began)"
        )
        if detail and flood_start is not None:
            for a in (ax, axp):
                a.axvline(first_alert, color=AMBER, linewidth=1.4, alpha=0.85)
                a.axvline(flood_start, color=BLUE, linewidth=1.4, alpha=0.85)
            # Anchor labels away from each other in chronological order so a
            # late alert (alert AFTER crossing) cannot overprint the labels.
            events = sorted(
                [(first_alert, "model alert", AMBER), (flood_start, "flooding begins", BLUE)]
            )
            for (ts, label, color), (offset, align) in zip(events, ((-8, "right"), (8, "left"))):
                ax.annotate(f"{label}\n{ts:%d %b %H:%M}",
                            xy=(ts, observed.min()), xytext=(offset, 18),
                            textcoords="offset points", ha=align,
                            color=color, fontsize=11, fontweight="bold")
            mid_y = observed.min() + (observed.max() - observed.min()) * 0.28
            ax.annotate("", xy=(flood_start, mid_y), xytext=(first_alert, mid_y),
                        arrowprops=dict(arrowstyle="<->", color=INK, lw=1.3))
            arrow_label = (
                f"{lead_h} h warning" if lead_h >= 0 else f"alert {-lead_h} h late"
            )
            ax.annotate(arrow_label,
                        xy=(first_alert + (flood_start - first_alert) / 2, mid_y),
                        xytext=(0, 8), textcoords="offset points", ha="center",
                        color=INK, fontsize=11.5, fontweight="bold")
        else:
            axp.annotate(f"first alert · {first_alert:%d %b %H:%M}{lead_txt}",
                         xy=(first_alert, 0.62), xytext=(10, 0),
                         textcoords="offset points", color=RED, fontsize=10.5,
                         bbox=dict(facecolor="white", alpha=0.75, edgecolor="none"))
    locator = mdates.DayLocator(interval=1 if detail else 3)
    axp.xaxis.set_major_locator(locator)
    axp.xaxis.set_major_formatter(mdates.DateFormatter("%d %b"))
    fig.suptitle(f"{title}\n{subtitle}", x=0.07, y=0.985, ha="left",
                 fontsize=15, color=INK)
    fig.subplots_adjust(top=0.885, left=0.07, right=0.97, bottom=0.07)
    fig.savefig(out_path, dpi=110)
    plt.close(fig)
    print(f"wrote {out_path}")
 def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--db-url", default=None)
    parser.add_argument("--out-dir", default=os.path.join("docs", "img"))
    parser.add_argument("--dam", action="store_true",
                        help="ablation: include Mae Ngat reservoir features "
                             "(2026-08 result: costs 1-3 h of alert lead)")
    parser.add_argument("--no-hii-fill", action="store_true",
                        help="ablation: load without the HII gap-fill merge")
    args = parser.parse_args(argv)
    df = data.load_measurements(
        db_url=args.db_url, hii_fill=not args.no_hii_fill
    )
    if df.empty:
        print("no measurement data available", file=sys.stderr)
        return 1
    os.makedirs(args.out_dir, exist_ok=True)
    # --- October 2024 record flood: trained only on data before 1 Sep 2024 ---
    X, reg, clf = fit_backtest_model(df, "2024-08-31", use_dam=args.dam)
    obs, fc, flood_start, first_alert = event_series(
        df, X, reg, clf, "2024-09-10", "2024-10-14 23:00")
    peak = float(obs.max())
    render(obs, fc, flood_start, first_alert,
           os.path.join(args.out_dir, "backtest-2024-p1.png"),
           title="October 2024 flood: what the model saw coming",
           subtitle="P.1 Nawarat Bridge — model trained only on data before 1 Sep 2024",
           show_stage7=True, peak_note=f"record peak {peak:.2f} m")
    obs_d, fc_d, flood_d, alert_d = event_series(
        df, X, reg, clf, "2024-09-21 18:00", "2024-09-28 06:00")
    lead_h = None
    if alert_d is not None and flood_d is not None:
        lead_h = int((flood_d - alert_d).total_seconds() // 3600)
    render(obs_d, fc_d, flood_d, alert_d,
           os.path.join(args.out_dir, "backtest-2024-p1-detail.png"),
           title="Detection in detail: 22–28 September 2024, hour by hour",
           subtitle=(
               f"the model alerts {lead_h} h before the river crosses the flooding line"
               if lead_h is not None and lead_h > 0
               else "model alert vs the river crossing the flooding line"
           ),
           detail=True)
    # --- September 2025 flood: the deployed configuration (trained <= 2024) ---
    X25, reg25, clf25 = fit_backtest_model(df, "2024-12-31", use_dam=args.dam)
    obs25, fc25, flood25, alert25 = event_series(
        df, X25, reg25, clf25, "2025-09-22", "2025-10-02 12:00")
    pred_at_alert = float(fc25.loc[alert25:, "pred_max"].iloc[:24].max()) if alert25 is not None else None
    note = f"peak {float(obs25.max()):.2f} m" + (
        f" (predicted {pred_at_alert:.2f} m)" if pred_at_alert is not None else "")
    render(obs25, fc25, flood25, alert25,
           os.path.join(args.out_dir, "backtest-2025-p1.png"),
           title="The September 2025 flood — as forecast by the deployed configuration",
           subtitle="model trained only on data through 2024; this event was never seen in training",
           detail=True, peak_note=note)
    print(f"2024: flooding began {flood_start}, first alert {first_alert}")
    print(f"2025: flooding began {flood25}, first alert {alert25}")
    # Acceptance gate: the flagship 2024 event must keep a >= 12 h warning
    if first_alert is None or flood_start is None:
        print("FAIL: 2024 event alert or crossing not found", file=sys.stderr)
        return 1
    lead = (flood_start - first_alert).total_seconds() / 3600
    if lead < 12:
        print(f"FAIL: 2024 first-alert lead {lead:.0f} h < 12 h", file=sys.stderr)
        return 1
    print(f"PASS: 2024 first-alert lead {lead:.0f} h")
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,58 @@
 """Serve the working-copy dashboard locally with API calls proxied to the
 live server, so browser-side changes can be checked against real data
 before deploy. Usage: python scripts/dev_proxy.py [port]"""
 import http.server
 import os
 import sys
 import urllib.request
 from pathlib import Path
 UPSTREAM = "https://water.buildfor.life"
 STATIC = Path(__file__).resolve().parents[1] / "src" / "static"
 class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/" or self.path.startswith("/?"):
            body = (STATIC / "dashboard.html").read_bytes()
            self._send(200, "text/html; charset=utf-8", body)
            return
        # Local overrides for endpoints not yet deployed: DEV_PROXY_LOCAL=/api/x=file.json,...
        for pair in filter(None, os.environ.get("DEV_PROXY_LOCAL", "").split(",")):
            prefix, file = pair.split("=", 1)
            if self.path.split("?")[0] == prefix:
                self._send(200, "application/json", Path(file).read_bytes())
                return
        if self.path.startswith("/static/"):
            f = STATIC / self.path[len("/static/"):].split("?")[0]
            if f.is_file():
                ctype = "application/json" if f.suffix in (".json", ".geojson") else "application/octet-stream"
                self._send(200, ctype, f.read_bytes())
                return
        try:
            req = urllib.request.Request(
                UPSTREAM + self.path,
                headers={"User-Agent": "Mozilla/5.0 (dev_proxy; +https://buildfor.life)", "Accept": "application/json"},
            )
            with urllib.request.urlopen(req, timeout=60) as r:
                self._send(r.status, r.headers.get("Content-Type", "application/json"), r.read())
        except urllib.error.HTTPError as e:
            self._send(e.code, "application/json", e.read())
    def _send(self, code, ctype, body):
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Cache-Control", "no-store")
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a):
        pass
 if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8765
    print(f"http://localhost:{port}/  (API -> {UPSTREAM})")
    http.server.ThreadingHTTPServer(("127.0.0.1", port), Handler).serve_forever()
@@ -0,0 +1,132 @@
 """Drive the production notify path in-process: startup init -> seeded readings
 -> forecast cache -> _notify_transitions -> sqlite state -> real ntfy."""
 import asyncio
 import datetime
 import json
 import os
 import sys
 import requests
 os.environ.update(
    DB_TYPE="sqlite",
    WATER_DB_PATH=os.path.join(os.environ["LOCALAPPDATA"], "Temp", "smoke3.db"),
    NTFY_SERVER="http://127.0.0.1:2586",
    NTFY_TOKEN=os.environ.get("NTFY_TOKEN", ""),
    NTFY_TOPIC_PREFIX="ping",
 )
 for f in ("smoke3.db",):
    p = os.path.join(os.environ["LOCALAPPDATA"], "Temp", f)
    if os.path.exists(p):
        os.remove(p)
 from src import web_api  # noqa: E402
 from src.config import Config  # noqa: E402
 assert Config.NTFY_SERVER
 async def main():
    # what the lifespan does at startup, minus the scheduler
    from src import notify as notify_mod
    from src.forecast_history import ForecastHistoryStore
    from src.water_scraper_v3 import EnhancedWaterMonitorScraper
    db_config = Config.get_database_config()
    web_api.app_state["scraper"] = EnhancedWaterMonitorScraper(db_config)
    store = ForecastHistoryStore(db_config["connection_string"], db_config["type"])
    store.connect()
    web_api.app_state["forecast_store"] = store
    state = notify_mod.NotificationState(store.engine, store.db_type)
    pub = notify_mod.NtfyPublisher(
        Config.NTFY_SERVER, prefix=Config.NTFY_TOPIC_PREFIX, token=Config.NTFY_TOKEN
    )
    web_api.app_state["notify"] = (pub, state)
    scraper = web_api.app_state["scraper"]
    now = datetime.datetime.now().replace(minute=0, second=0, microsecond=0)
    def seed(level_p1, level_p103, ts):
        rows = [
            {
                "station_code": "P.1",
                "station_id": 1,
                "timestamp": ts,
                "water_level": level_p1,
                "discharge": 400.0,
                "station_name_en": "Nawarat Bridge",
                "station_name_th": "สะพานนวรัฐ",
                "discharge_percent": 30.0,
                "status": "active",
            },
            {
                "station_code": "P.103",
                "station_id": 2,
                "timestamp": ts,
                "water_level": level_p103,
                "discharge": 300.0,
                "station_name_en": "Ring Road 3",
                "station_name_th": "วงแหวน 3",
                "discharge_percent": 20.0,
                "status": "active",
            },
        ]
        scraper.db_adapter.save_measurements(rows)
    def forecast(p):
        with web_api.FORECAST_CACHE_LOCK:
            web_api.FORECAST_CACHE["all"] = (
                0,
                [
                    {
                        "station_code": "P.1",
                        "horizon_hours": 24,
                        "p_warning": p,
                        "predicted_max_level": 3.9,
                        "source": "model",
                    }
                ],
            )
    def poll(topic):
        out = []
        for line in (
            requests.get(f"{Config.NTFY_SERVER}/{topic}/json?poll=1", timeout=5)
            .text.strip()
            .splitlines()
        ):
            m = json.loads(line)
            if m.get("event") == "message":
                out.append(m.get("title") or m.get("message", "")[:40])
        return out
    # cycle 1: quiet
    seed(1.6, 3.2, now - datetime.timedelta(hours=2))
    forecast(0.02)
    await web_api._notify_transitions()
    # cycle 2: P.1 crosses warning, model outlook on
    seed(3.75, 3.3, now - datetime.timedelta(hours=1))
    forecast(0.7)
    await web_api._notify_transitions()
    # cycle 3: same state -> silence
    seed(3.80, 3.3, now)
    forecast(0.65)
    await web_api._notify_transitions()
    print("ping-p1-warning:", poll("ping-p1-warning"))
    print("ping-warning:   ", poll("ping-warning"))
    print("ping-p1-outlook:", poll("ping-p1-outlook"))
    print("ping-p103-warning:", poll("ping-p103-warning"))
    from sqlalchemy import text
    with store.engine.connect() as c:
        print(
            "state table:",
            c.execute(
                text("SELECT key, state, value FROM notification_state ORDER BY key")
            ).fetchall(),
        )
 asyncio.run(main())
@@ -1,57 +0,0 @@
 #!/usr/bin/env python3
 """
 Password URL encoder for PostgreSQL connection strings
 """
 import urllib.parse
 import sys
 def encode_password(password: str) -> str:
    """URL encode a password for use in connection strings"""
    return urllib.parse.quote(password, safe='')
 def build_connection_string(username: str, password: str, host: str, port: int, database: str) -> str:
    """Build a properly encoded PostgreSQL connection string"""
    encoded_password = encode_password(password)
    return f"postgresql://{username}:{encoded_password}@{host}:{port}/{database}"
 def main():
    print("PostgreSQL Password URL Encoder")
    print("=" * 40)
    if len(sys.argv) > 1:
        # Password provided as argument
        password = sys.argv[1]
    else:
        # Interactive mode
        password = input("Enter your password: ")
    encoded = encode_password(password)
    print(f"\nOriginal password: {password}")
    print(f"URL encoded: {encoded}")
    # Optional: build full connection string
    try:
        build_full = input("\nBuild full connection string? (y/N): ").strip().lower() == 'y'
    except (EOFError, KeyboardInterrupt):
        print("\nDone!")
        return
    if build_full:
        username = input("Username: ").strip()
        host = input("Host: ").strip()
        port = input("Port [5432]: ").strip() or "5432"
        database = input("Database [water_monitoring]: ").strip() or "water_monitoring"
        connection_string = build_connection_string(username, password, host, int(port), database)
        print(f"\nComplete connection string:")
        print(f"POSTGRES_CONNECTION_STRING={connection_string}")
        print(f"\nAdd this to your .env file:")
        print(f"DB_TYPE=postgresql")
        print(f"POSTGRES_CONNECTION_STRING={connection_string}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,18 @@
 #!/usr/bin/env python3
 """CLI for the rolling-origin model-variant evaluation.
 Usage:
    uv run scripts/evaluate_variants.py                        # P.1, all variants
    uv run scripts/evaluate_variants.py --stations P.1,P.103
    uv run scripts/evaluate_variants.py --variants baseline_abs,rise_quantile
 """
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
 from src.ml.evaluate import main
 if __name__ == "__main__":
    sys.exit(main())
@@ -1,51 +0,0 @@
 #!/usr/bin/env python3
 """
 Generate status badges for README.md
 """
 import json
 import requests
 from datetime import datetime
 def generate_badge_url(label, message, color="brightgreen"):
    """Generate a shields.io badge URL"""
    return f"https://img.shields.io/badge/{label}-{message}-{color}"
 def generate_workflow_badge(repo_url, workflow_name, branch="main"):
    """Generate workflow status badge"""
    # For Gitea, you might need to adjust this based on your instance
    badge_url = f"{repo_url}/actions/workflows/{workflow_name}/badge.svg?branch={branch}"
    return badge_url
 def main():
    """Generate badges for the project"""
    repo_url = "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor"
    badges = {
        "CI/CD": generate_workflow_badge(repo_url, "ci.yml"),
        "Security": generate_workflow_badge(repo_url, "security.yml"),
        "Documentation": generate_workflow_badge(repo_url, "docs.yml"),
        "Python": generate_badge_url("Python", "3.9%2B", "blue"),
        "FastAPI": generate_badge_url("FastAPI", "0.104%2B", "green"),
        "Docker": generate_badge_url("Docker", "Ready", "blue"),
        "License": generate_badge_url("License", "MIT", "green"),
        "Version": generate_badge_url("Version", "v3.1.3", "blue"),
    }
    print("# Status Badges")
    print()
    print("Add these badges to your README.md:")
    print()
    for name, url in badges.items():
        print(f"[![{name}]({url})]({repo_url})")
    print()
    print("# Markdown Format")
    print()
    badge_line = " ".join([f"[![{name}]({url})]({repo_url})" for name, url in badges.items()])
    print(badge_line)
 if __name__ == "__main__":
    main()
@@ -1,35 +0,0 @@
@echo off
 REM Git initialization script for Northern Thailand Ping River Monitor
 echo 🏔️ Initializing Git repository for Northern Thailand Ping River Monitor
 REM Initialize git repository
 git init
 REM Add remote origin
 git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 REM Add all files
 git add .
 REM Initial commit
 git commit -m "Initial commit: Northern Thailand Ping River Monitor v3.1.3
 Features:
 - Real-time water level monitoring for Ping River Basin
 - 16 monitoring stations from Chiang Dao to Nakhon Sawan
 - FastAPI web interface with station management
 - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
 - Comprehensive monitoring and health checks
 - Docker deployment with Grafana integration
 - Production-ready architecture with CI/CD pipeline"
 echo ✅ Git repository initialized successfully!
 echo.
 echo Next steps:
 echo 1. Review and edit .env file with your configuration
 echo 2. Push to remote repository:
 echo    git push -u origin main
 echo.
 echo 3. Start the application:
 echo    python run.py --web-api
@@ -1,89 +0,0 @@
 #!/bin/bash
 # Git initialization script for Northern Thailand Ping River Monitor
 echo "🏔️ Initializing Git repository for Northern Thailand Ping River Monitor"
 # Initialize git repository
 git init
 # Add remote origin
 git remote add origin https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
 # Create .gitignore if it doesn't exist
 if [ ! -f .gitignore ]; then
    echo "Creating .gitignore file..."
    cat > .gitignore << 'EOF'
 # Python
 __pycache__/
 *.py[cod]
 *.so
 .Python
 build/
 develop-eggs/
 dist/
 downloads/
 eggs/
 .eggs/
 lib/
 lib64/
 parts/
 sdist/
 var/
 wheels/
 *.egg-info/
 .installed.cfg
 *.egg
 # Virtual environments
 .env
 .venv
 env/
 venv/
 ENV/
 # IDE
 .vscode/
 .idea/
 *.swp
 *.swo
 # Logs
 *.log
 logs/
 # Database files
 *.db
 *.sqlite
 *.sqlite3
 # OS
 .DS_Store
 Thumbs.db
 EOF
 fi
 # Add all files
 git add .
 # Initial commit
 git commit -m "Initial commit: Northern Thailand Ping River Monitor v3.1.3
 Features:
 - Real-time water level monitoring for Ping River Basin
 - 16 monitoring stations from Chiang Dao to Nakhon Sawan
 - FastAPI web interface with station management
 - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
 - Comprehensive monitoring and health checks
 - Docker deployment with Grafana integration
 - Production-ready architecture with CI/CD pipeline"
 echo "✅ Git repository initialized successfully!"
 echo ""
 echo "Next steps:"
 echo "1. Review and edit .env file with your configuration"
 echo "2. Push to remote repository:"
 echo "   git push -u origin main"
 echo ""
 echo "3. Start the application:"
 echo "   make run-api"
 echo "   # or: python run.py --web-api"
@@ -18,6 +18,7 @@ APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
 SERVICE_USER="${SERVICE_USER:-water-monitor}"
 SERVICE_GROUP="${SERVICE_GROUP:-${SERVICE_USER}}"
 SERVICE_NAME="water-monitor.service"
 RETRAIN_NAME="water-monitor-retrain"
 # Resolve the repo root (parent of this scripts/ directory).
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -72,11 +73,18 @@ if ! command -v uv >/dev/null 2>&1; then
 fi
 UV="$(command -v uv)"
-log "Creating virtualenv at ${APP_DIR}/venv"
+log "Syncing uv-managed virtualenv at ${APP_DIR}/.venv"
 cd "${APP_DIR}"
-# Named 'venv' (not uv's default .venv) to match the systemd unit's ExecStart.
+# ONE environment: uv sync owns .venv/ (from pyproject.toml + uv.lock, so the
-"${UV}" venv venv
+# ML extras such as scikit-learn/joblib are present) and both systemd units
-"${UV}" pip install --python venv/bin/python -r requirements.txt
+# run its interpreter directly. Never create a second env by another name --
 # a stale 'venv/' once coexisted here and broke manual retrains with
 # ModuleNotFoundError while the service itself ran fine.
 "${UV}" sync --python 3.11 --frozen
 if [ -d "${APP_DIR}/venv" ]; then
    warn "Removing stale ${APP_DIR}/venv (superseded by .venv)"
    rm -rf "${APP_DIR}/venv"
 fi
 # 4. Environment file ----------------------------------------------------------
 if [ ! -f "${APP_DIR}/.env" ]; then
@@ -100,11 +108,14 @@ if [ -f "${APP_DIR}/.env" ]; then
    chmod 0600 "${APP_DIR}/.env"
 fi
-# 6. Install and enable the systemd unit --------------------------------------
+# 6. Install and enable the systemd units -------------------------------------
-log "Installing systemd unit"
+log "Installing systemd units"
 install -m 0644 "${SCRIPT_DIR}/${SERVICE_NAME}" "/etc/systemd/system/${SERVICE_NAME}"
 install -m 0644 "${SCRIPT_DIR}/${RETRAIN_NAME}.service" "/etc/systemd/system/${RETRAIN_NAME}.service"
 install -m 0644 "${SCRIPT_DIR}/${RETRAIN_NAME}.timer" "/etc/systemd/system/${RETRAIN_NAME}.timer"
 systemctl daemon-reload
 systemctl enable "${SERVICE_NAME}"
 systemctl enable --now "${RETRAIN_NAME}.timer"
 log "Done."
 echo
@@ -112,3 +123,5 @@ echo "Next steps:"
 echo "  sudo systemctl start ${SERVICE_NAME}"
 echo "  systemctl status ${SERVICE_NAME}"
 echo "  sudo journalctl -u ${SERVICE_NAME} -f"
 echo "  systemctl list-timers ${RETRAIN_NAME}.timer   # monthly flood-model retrain"
 echo "  sudo systemctl start ${RETRAIN_NAME}.service   # retrain now"
@@ -0,0 +1,102 @@
 #!/usr/bin/env bash
 # Install ntfy (https://ntfy.sh) as the public notification server for the
 # Ping River Monitor. Run as root on the monitor VPS. Idempotent.
 #
 #   NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
 #
 # What it does:
 #   - installs the ntfy .deb from the official GitHub release (single Go
 #     binary, ~30 MB RSS, sqlite message cache)
 #   - writes /etc/ntfy/server.yml: listens on the Tailscale address only
 #     (the reverse proxy is another VPS on the tailnet; nothing is exposed
 #     on a public interface), anonymous READ on all topics, WRITE only with
 #     a token. Override with NTFY_LISTEN=host:port.
 #   - creates the `monitor` publishing user + token, writes NTFY_SERVER /
 #     NTFY_TOKEN into /opt/thailand-water-monitor/.env if not present
 #
 # Reverse proxy (on the Caddy VPS, over Tailscale):
 #     ntfy.buildfor.life {
 #         reverse_proxy <this host's tailscale ip>:2586
 #     }
 # Caddy passes websockets and keeps long-poll connections open by default;
 # subscribers hold one open. ntfy runs with behind-proxy: true so rate
 # limits key on X-Forwarded-For, not on the proxy's address.
 set -euo pipefail
 NTFY_DOMAIN="${NTFY_DOMAIN:?set NTFY_DOMAIN, e.g. ntfy.buildfor.life}"
 NTFY_VERSION="${NTFY_VERSION:-2.28.0}"
 MONITOR_DIR="${MONITOR_DIR:-/opt/thailand-water-monitor}"
 TS_IP="$(tailscale ip -4 2>/dev/null | head -1 || true)"
 LISTEN="${NTFY_LISTEN:-${TS_IP:-127.0.0.1}:2586}"
 echo "ntfy will listen on ${LISTEN}"
 if ! command -v ntfy >/dev/null || [[ "$(ntfy --version 2>/dev/null | awk '{print $3}')" != "$NTFY_VERSION" ]]; then
    tmp=$(mktemp -d)
    curl -fsSL -o "$tmp/ntfy.deb" \
        "https://github.com/binwiederhier/ntfy/releases/download/v${NTFY_VERSION}/ntfy_${NTFY_VERSION}_linux_amd64.deb"
    dpkg -i "$tmp/ntfy.deb"
    rm -rf "$tmp"
 fi
 install -d -m 755 /var/cache/ntfy /var/lib/ntfy
 cat > /etc/ntfy/server.yml <<EOF
 # Ping River Monitor notification server. Managed by scripts/install_ntfy.sh.
 base-url: "https://${NTFY_DOMAIN}"
 listen-http: "${LISTEN}"
 behind-proxy: true
 # Messages are kept so a phone that was offline still gets the crossing.
 cache-file: "/var/cache/ntfy/cache.db"
 cache-duration: "72h"
 # Everyone may subscribe; only the monitor (token) may publish.
 auth-file: "/var/lib/ntfy/user.db"
 auth-default-access: "read-only"
 # The monitor publishes a handful of messages per flood; be strict with
 # everything else so the box cannot be used as a free relay.
 visitor-request-limit-burst: 30
 visitor-request-limit-replenish: "10s"
 visitor-subscription-limit: 60
 visitor-message-daily-limit: 200
 attachment-cache-dir: ""
 enable-signup: false
 enable-login: false
 enable-metrics: false
 EOF
 systemctl enable --now ntfy
 systemctl restart ntfy
 sleep 1
 curl -fsS "http://${LISTEN}/v1/health" >/dev/null && echo "ntfy up on ${LISTEN}"
 # Publishing identity for the monitor
 if ! ntfy user list 2>/dev/null | grep -q '^user monitor (role'; then
    NTFY_PASSWORD="$(openssl rand -base64 24)" ntfy user add --role=user monitor
 fi
 ntfy access monitor 'ping-*' write-only >/dev/null
 # 'ping-*' read stays anonymous via auth-default-access
 token=$(ntfy token list monitor 2>/dev/null | awk '/^- tk_/{print $2; exit}')  # '- tk_xxx (label), ...'
 if [[ -z "$token" ]]; then
    token=$(ntfy token add --label "water-monitor" monitor | grep -oE 'tk_[A-Za-z0-9]+' | head -1)  # 'token tk_xxx created for user monitor'
 fi
 env_file="${MONITOR_DIR}/.env"
 if [[ -f "$env_file" ]] && ! grep -q '^NTFY_SERVER=' "$env_file"; then
    {
        echo ""
        echo "# ntfy public notifications (scripts/install_ntfy.sh)"
        echo "NTFY_SERVER=https://${NTFY_DOMAIN}"
        echo "NTFY_PUBLISH_URL=http://${LISTEN}"
        echo "NTFY_TOPIC_PREFIX=ping"
        echo "NTFY_TOKEN=${token}"
    } >> "$env_file"
    echo "wrote NTFY_* to ${env_file}; restart water-monitor to enable"
 else
    echo "NTFY_TOKEN=${token}"
 fi
 echo
 echo "Subscribe test (anonymous read):  curl -s 'http://${LISTEN}/ping-status/json?poll=1'"
 echo "Publish test (needs token):       curl -s -H 'Authorization: Bearer ${token}' -d 'hello' http://${LISTEN}/ping-status"
@@ -0,0 +1,148 @@
 #!/usr/bin/env python3
 """Staged load / client-stress test for the Ping River Monitor API + dashboard.
 Simulates a realistic traffic mix (dashboard page loads, the API calls the
 dashboard itself makes, heavy history queries, external API consumers) at
 increasing concurrency stages, and reports throughput, latency percentiles,
 and errors per stage plus the slowest endpoints.
 Run against a LOCAL instance for full stress (never full-stress production —
 it hosts live flood monitoring):
    python -m uvicorn src.web_api:app --port 8125   # separate shell
    python scripts/load_test.py http://localhost:8125
 A gentle production baseline (low, fixed concurrency):
    python scripts/load_test.py https://water.buildfor.life --gentle
 """
 import argparse
 import random
 import statistics
 import threading
 import time
 from collections import Counter
 import requests
 # Weighted endpoint mix: dashboard session + API consumers
 ENDPOINTS = [
    ("/", 10),
    ("/measurements/latest?limit=500", 20),
    ("/stations", 10),
    ("/api/hii/rainfall/latest", 15),
    ("/api/hii/waterlevel/latest", 15),
    ("/forecast", 10),
    ("/api/stats", 5),
    ("/measurements/history/P.1?hours=168", 10),
    ("/measurements/history/P.67?hours=720", 5),
    ("/health", 5),
 ]
 POOL = [endpoint for endpoint, weight in ENDPOINTS for _ in range(weight)]
 FULL_STAGES = [(10, 20), (50, 20), (200, 25)]  # (clients, seconds)
 GENTLE_STAGES = [(3, 15), (8, 15)]
 def _worker(base, stop_at, results, errors):
    session = requests.Session()
    while time.time() < stop_at:
        path = random.choice(POOL)
        start = time.perf_counter()
        try:
            response = session.get(f"{base}{path}", timeout=30)
            elapsed = time.perf_counter() - start
            if response.status_code == 200:
                results.append((path, elapsed))
            else:
                errors.append((path, response.status_code))
        except Exception as error:
            errors.append((path, type(error).__name__))
 def _pct(values, p):
    if len(values) >= 100:
        return statistics.quantiles(values, n=100)[p - 1]
    return max(values)
 def run_stage(base, clients, seconds):
    results, errors = [], []
    stop_at = time.time() + seconds
    threads = [
        threading.Thread(
            target=_worker, args=(base, stop_at, results, errors), daemon=True
        )
        for _ in range(clients)
    ]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join(timeout=seconds + 35)
    latencies = [elapsed for _, elapsed in results]
    total = len(results) + len(errors)
    print(f"\n== {clients} clients x {seconds}s ==")
    print(
        f"requests: {total}  ok: {len(results)}  errors: {len(errors)}  "
        f"rps: {total / seconds:.1f}"
    )
    if latencies:
        print(
            f"latency ms  p50: {statistics.median(latencies) * 1000:.0f}  "
            f"p95: {_pct(latencies, 95) * 1000:.0f}  "
            f"p99: {_pct(latencies, 99) * 1000:.0f}  "
            f"max: {max(latencies) * 1000:.0f}"
        )
    by_endpoint = {}
    for path, elapsed in results:
        by_endpoint.setdefault(path, []).append(elapsed)
    slowest = sorted(
        by_endpoint.items(), key=lambda kv: -statistics.median(kv[1])
    )[:4]
    for path, values in slowest:
        print(
            f"  slow: {path:45} n={len(values):5} "
            f"p50={statistics.median(values) * 1000:6.0f}ms "
            f"max={max(values) * 1000:7.0f}ms"
        )
    if errors:
        top = Counter(f"{path} {code}" for path, code in errors).most_common(5)
        print(f"  errors: {top}")
    return {"clients": clients, "total": total, "errors": len(errors)}
 def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("base", nargs="?", default="http://localhost:8125")
    parser.add_argument(
        "--gentle",
        action="store_true",
        help="low fixed concurrency (safe for the production instance)",
    )
    args = parser.parse_args(argv)
    base = args.base.rstrip("/")
    # Warm caches first so stage 1 doesn't measure cold-start work
    for path in ("/forecast", "/api/stats", "/measurements/latest?limit=500"):
        try:
            requests.get(f"{base}{path}", timeout=60)
        except Exception:
            pass
    print(f"target: {base}  mode: {'gentle' if args.gentle else 'full'}")
    stages = GENTLE_STAGES if args.gentle else FULL_STAGES
    summary = [run_stage(base, clients, seconds) for clients, seconds in stages]
    worst = max(
        (stage["errors"] / stage["total"] for stage in summary if stage["total"]),
        default=1.0,
    )
    print(f"\nworst-stage error rate: {worst:.1%}")
    return 0 if worst < 0.05 else 1
 if __name__ == "__main__":
    import sys
    sys.exit(main())
@@ -0,0 +1,98 @@
 """Locust load profile for the Ping River Monitor API + dashboard.
 Two user types mirror real traffic: dashboard visitors (page + the API calls
 the page makes, polling like the auto-refresh does) and API consumers
 (direct endpoint hits, including heavy history queries).
 Full stress against a LOCAL instance (never full-stress production — it hosts
 live flood monitoring):
    # separate shell: python -m uvicorn src.web_api:app --port 8125
    .venv/Scripts/python.exe -m locust -f scripts/locustfile.py \
        --host http://localhost:8125 --headless \
        --users 200 --spawn-rate 20 --run-time 2m \
        --html load-report.html
 Interactive UI instead: drop --headless and open http://localhost:8089.
 """
 import random
 from locust import FastHttpUser, between, task
 # Explicit so measurements reflect compressed transfer (browsers always send this)
 GZIP = {"Accept-Encoding": "gzip, deflate"}
 class DashboardVisitor(FastHttpUser):
    """A browser session: initial page load, then periodic refresh polling."""
    weight = 3
    wait_time = between(2, 6)
    def on_start(self):
        # What one real page load requests
        self.client.get("/", headers=GZIP)
        self.client.get("/stations", headers=GZIP)
        self.client.get("/measurements/latest?limit=500", headers=GZIP)
        self.client.get("/api/hii/waterlevel/latest", headers=GZIP)
        self.client.get("/api/hii/rainfall/latest", headers=GZIP)
    @task(4)
    def poll_latest(self):
        self.client.get("/measurements/latest?limit=500", headers=GZIP)
    @task(2)
    def poll_forecast(self):
        self.client.get("/forecast", headers=GZIP)
    @task(2)
    def poll_rain(self):
        self.client.get("/api/hii/rainfall/latest", headers=GZIP)
    @task(1)
    def view_history(self):
        station = random.choice(["P.1", "P.67", "P.103", "P.75", "P.20"])
        hours = random.choice([24, 168, 720])
        self.client.get(
            f"/measurements/history/{station}?hours={hours}",
            headers=GZIP,
            name="/measurements/history/[station]",
        )
    @task(1)
    def stats(self):
        self.client.get("/api/stats", headers=GZIP)
 class ApiConsumer(FastHttpUser):
    """A script/integration hitting the JSON API directly, no think time."""
    weight = 1
    wait_time = between(0.1, 1)
    @task(3)
    def latest(self):
        self.client.get("/measurements/latest?limit=100", headers=GZIP)
    @task(3)
    def hii_feeds(self):
        self.client.get(random.choice(
            ["/api/hii/waterlevel/latest", "/api/hii/rainfall/latest"]
        ), headers=GZIP, name="/api/hii/[feed]/latest")
    @task(2)
    def forecast(self):
        self.client.get("/forecast", headers=GZIP)
    @task(2)
    def heavy_history(self):
        self.client.get(
            "/measurements/history/P.1?hours=8760",
            headers=GZIP,
            name="/measurements/history/P.1 [heavy]",
        )
    @task(1)
    def health(self):
        self.client.get("/health", headers=GZIP)
@@ -1,294 +0,0 @@
 #!/usr/bin/env python3
 """
 Migration script to add geolocation columns to existing water monitoring database
 """
 import os
 import sys
 import sqlite3
 import logging
 from typing import Dict, Any
 # Configure logging
 logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
 )
 def migrate_sqlite(db_path: str = 'water_monitoring.db') -> bool:
    """Migrate SQLite database to add geolocation columns"""
    try:
        logging.info(f"Migrating SQLite database: {db_path}")
        # Connect to database
        conn = sqlite3.connect(db_path)
        cursor = conn.cursor()
        # Check if columns already exist
        cursor.execute("PRAGMA table_info(stations)")
        columns = [column[1] for column in cursor.fetchall()]
        logging.info(f"Current columns in stations table: {columns}")
        # Add columns if they don't exist
        columns_added = []
        if 'latitude' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN latitude REAL")
            columns_added.append('latitude')
            logging.info("Added latitude column")
        if 'longitude' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN longitude REAL")
            columns_added.append('longitude')
            logging.info("Added longitude column")
        if 'geohash' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN geohash TEXT")
            columns_added.append('geohash')
            logging.info("Added geohash column")
        if columns_added:
            # Update P.1 station with sample geolocation data
            cursor.execute("""
                UPDATE stations 
                SET latitude = 15.6944, longitude = 100.2028, geohash = 'w5q6uuhvfcfp25'
                WHERE station_code = 'P.1'
            """)
            # Commit changes
            conn.commit()
            logging.info(f"Successfully added columns: {', '.join(columns_added)}")
            logging.info("Updated P.1 station with sample geolocation data")
        else:
            logging.info("All geolocation columns already exist")
        # Verify the changes
        cursor.execute("SELECT station_code, latitude, longitude, geohash FROM stations WHERE station_code = 'P.1'")
        result = cursor.fetchone()
        if result:
            logging.info(f"P.1 station geolocation: {result}")
        conn.close()
        return True
    except Exception as e:
        logging.error(f"Error migrating SQLite database: {e}")
        return False
 def migrate_postgresql(connection_string: str) -> bool:
    """Migrate PostgreSQL database to add geolocation columns"""
    try:
        import psycopg2
        from urllib.parse import urlparse
        logging.info("Migrating PostgreSQL database")
        # Parse connection string
        parsed = urlparse(connection_string)
        # Connect to database
        conn = psycopg2.connect(
            host=parsed.hostname,
            port=parsed.port or 5432,
            database=parsed.path[1:],  # Remove leading slash
            user=parsed.username,
            password=parsed.password
        )
        cursor = conn.cursor()
        # Check if columns exist
        cursor.execute("""
            SELECT column_name 
            FROM information_schema.columns 
            WHERE table_name = 'stations'
        """)
        columns = [row[0] for row in cursor.fetchall()]
        logging.info(f"Current columns in stations table: {columns}")
        # Add columns if they don't exist
        columns_added = []
        if 'latitude' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN latitude DECIMAL(10,8)")
            columns_added.append('latitude')
            logging.info("Added latitude column")
        if 'longitude' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN longitude DECIMAL(11,8)")
            columns_added.append('longitude')
            logging.info("Added longitude column")
        if 'geohash' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN geohash VARCHAR(20)")
            columns_added.append('geohash')
            logging.info("Added geohash column")
        if columns_added:
            # Update P.1 station with sample geolocation data
            cursor.execute("""
                UPDATE stations 
                SET latitude = 15.6944, longitude = 100.2028, geohash = 'w5q6uuhvfcfp25'
                WHERE station_code = 'P.1'
            """)
            # Commit changes
            conn.commit()
            logging.info(f"Successfully added columns: {', '.join(columns_added)}")
            logging.info("Updated P.1 station with sample geolocation data")
        else:
            logging.info("All geolocation columns already exist")
        conn.close()
        return True
    except ImportError:
        logging.error("psycopg2 not installed. Run: pip install psycopg2-binary")
        return False
    except Exception as e:
        logging.error(f"Error migrating PostgreSQL database: {e}")
        return False
 def migrate_mysql(connection_string: str) -> bool:
    """Migrate MySQL database to add geolocation columns"""
    try:
        import pymysql
        from urllib.parse import urlparse
        logging.info("Migrating MySQL database")
        # Parse connection string
        parsed = urlparse(connection_string)
        # Connect to database
        conn = pymysql.connect(
            host=parsed.hostname,
            port=parsed.port or 3306,
            database=parsed.path[1:],  # Remove leading slash
            user=parsed.username,
            password=parsed.password
        )
        cursor = conn.cursor()
        # Check if columns exist
        cursor.execute("DESCRIBE stations")
        columns = [row[0] for row in cursor.fetchall()]
        logging.info(f"Current columns in stations table: {columns}")
        # Add columns if they don't exist
        columns_added = []
        if 'latitude' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN latitude DECIMAL(10,8)")
            columns_added.append('latitude')
            logging.info("Added latitude column")
        if 'longitude' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN longitude DECIMAL(11,8)")
            columns_added.append('longitude')
            logging.info("Added longitude column")
        if 'geohash' not in columns:
            cursor.execute("ALTER TABLE stations ADD COLUMN geohash VARCHAR(20)")
            columns_added.append('geohash')
            logging.info("Added geohash column")
        if columns_added:
            # Update P.1 station with sample geolocation data
            cursor.execute("""
                UPDATE stations 
                SET latitude = 15.6944, longitude = 100.2028, geohash = 'w5q6uuhvfcfp25'
                WHERE station_code = 'P.1'
            """)
            # Commit changes
            conn.commit()
            logging.info(f"Successfully added columns: {', '.join(columns_added)}")
            logging.info("Updated P.1 station with sample geolocation data")
        else:
            logging.info("All geolocation columns already exist")
        conn.close()
        return True
    except ImportError:
        logging.error("pymysql not installed. Run: pip install pymysql")
        return False
    except Exception as e:
        logging.error(f"Error migrating MySQL database: {e}")
        return False
 def load_config_from_env() -> Dict[str, Any]:
    """Load database configuration from environment variables"""
    db_type = os.getenv('DB_TYPE', 'sqlite').lower()
    if db_type == 'postgresql':
        return {
            'type': 'postgresql',
            'connection_string': os.getenv('POSTGRES_CONNECTION_STRING',
                'postgresql://postgres:password@localhost/water_monitoring')
        }
    elif db_type == 'mysql':
        return {
            'type': 'mysql',
            'connection_string': os.getenv('MYSQL_CONNECTION_STRING', 
                'mysql://root:password@localhost/water_monitoring')
        }
    elif db_type == 'victoriametrics':
        logging.info("VictoriaMetrics doesn't require schema migration")
        return {'type': 'victoriametrics'}
    elif db_type == 'influxdb':
        logging.info("InfluxDB doesn't require schema migration")
        return {'type': 'influxdb'}
    else:
        # Default to SQLite
        return {
            'type': 'sqlite',
            'db_path': os.getenv('SQLITE_DB_PATH', 'water_monitoring.db')
        }
 def main():
    """Main migration function"""
    logging.info("Starting geolocation column migration...")
    # Load configuration
    config = load_config_from_env()
    db_type = config['type']
    logging.info(f"Detected database type: {db_type.upper()}")
    success = False
    if db_type == 'sqlite':
        db_path = config.get('db_path', 'water_monitoring.db')
        if not os.path.exists(db_path):
            logging.error(f"Database file not found: {db_path}")
            sys.exit(1)
        success = migrate_sqlite(db_path)
    elif db_type == 'postgresql':
        success = migrate_postgresql(config['connection_string'])
    elif db_type == 'mysql':
        success = migrate_mysql(config['connection_string'])
    elif db_type in ['victoriametrics', 'influxdb']:
        logging.info(f"{db_type.upper()} doesn't require schema migration")
        success = True
    else:
        logging.error(f"Unsupported database type: {db_type}")
        sys.exit(1)
    if success:
        logging.info("✅ Migration completed successfully!")
        logging.info("You can now restart your water monitoring application")
        logging.info("The system will automatically use the new geolocation columns")
    else:
        logging.error("❌ Migration failed!")
        sys.exit(1)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,90 @@
 #!/usr/bin/env bash
 #
 # Retrain the flood forecast models safely. Run by water-monitor-retrain.timer
 # (monthly) or by hand:  sudo systemctl start water-monitor-retrain.service
 #
 # Why a script rather than ExecStart=train_flood_model.py:
 #   * train.py writes each station's bundle straight into models/ over ~12 min,
 #     and the API's hourly precompute reloads bundles by mtime. Training into
 #     a staging dir and mv-ing (atomic on one filesystem) means the API never
 #     sees a half-written joblib file or a mixed old/new set.
 #   * A run that produced gauge-only (v2) bundles, or trained too few stations,
 #     must NOT replace the deployed models. train.py already aborts on a
 #     missing rain series; this script re-checks the written metrics anyway.
 #   * No API restart is needed: predict.py reloads changed bundles on the next
 #     precompute (every scrape cycle, hourly), so the new models are live
 #     within an hour. Restart manually if you want them live immediately.
 #
 # Exit codes: 0 ok, 2 training refused (see log), 3 verification failed.
 set -euo pipefail
 APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
 PYTHON="${PYTHON:-${APP_DIR}/.venv/bin/python}"
 MODELS_DIR="${APP_DIR}/models"
 STAGE_DIR="${MODELS_DIR}/.staging"
 # P.4A is NOT_TRAINABLE by design (17% fill); 15 of 16 is the normal outcome.
 MIN_TRAINED="${MIN_TRAINED:-14}"
 EXPECT_VERSION_PREFIX="${EXPECT_VERSION_PREFIX:-hgb-v3+}"
 log() { printf '%s retrain: %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }
 cd "${APP_DIR}"
 [ -x "${PYTHON}" ] || { log "no interpreter at ${PYTHON} (run uv sync)"; exit 3; }
 rm -rf "${STAGE_DIR}"
 mkdir -p "${STAGE_DIR}"
 log "training into ${STAGE_DIR} (python=${PYTHON}, OMP_NUM_THREADS=${OMP_NUM_THREADS:-unset})"
 # train_flood_model.py exits 2 on a missing rain series (RainUnavailableError)
 # instead of silently writing v2 bundles -- propagate that unchanged.
 set +e
 "${PYTHON}" scripts/train_flood_model.py --stations all --models-dir "${STAGE_DIR}" "$@"
 rc=$?
 set -e
 if [ "${rc}" -ne 0 ]; then
    log "training failed (exit ${rc}); deployed models untouched"
    rm -rf "${STAGE_DIR}"
    exit "${rc}"
 fi
 # Verify before promoting. Reads metrics.json from the stage dir.
 VERSION="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
 import json, sys
 m = json.load(open(sys.argv[1]))
 print(m["model_version"])
 PY
 )"
 TRAINED="$("${PYTHON}" - "${STAGE_DIR}/metrics.json" <<'PY'
 import json, sys
 m = json.load(open(sys.argv[1]))
 print(sum(1 for s in m["stations"].values() if s.get("status") == "trained"))
 PY
 )"
 log "staged model_version=${VERSION} trained_stations=${TRAINED}"
 case "${VERSION}" in
    "${EXPECT_VERSION_PREFIX}"*) ;;
    *)
        log "REFUSING to deploy: version '${VERSION}' does not start with '${EXPECT_VERSION_PREFIX}'"
        rm -rf "${STAGE_DIR}"
        exit 3
        ;;
 esac
 if [ "${TRAINED}" -lt "${MIN_TRAINED}" ]; then
    log "REFUSING to deploy: only ${TRAINED} stations trained (< ${MIN_TRAINED})"
    rm -rf "${STAGE_DIR}"
    exit 3
 fi
 # Promote: per-file rename is atomic; readers see either the old or the new
 # bundle, never a partial one. Keep one previous generation for rollback.
 mkdir -p "${MODELS_DIR}/.previous"
 for f in "${STAGE_DIR}"/flood_*.joblib "${STAGE_DIR}/metrics.json"; do
    name="$(basename "${f}")"
    if [ -f "${MODELS_DIR}/${name}" ]; then
        mv -f "${MODELS_DIR}/${name}" "${MODELS_DIR}/.previous/${name}"
    fi
    mv -f "${f}" "${MODELS_DIR}/${name}"
 done
 rm -rf "${STAGE_DIR}"
 log "deployed ${VERSION} (${TRAINED} stations); previous generation in models/.previous. The API picks it up on its next hourly precompute."
@@ -0,0 +1,62 @@
 """Summarise rolling-origin harness output side by side.
 Usage:
    uv run python scripts/summarize_eval.py models/eval_2026-09-12.json [more.json ...]
 Aggregates each (station, variant) across folds: mean MAE, mean flood-regime
 MAE, mean Brier, total false-alarm episodes, and every warning event with its
 first-alert lead and the 24 h-ahead peak error -- the operational numbers that
 decide whether a variant ships.
 """
 import json
 import statistics
 import sys
 from collections import OrderedDict
 def summarize(paths):
    for path in paths:
        results = json.load(open(path, encoding="utf-8"))
        print(f"\n##### {path}")
        for station in results:
            print(f"\n=== {station['station']} (warn {station['warn_thr']:.2f} m) ===")
            agg = OrderedDict()
            for fold in station["folds"]:
                for name, m in fold["variants"].items():
                    a = agg.setdefault(
                        name, {"mae": [], "mae_hi": [], "brier": [], "fa": 0, "events": []}
                    )
                    a["mae"].append(m["mae"])
                    if m.get("mae_above_2p5") is not None:
                        a["mae_hi"].append(m["mae_above_2p5"])
                    if m.get("brier_warn") is not None:
                        a["brier"].append(m["brier_warn"])
                    a["fa"] += m["false_alarm_episodes"]
                    for e in m["events"]:
                        err = (
                            None
                            if e["peak_pred_24h_before"] is None
                            else e["peak_pred_24h_before"] - e["peak_level"]
                        )
                        a["events"].append((fold["year"], e["crossing"][:10], e["lead_h"], e["peak_level"], err))
            print(f"{'variant':22} {'MAE':>6} {'MAE_hi':>7} {'Brier':>7} {'FA':>3}  events: year crossing lead_h peak(err24h)")
            for name, a in agg.items():
                ev = "  ".join(
                    f"{y} {d} {'—' if l is None else format(l, '+.0f')}h {p:.2f}({'—' if err is None else format(err, '+.2f')})"
                    for y, d, l, p, err in a["events"]
                )
                leads = [l for *_, l, _, _ in a["events"] if l is not None]
                print(
                    f"{name:22} {statistics.mean(a['mae']):6.3f} "
                    f"{statistics.mean(a['mae_hi']) if a['mae_hi'] else float('nan'):7.3f} "
                    f"{statistics.mean(a['brier']) if a['brier'] else float('nan'):7.4f} "
                    f"{a['fa']:>3}  {ev}"
                )
                if leads:
                    print(f"{'':22} lead: mean {statistics.mean(leads):+.1f} h, min {min(leads):+.0f} h, "
                          f"missed {sum(1 for *_, l, _, _ in a['events'] if l is None)}/{len(a['events'])}")
 if __name__ == "__main__":
    summarize(sys.argv[1:] or ["models/eval_variants.json"])
@@ -11,7 +11,7 @@ import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
-from src.ml.train import main
+from src.ml.train import cli
 if __name__ == "__main__":
-    main()
+    raise SystemExit(cli())
@@ -0,0 +1,39 @@
 [Unit]
 Description=Retrain the Ping River flood forecast models
 Documentation=https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/-/blob/master/docs/FLOOD_FORECASTING.md
 After=network-online.target
 Wants=network-online.target
 [Service]
 Type=oneshot
 User=water-monitor
 Group=water-monitor
 WorkingDirectory=/opt/thailand-water-monitor
 EnvironmentFile=/opt/thailand-water-monitor/.env
 # Same interpreter as water-monitor.service -- the uv-managed .venv.
 # scripts/retrain.sh trains into models/.staging, refuses to promote anything
 # that is not a rain-enabled (hgb-v3) set covering the expected stations, then
 # renames the bundles into place. The API reloads them on its next hourly
 # precompute; no restart, so a failed run leaves the old models serving.
 ExecStart=/bin/bash /opt/thailand-water-monitor/scripts/retrain.sh
 # HistGradientBoosting is CPU-bound; cap threads so training cannot starve
 # the API (docs/FLOOD_FORECASTING.md section 6 measured 4 as the sweet spot).
 Environment=OMP_NUM_THREADS=4
 Environment=PYTHONPATH=/opt/thailand-water-monitor
 Environment=PYTHONUNBUFFERED=1
 Nice=15
 IOSchedulingClass=idle
 # 15 stations at ~50 s each plus data load: 12 min observed on 2026-09-12.
 TimeoutStartSec=45min
 # Same sandbox as the API unit.
 NoNewPrivileges=true
 PrivateTmp=true
 ProtectSystem=strict
 ProtectHome=true
 ReadWritePaths=/opt/thailand-water-monitor
 CapabilityBoundingSet=
 StandardOutput=journal
 StandardError=journal
 SyslogIdentifier=water-monitor-retrain
@@ -0,0 +1,17 @@
 [Unit]
 Description=Monthly flood-model retrain (docs/FLOOD_FORECASTING.md section 7)
 [Timer]
 # Policy: at minimum once pre-monsoon (May-June), monthly through the season
 # (July-November), and after any major flood. A retrain costs ~12 min and RAM
 # peaks ~300 MB, so running it every month all year is cheaper than remembering
 # which months matter. 1st of the month, 03:30 server-local -- between the
 # hourly scrapes and outside Thai daytime traffic.
 OnCalendar=*-*-01 03:30:00
 # Catch up if the box was off at the scheduled time.
 Persistent=true
 RandomizedDelaySec=20min
 Unit=water-monitor-retrain.service
 [Install]
 WantedBy=timers.target
@@ -9,17 +9,19 @@ Type=simple
 User=water-monitor
 Group=water-monitor
 WorkingDirectory=/opt/thailand-water-monitor
-ExecStart=/opt/thailand-water-monitor/venv/bin/python src/water_scraper_v3.py
+# The uv-managed env (uv sync -> .venv). Same interpreter for water-monitor-retrain.service.
 ExecStart=/opt/thailand-water-monitor/.venv/bin/python run.py --web-api
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=60
 TimeoutStopSec=30
-# Environment variables
+# DB_TYPE / POSTGRES_CONNECTION_STRING / MATRIX_* come from the .env file.
-Environment=DB_TYPE=victoriametrics
+EnvironmentFile=/opt/thailand-water-monitor/.env
 Environment=VM_HOST=localhost
 Environment=VM_PORT=8428
 Environment=PYTHONPATH=/opt/thailand-water-monitor
 # Serving path is latency-bound; single-threaded BLAS is 2.6x faster per call
 # (docs/FLOOD_FORECASTING.md section 6). Training sets its own value.
 Environment=OMP_NUM_THREADS=1
 Environment=PYTHONUNBUFFERED=1
 # Security settings
@@ -1,106 +0,0 @@
 #!/usr/bin/env python3
 """
 Setup script for Northern Thailand Ping River Monitor
 """
 from setuptools import setup, find_packages
 import os
 # Read the README file
 with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()
 # Read requirements
 try:
    with open("requirements.txt", "r", encoding="utf-8") as fh:
        requirements = [line.strip() for line in fh if line.strip() and not line.startswith("#")]
 except FileNotFoundError:
    # Fallback to minimal requirements if file not found
    requirements = [
        "requests>=2.31.0",
        "schedule>=1.2.0",
        "pandas>=2.1.0",
        "fastapi>=0.104.0",
        "uvicorn>=0.24.0",
    ]
 # Extract core requirements (exclude dev dependencies)
 core_requirements = []
 for req in requirements:
    if not any(dev_keyword in req.lower() for dev_keyword in ['pytest', 'black', 'flake8', 'mypy', 'sphinx']):
        core_requirements.append(req)
 setup(
    name="northern-thailand-ping-river-monitor",
    version="3.1.3",
    author="Ping River Monitor Team",
    author_email="contact@example.com",
    description="Real-time water level monitoring system for the Ping River Basin in Northern Thailand",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor",
    project_urls={
        "Bug Tracker": "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues",
        "Documentation": "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/wiki",
        "Source Code": "https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor",
    },
    packages=find_packages(),
    classifiers=[
        "Development Status :: 4 - Beta",
        "Intended Audience :: Science/Research",
        "Intended Audience :: System Administrators",
        "Topic :: Scientific/Engineering :: Hydrology",
        "Topic :: System :: Monitoring",
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
        "Programming Language :: Python :: 3.11",
        "Programming Language :: Python :: 3.12",
        "Operating System :: OS Independent",
        "Environment :: Web Environment",
        "Framework :: FastAPI",
    ],
    python_requires=">=3.9",
    install_requires=core_requirements,
    extras_require={
        "dev": [
            "pytest>=7.4.3",
            "pytest-cov>=4.1.0",
            "black>=23.11.0",
            "flake8>=6.1.0",
            "mypy>=1.7.1",
            "pre-commit>=3.5.0",
        ],
        "docs": [
            "sphinx>=7.2.6",
            "sphinx-rtd-theme>=1.3.0",
        ],
        "all": [
            "influxdb>=5.3.1",
            "pymysql>=1.1.0",
            "psycopg2-binary>=2.9.9",
        ],
    },
    entry_points={
        "console_scripts": [
            "ping-river-monitor=src.main:main",
            "ping-river-api=src.web_api:main",
        ],
    },
    include_package_data=True,
    package_data={
        "src": ["*.py"],
    },
    keywords=[
        "water monitoring",
        "hydrology",
        "thailand",
        "ping river",
        "environmental monitoring",
        "time series",
        "fastapi",
        "real-time data",
    ],
    zip_safe=False,
 )
@@ -12,9 +12,13 @@ __description__ = "Northern Thailand Ping River Monitoring System"
 from .config import Config
 from .database_adapters import DatabaseAdapter, create_database_adapter
-from .exceptions import (APIConnectionError, ConfigurationError,
+from .exceptions import (
-                         DatabaseConnectionError, DataValidationError,
+    APIConnectionError,
-                         WaterMonitorException)
+    ConfigurationError,
    DatabaseConnectionError,
    DataValidationError,
    WaterMonitorException,
 )
 from .models import DatabaseConfig, StationInfo, WaterMeasurement
 from .water_scraper_v3 import EnhancedWaterMonitorScraper
@@ -155,9 +155,9 @@ class MatrixNotifier:
        if alert.message:
            message += f"\n**Details:** {alert.message}\n"
-        # Add Grafana public dashboard link
+        # Add live dashboard link
-        grafana_url = "https://metrics.b4l.co.th/public-dashboards/655730aa044f44f49b355d01386018ca"
+        dashboard_url = os.getenv("ALERT_DASHBOARD_URL", "https://water.buildfor.life/")
-        message += f"\n📈 **View Dashboard:** {grafana_url}"
+        message += f"\n📈 **View Dashboard:** {dashboard_url}"
        return self.send_message(message)
@@ -38,6 +38,17 @@ class Config:
    TARGET_URL = "https://hyd-app-db.rid.go.th/hydro1h.html"
    API_URL = "https://hyd-app-db.rid.go.th/webservice/getGroupHourlyWaterLevelReportAllHL.ashx"
    THAIWATER_API_KEY = os.getenv("THAIWATER_API_KEY")
    # Public flood notifications (ntfy). Off unless NTFY_SERVER is set.
    # NTFY_SERVER is what subscribers use (public https URL, shown on the
    # dashboard). NTFY_PUBLISH_URL is where the monitor POSTs; defaults to
    # NTFY_SERVER, set it to http://127.0.0.1:2586 when ntfy runs on the same
    # host so publishing never depends on DNS/proxy/tunnel being up.
    NTFY_SERVER = os.getenv("NTFY_SERVER", "").strip()
    NTFY_PUBLISH_URL = os.getenv("NTFY_PUBLISH_URL", "").strip() or NTFY_SERVER
    NTFY_TOPIC_PREFIX = os.getenv("NTFY_TOPIC_PREFIX", "ping").strip()
    NTFY_TOKEN = os.getenv("NTFY_TOKEN", "").strip()  # publish token if ACL enabled
    PUBLIC_URL = os.getenv("PUBLIC_URL", "https://water.buildfor.life/").strip()
    REQUEST_TIMEOUT = int(os.getenv("REQUEST_TIMEOUT", "30"))
    USER_AGENT = (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
@@ -78,6 +89,49 @@ class Config:
    # MySQL settings
    MYSQL_CONNECTION_STRING = os.getenv("MYSQL_CONNECTION_STRING")
    # HII/ThaiWater open api-v3 collection (rainfall + backup water level)
    # See docs/DATA_SOURCES.md. Requires a SQL DB_TYPE (sqlite/postgresql/mysql).
    ENABLE_HII_COLLECTION = os.getenv("ENABLE_HII_COLLECTION", "true").lower() in (
        "1",
        "true",
        "yes",
    )
    HII_BASIN_CODE = int(os.getenv("HII_BASIN_CODE", "6"))  # 6 = Ping Basin
    # RID large-dam daily status (app.rid.go.th/reservoir) — Mae Ngat et al.
    ENABLE_RESERVOIR_COLLECTION = os.getenv(
        "ENABLE_RESERVOIR_COLLECTION", "true"
    ).lower() in ("1", "true", "yes")
    # TTL for the /api/hii/*/latest response cache; source data changes hourly
    HII_CACHE_TTL_SECONDS = int(os.getenv("HII_CACHE_TTL_SECONDS", "120"))
    # TTL for the /measurements/latest response cache (hottest endpoint)
    LATEST_CACHE_TTL_SECONDS = int(os.getenv("LATEST_CACHE_TTL_SECONDS", "45"))
    # TTL for /health check results (includes an external RID-API probe)
    HEALTH_CACHE_TTL_SECONDS = int(os.getenv("HEALTH_CACHE_TTL_SECONDS", "30"))
    # Thread-pool size for blocking work in the web process (DB queries,
    # inference, health probes). Waiting threads are cheap; starving the pool
    # stalls every endpoint that needs a thread.
    EXECUTOR_THREADS = int(os.getenv("EXECUTOR_THREADS", "48"))
    # Web server worker processes. Above 1, uvicorn forks workers and a
    # localhost lock port elects a single background-collection leader.
    WEB_WORKERS = int(os.getenv("WEB_WORKERS", "2"))
    COLLECTION_LEADER_PORT = int(os.getenv("COLLECTION_LEADER_PORT", "8901"))
    # Umami analytics (self-hosted). The website id is public (it ships in the
    # dashboard <script> tag); server-side API tracking posts to /api/send.
    UMAMI_API_URL = os.getenv("UMAMI_API_URL", "https://stats.buildfor.life/api/send")
    UMAMI_WEBSITE_ID = os.getenv(
        "UMAMI_WEBSITE_ID", "00b2be73-8f5f-4400-9029-3be852eb08f7"
    )
    UMAMI_TRACK_API = os.getenv("UMAMI_TRACK_API", "true").lower() in (
        "1",
        "true",
        "yes",
    )
    # Scheduler settings
    SCRAPING_INTERVAL_HOURS = int(os.getenv("SCRAPING_INTERVAL_HOURS", "1"))
@@ -36,6 +36,34 @@ class DatabaseAdapter(ABC):
    def get_measurements_for_date(self, target_date: datetime.datetime) -> List[Dict]:
        pass
    def get_measurement_date_range(
        self,
    ) -> Optional[tuple]:
        """Return (min_timestamp, max_timestamp) of stored measurements.
        Returns None when the backend has no data or does not support the query.
        """
        return None
    def get_recorded_hours_by_day(
        self, start_date: datetime.date, end_date: datetime.date
    ) -> Optional[Dict[datetime.date, set]]:
        """Map each day in [start_date, end_date] to the set of hours (0-23)
        that have at least one measurement.
        Returns None when the backend does not support hour-granular gap
        detection (callers should fall back to day-granular checks).
        """
        return None
    def get_database_stats(self) -> Optional[Dict]:
        """Summary statistics over stored measurements: total count, distinct
        stations, first/last timestamp, and hourly-slot coverage.
        Returns None when the backend has no data or does not support the query.
        """
        return None
 # InfluxDB Adapter
 class InfluxDBAdapter(DatabaseAdapter):
@@ -111,12 +139,16 @@ class InfluxDBAdapter(DatabaseAdapter):
                    "time": measurement["timestamp"].isoformat(),
                    "fields": {
                        "water_level": float(measurement["water_level"]),
-                        "discharge": float(measurement["discharge"])
+                        "discharge": (
-                        if measurement.get("discharge") is not None
+                            float(measurement["discharge"])
-                        else None,
+                            if measurement.get("discharge") is not None
-                        "discharge_percent": float(measurement["discharge_percent"])
+                            else None
-                        if measurement.get("discharge_percent")
+                        ),
-                        else None,
+                        "discharge_percent": (
                            float(measurement["discharge_percent"])
                            if measurement.get("discharge_percent")
                            else None
                        ),
                    },
                }
                points.append(point)
@@ -523,13 +555,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_code": row[1],
                            "station_name_en": row[2],
                            "station_name_th": row[3],
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -583,13 +615,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_code": row[1],
                            "station_name_en": row[2],
                            "station_name_th": row[3],
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -638,13 +670,13 @@ class SQLAdapter(DatabaseAdapter):
                            "station_id": row[1],
                            "station_code": row[2] or f"Station_{row[1]}",
                            "station_name_th": row[3] or f"Station {row[1]}",
-                            "water_level": float(row[4])
+                            "water_level": (
-                            if row[4] is not None
+                                float(row[4]) if row[4] is not None else None
-                            else None,
+                            ),
                            "discharge": float(row[5]) if row[5] is not None else None,
-                            "discharge_percent": float(row[6])
+                            "discharge_percent": (
-                            if row[6] is not None
+                                float(row[6]) if row[6] is not None else None
-                            else None,
+                            ),
                            "status": row[7],
                        }
                    )
@@ -657,6 +689,151 @@ class SQLAdapter(DatabaseAdapter):
            )
            return []
    @staticmethod
    def _coerce_date(value) -> Optional[datetime.date]:
        """Normalize a DB-returned day value (str/date/datetime) to a date."""
        if value is None:
            return None
        if isinstance(value, datetime.datetime):
            return value.date()
        if isinstance(value, datetime.date):
            return value
        # SQLite returns strings, e.g. '2024-09-15'
        return datetime.datetime.strptime(str(value)[:10], "%Y-%m-%d").date()
    def get_measurement_date_range(self) -> Optional[tuple]:
        if not self.engine:
            return None
        try:
            from sqlalchemy import text
            query = "SELECT MIN(timestamp), MAX(timestamp) FROM water_measurements"
            with self.engine.connect() as conn:
                row = conn.execute(text(query)).fetchone()
            if not row or row[0] is None:
                return None
            def to_datetime(value):
                if isinstance(value, datetime.datetime):
                    return value
                return datetime.datetime.fromisoformat(str(value)[:19])
            return (to_datetime(row[0]), to_datetime(row[1]))
        except Exception as e:
            logging.error(f"Error querying {self.db_type.upper()} date range: {e}")
            return None
    def get_recorded_hours_by_day(
        self, start_date: datetime.date, end_date: datetime.date
    ) -> Optional[Dict[datetime.date, set]]:
        if not self.engine:
            return None
        try:
            from sqlalchemy import text
            if self.db_type == "sqlite":
                day_expr = "DATE(timestamp)"
                hour_expr = "CAST(strftime('%H', timestamp) AS INTEGER)"
            elif self.db_type == "postgresql":
                day_expr = "CAST(timestamp AS DATE)"
                hour_expr = "CAST(EXTRACT(HOUR FROM timestamp) AS INTEGER)"
            else:  # MySQL
                day_expr = "DATE(timestamp)"
                hour_expr = "HOUR(timestamp)"
            query = f"""
            SELECT {day_expr} AS day, {hour_expr} AS hour
            FROM water_measurements
            WHERE timestamp >= :start_time AND timestamp < :end_time
            GROUP BY {day_expr}, {hour_expr}
            """
            start_time = datetime.datetime.combine(start_date, datetime.time.min)
            end_time = datetime.datetime.combine(
                end_date + datetime.timedelta(days=1), datetime.time.min
            )
            hours_by_day: Dict[datetime.date, set] = {}
            with self.engine.connect() as conn:
                result = conn.execute(
                    text(query), {"start_time": start_time, "end_time": end_time}
                )
                for row in result:
                    day = self._coerce_date(row[0])
                    if day is None:
                        continue
                    hours_by_day.setdefault(day, set()).add(int(row[1]))
            return hours_by_day
        except Exception as e:
            logging.error(f"Error querying {self.db_type.upper()} recorded hours: {e}")
            return None
    def get_database_stats(self) -> Optional[Dict]:
        if not self.engine:
            return None
        try:
            from sqlalchemy import text
            if self.db_type == "sqlite":
                slot_expr = "strftime('%Y-%m-%d %H', timestamp)"
            elif self.db_type == "postgresql":
                slot_expr = "TO_CHAR(timestamp, 'YYYY-MM-DD HH24')"
            else:  # MySQL
                # %-free expression: a bare % inside text() breaks as soon as the
                # query gains a bind parameter (pyformat interpolation)
                slot_expr = "CONCAT(DATE(timestamp), ' ', HOUR(timestamp))"
            query = f"""
            SELECT COUNT(*),
                   COUNT(DISTINCT station_id),
                   MIN(timestamp),
                   MAX(timestamp),
                   COUNT(DISTINCT {slot_expr})
            FROM water_measurements
            """
            with self.engine.connect() as conn:
                row = conn.execute(text(query)).fetchone()
            if not row or not row[0]:
                return None
            def to_datetime(value):
                if isinstance(value, datetime.datetime):
                    return value
                return datetime.datetime.fromisoformat(str(value)[:19])
            first_ts = to_datetime(row[2])
            last_ts = to_datetime(row[3])
            # Truncate to the hour before differencing so the slot count matches
            # the DISTINCT day-hour slots and coverage cannot exceed 100%
            first_slot = first_ts.replace(minute=0, second=0, microsecond=0)
            last_slot = last_ts.replace(minute=0, second=0, microsecond=0)
            expected_hours = int((last_slot - first_slot).total_seconds() // 3600) + 1
            recorded_hours = int(row[4])
            coverage_percent = round(100.0 * recorded_hours / expected_hours, 1)
            return {
                "total_measurements": int(row[0]),
                "station_count": int(row[1]),
                "first_timestamp": first_ts,
                "last_timestamp": last_ts,
                "recorded_hours": recorded_hours,
                "expected_hours": expected_hours,
                "coverage_percent": coverage_percent,
            }
        except Exception as e:
            logging.error(f"Error querying {self.db_type.upper()} stats: {e}")
            return None
 # VictoriaMetrics Adapter (using Prometheus format)
 class VictoriaMetricsAdapter(DatabaseAdapter):
@@ -0,0 +1,172 @@
 """Persistence for issued flood forecasts.
 Every background precompute stores what the deployed model predicted at that
 moment — predicted 24/12/6 h peak, warning/danger probabilities, model
 version. Keyed by (as_of, station, horizon), so hourly data yields one row
 per station-horizon per hour regardless of how often the precompute runs.
 This is the operational record that lets "predicted vs actual" be graphed
 later without retraining historical models.
 """
 import datetime
 import logging
 from typing import Dict, List, Optional
 logger = logging.getLogger(__name__)
 class ForecastHistoryStore:
    """SQL store (sqlite / postgresql / mysql), same pattern as HiiStore."""
    def __init__(self, connection_string: str, db_type: str):
        self.db_type = db_type.lower()
        if self.db_type not in ("sqlite", "postgresql", "mysql"):
            raise ValueError(
                f"Forecast history requires a SQL database, got '{db_type}'"
            )
        self.connection_string = connection_string
        self.engine = None
    def connect(self) -> bool:
        try:
            from sqlalchemy import create_engine, text
            self.engine = create_engine(self.connection_string, pool_pre_ping=True)
            ddl = """
                CREATE TABLE IF NOT EXISTS forecast_history (
                    as_of TIMESTAMP NOT NULL,
                    station_code VARCHAR(10) NOT NULL,
                    horizon_hours INTEGER NOT NULL,
                    predicted_max_level NUMERIC(8,3),
                    p_warning NUMERIC(7,5),
                    p_danger NUMERIC(7,5),
                    current_level NUMERIC(8,3),
                    model_version VARCHAR(64),
                    source VARCHAR(16),
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                    PRIMARY KEY (as_of, station_code, horizon_hours)
                )
            """
            index = (
                "CREATE INDEX IF NOT EXISTS idx_forecast_history_station "
                "ON forecast_history(station_code, as_of)"
            )
            with self.engine.begin() as conn:
                conn.execute(text(ddl))
                if self.db_type != "mysql":  # MySQL lacks IF NOT EXISTS for indexes
                    conn.execute(text(index))
            return True
        except Exception as error:
            logger.error(f"ForecastHistoryStore failed to connect: {error}")
            self.engine = None
            return False
    def save_rows(self, rows: List[Dict]) -> int:
        """Upsert forecast rows as returned by ml.predict (idempotent)."""
        if not rows:
            return 0
        if not self.engine and not self.connect():
            return 0
        from sqlalchemy import text
        cols = (
            "(as_of, station_code, horizon_hours, predicted_max_level, "
            "p_warning, p_danger, current_level, model_version, source)"
        )
        values = (
            "(:as_of, :station_code, :horizon_hours, :predicted_max_level, "
            ":p_warning, :p_danger, :current_level, :model_version, :source)"
        )
        update_cols = (
            "predicted_max_level",
            "p_warning",
            "p_danger",
            "current_level",
            "model_version",
            "source",
        )
        if self.db_type == "mysql":
            updates = ", ".join(f"{c} = VALUES({c})" for c in update_cols)
            sql = (
                f"INSERT INTO forecast_history {cols} VALUES {values} "
                f"ON DUPLICATE KEY UPDATE {updates}"
            )
        else:
            updates = ", ".join(f"{c} = EXCLUDED.{c}" for c in update_cols)
            sql = (
                f"INSERT INTO forecast_history {cols} VALUES {values} "
                f"ON CONFLICT (as_of, station_code, horizon_hours) "
                f"DO UPDATE SET {updates}"
            )
        params = []
        for row in rows:
            as_of = row.get("as_of")
            if isinstance(as_of, str):
                as_of = datetime.datetime.fromisoformat(as_of)
            if as_of is None or row.get("station_code") is None:
                continue
            params.append(
                {
                    "as_of": as_of,
                    "station_code": row["station_code"],
                    "horizon_hours": row.get("horizon_hours"),
                    "predicted_max_level": row.get("predicted_max_level"),
                    "p_warning": row.get("p_warning"),
                    "p_danger": row.get("p_danger"),
                    "current_level": row.get("current_level"),
                    "model_version": row.get("model_version"),
                    "source": row.get("source"),
                }
            )
        if not params:
            return 0
        try:
            with self.engine.begin() as conn:
                conn.execute(text(sql), params)
            return len(params)
        except Exception as error:
            logger.error(f"ForecastHistoryStore save failed: {error}")
            return 0
    def fetch(
        self,
        station_code: str,
        start: Optional[datetime.datetime] = None,
        end: Optional[datetime.datetime] = None,
        horizon_hours: Optional[int] = None,
        limit: int = 5000,
    ) -> List[Dict]:
        """Issued forecasts for one station, ascending by as_of."""
        if not self.engine and not self.connect():
            return []
        from sqlalchemy import text
        clauses = ["station_code = :code"]
        params: Dict = {"code": station_code, "limit": limit}
        if start is not None:
            clauses.append("as_of >= :start")
            params["start"] = start
        if end is not None:
            clauses.append("as_of <= :end")
            params["end"] = end
        if horizon_hours is not None:
            clauses.append("horizon_hours = :horizon")
            params["horizon"] = horizon_hours
        sql = (
            "SELECT as_of, station_code, horizon_hours, predicted_max_level, "
            "p_warning, p_danger, current_level, model_version, source "
            f"FROM forecast_history WHERE {' AND '.join(clauses)} "
            "ORDER BY as_of ASC, horizon_hours ASC LIMIT :limit"
        )
        try:
            with self.engine.connect() as conn:
                rows = [dict(r._mapping) for r in conn.execute(text(sql), params)]
            for row in rows:
                for key, value in row.items():
                    if hasattr(value, "is_finite"):  # Decimal -> float
                        row[key] = float(value)
            return rows
        except Exception as error:
            logger.error(f"ForecastHistoryStore fetch failed: {error}")
            return []
@@ -88,8 +88,11 @@ class DatabaseHealthCheck(HealthCheck):
            }
        try:
-            # Try to connect
+            # Connect only when there is no live engine yet: connect() re-runs
-            if hasattr(self.db_adapter, "connect"):
+            # the CREATE TABLE DDL suite, which is far too heavy per probe.
            if getattr(self.db_adapter, "engine", None) is None and hasattr(
                self.db_adapter, "connect"
            ):
                connected = self.db_adapter.connect()
                if not connected:
                    return {
@@ -122,9 +125,9 @@ class DatabaseHealthCheck(HealthCheck):
                "message": "Database connection OK",
                "details": {
                    "latest_data_count": len(latest_data),
-                    "latest_timestamp": str(latest_data[0].get("timestamp"))
+                    "latest_timestamp": (
-                    if latest_data
+                        str(latest_data[0].get("timestamp")) if latest_data else None
-                    else None,
+                    ),
                },
            }
@@ -140,6 +143,10 @@ class APIHealthCheck(HealthCheck):
    def __init__(self, api_url: str, session, name: str = "api"):
        super().__init__(name)
        # A liveness probe should fail fast: the default 30s timeout meant a
        # slow upstream pinned executor threads for longer than the /health
        # cache TTL, so the pool never drained under load.
        self.timeout_seconds = 5
        self.api_url = api_url
        self.session = session
@@ -0,0 +1,220 @@
 """Backfill historical water levels from the HII waterlevel_graph endpoint.
 The api-v3 waterlevel_graph archive reaches back to ~2019 with hourly
 wl_msl + discharge. This module walks a date range in chunks per station and
 upserts into hii_waterlevel (idempotent; safe to re-run and to overlap with
 the live snapshot collector). Station metadata comes from a live
 waterlevel_load fetch, so hii_wl_stations is populated/refreshed as a side
 effect.
 Usage: python scripts/backfill_hii_waterlevel.py --start 2019-01-01
 """
 import argparse
 import datetime
 import logging
 import time
 from typing import Dict, List, Optional
 from .hii_collector import (
    PING_BASIN_CODE,
    HiiClient,
    HiiStore,
    _parse_datetime,
    _to_float,
 )
 logger = logging.getLogger(__name__)
 DEFAULT_START = datetime.date(2019, 1, 1)
 DEFAULT_CHUNK_DAYS = 365  # full-year windows verified working (8,760 rows, ~700KB)
 DEFAULT_SLEEP_SECONDS = 1.0
 def parse_graph_rows(payload: Dict) -> List[Dict]:
    """Extract history rows from a waterlevel_graph payload (skips empty rows)."""
    rows = (payload.get("data") or {}).get("graph_data") or []
    records = []
    for row in rows:
        timestamp = _parse_datetime(row.get("datetime"))
        wl_msl = _to_float(row.get("value"))
        discharge = _to_float(row.get("discharge"))
        if timestamp is None or (wl_msl is None and discharge is None):
            continue
        records.append(
            {"timestamp": timestamp, "wl_msl": wl_msl, "discharge": discharge}
        )
    return records
 def fetch_waterlevel_history(
    client: HiiClient,
    station_id: int,
    start_date: datetime.date,
    end_date: datetime.date,
 ) -> List[Dict]:
    """Hourly wl_msl + discharge history (archive reaches back to ~2019)."""
    payload = client.get(
        "waterlevel_graph",
        params={
            "station_type": "tele_waterlevel",
            "station_id": station_id,
            "start_date": start_date.isoformat(),
            "end_date": end_date.isoformat(),
        },
    )
    return parse_graph_rows(payload)
 def chunk_date_range(
    start: datetime.date, end: datetime.date, chunk_days: int
 ) -> List[tuple]:
    """Split [start, end] into inclusive (start, end) windows."""
    chunks = []
    cursor = start
    while cursor <= end:
        chunk_end = min(cursor + datetime.timedelta(days=chunk_days - 1), end)
        chunks.append((cursor, chunk_end))
        cursor = chunk_end + datetime.timedelta(days=1)
    return chunks
 def select_stations(
    station_records: List[Dict],
    codes: Optional[List[str]] = None,
    all_stations: bool = False,
 ) -> List[Dict]:
    """Pick stations to backfill from parsed waterlevel_load records.
    Default: stations that mirror a RID gauge (rid_code) or are flagged
    is_key_station — the ones relevant to the flood model. Explicit codes
    match rid_code or oldcode; --all takes every station in the basin.
    """
    if all_stations:
        return station_records
    if codes:
        wanted = {c.strip().upper() for c in codes if c.strip()}
        return [
            r
            for r in station_records
            if (r.get("rid_code") or "").upper() in wanted
            or (r.get("oldcode") or "").upper() in wanted
        ]
    return [r for r in station_records if r.get("rid_code") or r.get("is_key_station")]
 def backfill(
    store: HiiStore,
    client: Optional[HiiClient] = None,
    start: datetime.date = DEFAULT_START,
    end: Optional[datetime.date] = None,
    codes: Optional[List[str]] = None,
    all_stations: bool = False,
    chunk_days: int = DEFAULT_CHUNK_DAYS,
    sleep_seconds: float = DEFAULT_SLEEP_SECONDS,
    basin_code: int = PING_BASIN_CODE,
 ) -> Dict[str, int]:
    """Run the backfill; returns {'stations': n, 'rows': n, 'errors': n}."""
    client = client or HiiClient()
    end = end or datetime.date.today()
    logger.info("Fetching station catalog from waterlevel_load...")
    station_records = client.fetch_waterlevel(basin_code)
    # Refresh station metadata (and today's snapshot) while we have it
    store.save_waterlevel(station_records)
    stations = select_stations(station_records, codes=codes, all_stations=all_stations)
    if not stations:
        logger.error("No stations matched the selection")
        return {"stations": 0, "rows": 0, "errors": 0}
    chunks = chunk_date_range(start, end, chunk_days)
    logger.info(
        f"Backfilling {len(stations)} stations x {len(chunks)} windows "
        f"({start} .. {end}, {chunk_days}-day chunks)"
    )
    totals = {"stations": len(stations), "rows": 0, "errors": 0}
    for station in stations:
        sid = station["station_id"]
        label = station.get("rid_code") or station.get("oldcode") or str(sid)
        station_rows = 0
        for chunk_start, chunk_end in chunks:
            try:
                rows = fetch_waterlevel_history(client, sid, chunk_start, chunk_end)
                station_rows += store.save_waterlevel_history(sid, rows)
            except Exception as e:
                totals["errors"] += 1
                logger.warning(f"{label}: {chunk_start}..{chunk_end} failed: {e}")
            time.sleep(sleep_seconds)
        totals["rows"] += station_rows
        logger.info(f"{label} (id {sid}): {station_rows} rows saved")
    logger.info(
        f"Backfill complete: {totals['rows']} rows across "
        f"{totals['stations']} stations, {totals['errors']} failed windows"
    )
    return totals
 def main(argv: Optional[List[str]] = None) -> bool:
    parser = argparse.ArgumentParser(
        description="Backfill hii_waterlevel from the HII waterlevel_graph archive"
    )
    parser.add_argument(
        "--start",
        type=datetime.date.fromisoformat,
        default=DEFAULT_START,
        help=f"First date to fetch (default {DEFAULT_START})",
    )
    parser.add_argument(
        "--end",
        type=datetime.date.fromisoformat,
        default=None,
        help="Last date to fetch (default today)",
    )
    parser.add_argument(
        "--stations",
        help="Comma-separated codes (rid_code or oldcode, e.g. P.1,P.67,CHM004). "
        "Default: all RID-mirror and key stations",
    )
    parser.add_argument(
        "--all",
        action="store_true",
        help="Backfill every Ping-basin station (125+; slow)",
    )
    parser.add_argument(
        "--chunk-days", type=int, default=DEFAULT_CHUNK_DAYS, help="Window size"
    )
    parser.add_argument(
        "--sleep",
        type=float,
        default=DEFAULT_SLEEP_SECONDS,
        help="Pause between requests in seconds",
    )
    args = parser.parse_args(argv)
    logging.basicConfig(
        level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
    )
    from .config import Config
    db_config = Config.get_database_config()
    if db_config["type"] not in ("sqlite", "postgresql", "mysql"):
        logger.error(f"Backfill requires a SQL DB_TYPE, got '{db_config['type']}'")
        return False
    store = HiiStore(db_config["connection_string"], db_config["type"])
    if not store.connect():
        return False
    totals = backfill(
        store,
        start=args.start,
        end=args.end,
        codes=args.stations.split(",") if args.stations else None,
        all_stations=args.all,
        chunk_days=args.chunk_days,
        sleep_seconds=args.sleep,
    )
    return totals["rows"] > 0 and totals["errors"] == 0
@@ -0,0 +1,478 @@
 """Collector for HII/ThaiWater open api-v3 feeds (rainfall + water level).
 Polls the unauthenticated api-v3.thaiwater.net public endpoints, filters to the
 Ping basin, and persists to dedicated tables alongside the RID data:
 - hii_rain_stations / hii_rainfall        (rain_1h / rain_24h gauge telemetry)
 - hii_wl_stations   / hii_waterlevel      (independent water-level source, m MSL)
 Water levels are kept in a separate table (not a column on water_measurements)
 because HII reports in m MSL from a different station universe; the per-station
 ``offset`` column (gauge zero in m MSL) converts to gauge datum when needed.
 See docs/DATA_SOURCES.md for the endpoint catalog and quirks.
 """
 import datetime
 import logging
 import re
 from typing import Any, Dict, List, Optional
 import requests
 logger = logging.getLogger(__name__)
 HII_API_BASE = "https://api-v3.thaiwater.net/api/v1/thaiwater30/public"
 PING_BASIN_CODE = 6
 # Matches 'P.1', 'ridhydro_P.67', 'ridtele_TUP.14' -> canonical RID code suffix
 _RID_CODE_RE = re.compile(r"(?:^|_)(P\.\d+[A-Z]?)$")
 def _to_float(value: Any) -> Optional[float]:
    """API numerics arrive as strings ('335.00'), numbers, or None."""
    if value is None or value == "":
        return None
    try:
        return float(value)
    except (TypeError, ValueError):
        return None
 def _parse_datetime(value: Any) -> Optional[datetime.datetime]:
    """Timestamps are Thai local time, e.g. '2026-08-11 13:00'."""
    if not value:
        return None
    for fmt in ("%Y-%m-%d %H:%M", "%Y-%m-%d %H:%M:%S"):
        try:
            return datetime.datetime.strptime(value, fmt)
        except ValueError:
            continue
    return None
 def _name(station: Dict, lang: str) -> Optional[str]:
    name = station.get("tele_station_name")
    if isinstance(name, dict):
        return name.get(lang)
    return name if lang == "th" else None
 def rid_code_from_oldcode(oldcode: Optional[str]) -> Optional[str]:
    """Normalize a ThaiWater oldcode to the RID P-code it mirrors, if any."""
    if not oldcode:
        return None
    match = _RID_CODE_RE.search(oldcode)
    return match.group(1) if match else None
 def parse_rain_records(payload: Dict, basin_code: int = PING_BASIN_CODE) -> List[Dict]:
    """Extract per-station rainfall rows from a rain_24h payload."""
    records = []
    for row in payload.get("data") or []:
        basin = row.get("basin") or {}
        if basin.get("basin_code") != basin_code:
            continue
        station = row.get("station") or {}
        station_id = station.get("id")
        timestamp = _parse_datetime(row.get("rainfall_datetime"))
        if station_id is None or timestamp is None:
            continue
        records.append(
            {
                "station_id": station_id,
                "oldcode": station.get("tele_station_oldcode"),
                "name_th": _name(station, "th"),
                "name_en": _name(station, "en"),
                "latitude": _to_float(station.get("tele_station_lat")),
                "longitude": _to_float(station.get("tele_station_long")),
                "sub_basin_id": str(station.get("sub_basin_id") or "") or None,
                "agency": ((row.get("agency") or {}).get("agency_shortname") or {}).get(
                    "en"
                ),
                "timestamp": timestamp,
                "rain_1h": _to_float(row.get("rain_1h")),
                "rain_24h": _to_float(row.get("rain_24h")),
            }
        )
    return records
 def parse_waterlevel_records(
    payload: Dict, basin_code: int = PING_BASIN_CODE
 ) -> List[Dict]:
    """Extract per-station water-level rows from a waterlevel_load payload."""
    data = (payload.get("waterlevel_data") or {}).get("data") or []
    records = []
    for row in data:
        basin = row.get("basin") or {}
        if basin.get("basin_code") != basin_code:
            continue
        station = row.get("station") or {}
        station_id = station.get("id")
        timestamp = _parse_datetime(row.get("waterlevel_datetime"))
        if station_id is None or timestamp is None:
            continue
        oldcode = station.get("tele_station_oldcode")
        records.append(
            {
                "station_id": station_id,
                "oldcode": oldcode,
                "rid_code": rid_code_from_oldcode(oldcode),
                "name_th": _name(station, "th"),
                "name_en": _name(station, "en"),
                "latitude": _to_float(station.get("tele_station_lat")),
                "longitude": _to_float(station.get("tele_station_long")),
                "agency": ((row.get("agency") or {}).get("agency_shortname") or {}).get(
                    "en"
                ),
                "river_name": row.get("river_name"),
                "offset_msl": _to_float(station.get("offset")),
                "ground_level_msl": _to_float(station.get("ground_level")),
                "min_bank_msl": _to_float(station.get("min_bank")),
                "critical_level_msl": _to_float(station.get("critical_level_msl")),
                "critical_level_m": _to_float(station.get("critical_level_m")),
                "qmax": _to_float(station.get("qmax")),
                "is_key_station": bool(station.get("is_key_station")),
                "timestamp": timestamp,
                "wl_msl": _to_float(row.get("waterlevel_msl")),
                "wl_m": _to_float(row.get("waterlevel_m")),
                "discharge": _to_float(row.get("discharge")),
                "flow_rate": _to_float(row.get("flow_rate")),
                "storage_percent": _to_float(row.get("storage_percent")),
                "situation_level": row.get("situation_level"),
                "diff_wl_bank": _to_float(row.get("diff_wl_bank")),
            }
        )
    return records
 class HiiClient:
    """HTTP client for the open api-v3 public endpoints."""
    def __init__(
        self,
        base_url: str = HII_API_BASE,
        session: Optional[requests.Session] = None,
        timeout: int = 90,
    ):
        self.base_url = base_url.rstrip("/")
        self.session = session or requests.Session()
        self.timeout = timeout
    def get(self, endpoint: str, params: Optional[Dict] = None) -> Dict:
        response = self.session.get(
            f"{self.base_url}/{endpoint}", params=params, timeout=self.timeout
        )
        response.raise_for_status()
        return response.json()
    def fetch_rain(self, basin_code: int = PING_BASIN_CODE) -> List[Dict]:
        return parse_rain_records(self.get("rain_24h"), basin_code)
    def fetch_waterlevel(self, basin_code: int = PING_BASIN_CODE) -> List[Dict]:
        return parse_waterlevel_records(self.get("waterlevel_load"), basin_code)
 class HiiStore:
    """SQL persistence for HII feeds (sqlite / postgresql / mysql).
    Reuses the app's main relational database (same connection string as
    the RID tables) but writes to its own hii_* tables.
    """
    def __init__(self, connection_string: str, db_type: str):
        self.db_type = db_type.lower()
        if self.db_type not in ("sqlite", "postgresql", "mysql"):
            raise ValueError(f"HII collection requires a SQL database, got '{db_type}'")
        self.connection_string = connection_string
        self.engine = None
    def connect(self) -> bool:
        try:
            from sqlalchemy import create_engine
            self.engine = create_engine(self.connection_string, pool_pre_ping=True)
            self._create_tables()
            return True
        except Exception as e:
            logger.error(f"HiiStore failed to connect: {e}")
            self.engine = None
            return False
    def _create_tables(self):
        from sqlalchemy import text
        bool_type = "BOOLEAN" if self.db_type != "mysql" else "TINYINT(1)"
        ddl = [
            """
            CREATE TABLE IF NOT EXISTS hii_rain_stations (
                id INTEGER PRIMARY KEY,
                oldcode VARCHAR(60),
                name_th VARCHAR(255),
                name_en VARCHAR(255),
                latitude NUMERIC(10,6),
                longitude NUMERIC(10,6),
                sub_basin_id VARCHAR(10),
                agency VARCHAR(40),
                updated_at TIMESTAMP
            )
            """,
            # Composite natural PK (no surrogate id): TimescaleDB hypertable
            # conversion requires every unique index to include the time column.
            """
            CREATE TABLE IF NOT EXISTS hii_rainfall (
                station_id INTEGER NOT NULL,
                timestamp TIMESTAMP NOT NULL,
                rain_1h NUMERIC(7,2),
                rain_24h NUMERIC(8,2),
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                PRIMARY KEY (station_id, timestamp)
            )
            """,
            f"""
            CREATE TABLE IF NOT EXISTS hii_wl_stations (
                id INTEGER PRIMARY KEY,
                oldcode VARCHAR(60),
                rid_code VARCHAR(10),
                name_th VARCHAR(255),
                name_en VARCHAR(255),
                latitude NUMERIC(10,6),
                longitude NUMERIC(10,6),
                agency VARCHAR(40),
                river_name VARCHAR(255),
                offset_msl NUMERIC(8,3),
                ground_level_msl NUMERIC(8,3),
                min_bank_msl NUMERIC(8,3),
                critical_level_msl NUMERIC(8,3),
                critical_level_m NUMERIC(8,3),
                qmax NUMERIC(10,2),
                is_key_station {bool_type},
                updated_at TIMESTAMP
            )
            """,
            """
            CREATE TABLE IF NOT EXISTS hii_waterlevel (
                station_id INTEGER NOT NULL,
                timestamp TIMESTAMP NOT NULL,
                wl_msl NUMERIC(8,3),
                wl_m NUMERIC(8,3),
                discharge NUMERIC(10,2),
                flow_rate NUMERIC(10,2),
                storage_percent NUMERIC(6,2),
                situation_level INTEGER,
                diff_wl_bank NUMERIC(8,3),
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                PRIMARY KEY (station_id, timestamp)
            )
            """,
            "CREATE INDEX IF NOT EXISTS idx_hii_rainfall_ts ON hii_rainfall(timestamp)",
            "CREATE INDEX IF NOT EXISTS idx_hii_waterlevel_ts ON hii_waterlevel(timestamp)",
        ]
        # MySQL (<8.0.13 semantics) lacks CREATE INDEX IF NOT EXISTS; the unique
        # constraints already cover the hot (station_id, timestamp) lookups there.
        if self.db_type == "mysql":
            ddl = ddl[:4]
        with self.engine.begin() as conn:
            for statement in ddl:
                conn.execute(text(statement))
    def _upsert(self, table: str, key_cols: List[str], value_cols: List[str]) -> str:
        cols = key_cols + value_cols
        col_list = ", ".join(cols)
        params = ", ".join(f":{c}" for c in cols)
        if self.db_type == "sqlite":
            return f"INSERT OR REPLACE INTO {table} ({col_list}) VALUES ({params})"
        if self.db_type == "postgresql":
            updates = ", ".join(f"{c} = EXCLUDED.{c}" for c in value_cols)
            conflict = ", ".join(key_cols)
            return (
                f"INSERT INTO {table} ({col_list}) VALUES ({params}) "
                f"ON CONFLICT ({conflict}) DO UPDATE SET {updates}"
            )
        updates = ", ".join(f"{c} = VALUES({c})" for c in value_cols)
        return (
            f"INSERT INTO {table} ({col_list}) VALUES ({params}) "
            f"ON DUPLICATE KEY UPDATE {updates}"
        )
    def save_rain(self, records: List[Dict]) -> int:
        return self._save(
            records,
            station_table="hii_rain_stations",
            station_cols=[
                "oldcode",
                "name_th",
                "name_en",
                "latitude",
                "longitude",
                "sub_basin_id",
                "agency",
            ],
            measurement_table="hii_rainfall",
            measurement_cols=["rain_1h", "rain_24h"],
        )
    def save_waterlevel(self, records: List[Dict]) -> int:
        return self._save(
            records,
            station_table="hii_wl_stations",
            station_cols=[
                "oldcode",
                "rid_code",
                "name_th",
                "name_en",
                "latitude",
                "longitude",
                "agency",
                "river_name",
                "offset_msl",
                "ground_level_msl",
                "min_bank_msl",
                "critical_level_msl",
                "critical_level_m",
                "qmax",
                "is_key_station",
            ],
            measurement_table="hii_waterlevel",
            measurement_cols=[
                "wl_msl",
                "wl_m",
                "discharge",
                "flow_rate",
                "storage_percent",
                "situation_level",
                "diff_wl_bank",
            ],
        )
    def save_waterlevel_history(self, station_id: int, rows: List[Dict]) -> int:
        """Upsert backfilled history rows, touching only wl_msl and discharge.
        Live-snapshot rows for the same (station, hour) keep their extra
        columns (storage_percent, situation_level, ...) untouched.
        """
        if not rows:
            return 0
        if not self.engine and not self.connect():
            return 0
        from sqlalchemy import text
        cols = "(station_id, timestamp, wl_msl, discharge)"
        values = "(:station_id, :timestamp, :wl_msl, :discharge)"
        if self.db_type == "mysql":
            sql = (
                f"INSERT INTO hii_waterlevel {cols} VALUES {values} "
                "ON DUPLICATE KEY UPDATE wl_msl = VALUES(wl_msl), "
                "discharge = VALUES(discharge)"
            )
        else:  # sqlite (>=3.24) and postgresql share upsert syntax
            sql = (
                f"INSERT INTO hii_waterlevel {cols} VALUES {values} "
                "ON CONFLICT (station_id, timestamp) DO UPDATE SET "
                "wl_msl = EXCLUDED.wl_msl, discharge = EXCLUDED.discharge"
            )
        params = [{**row, "station_id": station_id} for row in rows]
        try:
            with self.engine.begin() as conn:
                conn.execute(text(sql), params)
            return len(params)
        except Exception as e:
            logger.error(f"HiiStore history save failed: {e}")
            return 0
    def _save(
        self,
        records: List[Dict],
        station_table: str,
        station_cols: List[str],
        measurement_table: str,
        measurement_cols: List[str],
    ) -> int:
        if not records:
            return 0
        if not self.engine and not self.connect():
            return 0
        from sqlalchemy import text
        now = datetime.datetime.now()
        station_sql = self._upsert(station_table, ["id"], station_cols + ["updated_at"])
        measurement_sql = self._upsert(
            measurement_table, ["station_id", "timestamp"], measurement_cols
        )
        # Dedupe stations (one row per station per snapshot anyway) and build
        # parameter dicts limited to each statement's columns.
        stations = {}
        measurements = []
        for record in records:
            sid = record["station_id"]
            station_row = {c: record.get(c) for c in station_cols}
            station_row.update({"id": sid, "updated_at": now})
            stations[sid] = station_row
            measurement_row = {c: record.get(c) for c in measurement_cols}
            measurement_row.update(
                {"station_id": sid, "timestamp": record["timestamp"]}
            )
            measurements.append(measurement_row)
        try:
            with self.engine.begin() as conn:
                conn.execute(text(station_sql), list(stations.values()))
                conn.execute(text(measurement_sql), measurements)
            return len(measurements)
        except Exception as e:
            logger.error(f"HiiStore save to {measurement_table} failed: {e}")
            return 0
 class HiiCollector:
    """Fetch + persist one snapshot of both HII feeds."""
    def __init__(
        self,
        db_config: Dict,
        basin_code: int = PING_BASIN_CODE,
        client: Optional[HiiClient] = None,
    ):
        self.client = client or HiiClient()
        self.basin_code = basin_code
        self.store = HiiStore(
            connection_string=db_config["connection_string"],
            db_type=db_config["type"],
        )
    def run_cycle(self) -> Dict[str, int]:
        """Collect both feeds; each is independent and failure-isolated."""
        counts = {"rainfall": 0, "waterlevel": 0}
        try:
            counts["rainfall"] = self.store.save_rain(
                self.client.fetch_rain(self.basin_code)
            )
        except Exception as e:
            logger.error(f"HII rainfall collection failed: {e}")
        try:
            counts["waterlevel"] = self.store.save_waterlevel(
                self.client.fetch_waterlevel(self.basin_code)
            )
        except Exception as e:
            logger.error(f"HII waterlevel collection failed: {e}")
        logger.info(
            f"HII collection: {counts['rainfall']} rainfall, "
            f"{counts['waterlevel']} waterlevel rows saved"
        )
        return counts
 def create_collector_from_config() -> Optional[HiiCollector]:
    """Build a collector from app Config; None when disabled or non-SQL DB."""
    from .config import Config
    if not Config.ENABLE_HII_COLLECTION:
        return None
    db_config = Config.get_database_config()
    if db_config["type"] not in ("sqlite", "postgresql", "mysql"):
        logger.warning(
            f"HII collection skipped: DB_TYPE '{db_config['type']}' is not SQL"
        )
        return None
    return HiiCollector(db_config, basin_code=Config.HII_BASIN_CODE)
@@ -86,6 +86,19 @@ def run_continuous_monitoring():
        alerting = WaterLevelAlertSystem()
        # Initialize HII/ThaiWater collector (rainfall + backup water level)
        hii_collector = None
        try:
            from .hii_collector import create_collector_from_config
            hii_collector = create_collector_from_config()
            if hii_collector:
                logger.info(
                    "HII collection enabled (Ping-basin rainfall + water level)"
                )
        except Exception as e:
            logger.error(f"HII collector initialization failed: {e}")
        # Setup signal handlers
        setup_signal_handlers(scraper)
@@ -108,6 +121,14 @@ def run_continuous_monitoring():
        retry_mode = not initial_success
        last_successful_fetch = None if not initial_success else datetime.now()
        last_hii_run = None
        if hii_collector:
            try:
                hii_collector.run_cycle()
                last_hii_run = datetime.now()
            except Exception as e:
                logger.error(f"HII collection failed: {e}")
        if retry_mode:
            logger.warning("No data fetched in initial run - entering retry mode")
            next_run = datetime.now() + timedelta(minutes=1)
@@ -126,6 +147,18 @@ def run_continuous_monitoring():
                logger.info("Running scheduled data collection...")
                success = scraper.run_scraping_cycle()
                # HII feeds update hourly; keep collecting on that cadence even
                # when the RID scraper is in 1-minute retry mode.
                if hii_collector and (
                    last_hii_run is None
                    or current_time - last_hii_run >= timedelta(minutes=55)
                ):
                    try:
                        hii_collector.run_cycle()
                        last_hii_run = current_time
                    except Exception as e:
                        logger.error(f"HII collection failed: {e}")
                if success:
                    last_successful_fetch = current_time
@@ -180,9 +213,33 @@ def run_continuous_monitoring():
    return True
-def run_gap_filling(days_back: int):
+def run_hii_collection():
-    """Run gap filling for missing data"""
+    """Run a single HII/ThaiWater collection cycle (rainfall + water level)"""
-    logger.info(f"Checking for data gaps in the last {days_back} days...")
+    try:
        Config.validate_config()
        from .hii_collector import create_collector_from_config
        collector = create_collector_from_config()
        if not collector:
            logger.error(
                "HII collection unavailable (disabled via ENABLE_HII_COLLECTION "
                "or DB_TYPE is not a SQL database)"
            )
            return False
        counts = collector.run_cycle()
        return counts["rainfall"] > 0 or counts["waterlevel"] > 0
    except Exception as e:
        logger.error(f"HII collection failed: {e}")
        return False
 def run_gap_filling(days_back: Optional[int]):
    """Run gap filling for missing data (days_back=None scans the whole range)"""
    if days_back is None:
        logger.info("Checking for data gaps across the whole data range...")
    else:
        logger.info(f"Checking for data gaps in the last {days_back} days...")
    try:
        # Validate configuration
@@ -287,15 +344,24 @@ def run_web_api():
    try:
        import uvicorn
        from .web_api import app
        # Validate configuration
        Config.validate_config()
-        # Run the server
+        workers = max(1, Config.WEB_WORKERS)
-        uvicorn.run(
+        if workers > 1:
-            app, host="0.0.0.0", port=8000, log_config=None  # Use our custom logging
+            # Multi-worker needs the app as an import string; a localhost lock
-        )
+            # port keeps background collection in exactly one worker.
            uvicorn.run(
                "src.web_api:app",
                host="0.0.0.0",
                port=8000,
                workers=workers,
                log_config=None,
            )
        else:
            from .web_api import app
            uvicorn.run(app, host="0.0.0.0", port=8000, log_config=None)
    except ImportError:
        logger.error("FastAPI not installed. Run: pip install fastapi uvicorn")
@@ -445,6 +511,7 @@ Examples:
  %(prog)s                          # Run continuous monitoring
  %(prog)s --web-api                # Start web API server
  %(prog)s --fill-gaps 7            # Fill missing data for last 7 days
  %(prog)s --fill-gaps all          # Fill missing data across the whole data range
  %(prog)s --update-data 2          # Update existing data for last 2 days
  %(prog)s --import-historical 2024-01-01 2024-01-31  # Import historical data
  %(prog)s --status                 # Show system status
@@ -461,9 +528,11 @@ Examples:
    parser.add_argument(
        "--fill-gaps",
-        type=int,
+        metavar="DAYS|all",
-        metavar="DAYS",
+        help=(
-        help="Fill missing data gaps for the specified number of days back",
+            "Fill missing data gaps for the specified number of days back, "
            "or 'all' to scan the entire data range in the database"
        ),
    )
    parser.add_argument(
@@ -498,6 +567,12 @@ Examples:
        "--alert-test", action="store_true", help="Send test alert message to Matrix"
    )
    parser.add_argument(
        "--collect-hii",
        action="store_true",
        help="Run one HII/ThaiWater collection cycle (rainfall + water level)",
    )
    parser.add_argument(
        "--log-level",
        choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
@@ -529,7 +604,17 @@ Examples:
        elif args.web_api:
            success = run_web_api()
        elif args.fill_gaps is not None:
-            success = run_gap_filling(args.fill_gaps)
+            if args.fill_gaps.lower() == "all":
                success = run_gap_filling(None)
            else:
                try:
                    success = run_gap_filling(int(args.fill_gaps))
                except ValueError:
                    logger.error(
                        f"Invalid --fill-gaps value '{args.fill_gaps}': "
                        "expected a number of days or 'all'"
                    )
                    sys.exit(1)
        elif args.update_data is not None:
            success = run_data_update(args.update_data)
        elif args.import_historical is not None:
@@ -542,6 +627,8 @@ Examples:
            success = run_alert_check()
        elif args.alert_test:
            success = run_alert_test()
        elif args.collect_hii:
            success = run_hii_collection()
        else:
            success = run_continuous_monitoring()
@@ -0,0 +1,109 @@
 """Mae Ngat reservoir series for the flood models.
 rid_reservoir_daily (collected hourly by src/rid_reservoir.py, backfilled to
 2018) holds daily storage/inflow/outflow for every RID large dam. Mae Ngat
 Somboon Chon (DAM_ID 200103) is the only large dam upstream of Chiang Mai:
 in Oct 2024 its inflow hit 19-22 MCM/day and storage 114% of usable capacity
 days around the P.1 crossing — upstream state no river gauge carries.
 Leakage rule: RID publishes the daily report for date D on the morning of D,
 so the row becomes visible to features at D 07:00 local time, never earlier.
 Forward-fill is capped at FFILL_LIMIT_H so a stalled collector degrades to
 NaN (HGB-native) instead of silently serving stale reservoir state.
 Known residual optimism: the collector upserts keep-last (and re-fetches
 yesterday), so the stored row for date D is RID's FINAL revision, which
 training then back-dates to D 07:00 — values live serving may not have had
 that morning. This bias works IN FAVOR of dam features, so the 2026-08-13
 negative result (they cost 1-3 h of alert lead) holds a fortiori; but any
 future POSITIVE result must first validate intraday row stability or shift
 the flow columns to D+1 07:00.
 """
 import datetime
 import logging
 from pathlib import Path
 from typing import Optional
 import pandas as pd
 from ..rid_reservoir import MAE_NGAT_DAM_ID
 from .data import CACHE_DIR, resolve_db_url
 logger = logging.getLogger(__name__)
 REPORT_HOUR = 7  # daily value valid from 07:00 local on its own date
 FFILL_LIMIT_H = 48  # two missed daily reports -> NaN, not stale state
 DAM_COLUMNS = ("storage_pct", "inflow_mcm", "outflow_mcm")
 CACHE_FILE = f"dam_{MAE_NGAT_DAM_ID}.csv.gz"
 def load_daily(
    db_url: Optional[str] = None,
    dam_id: str = MAE_NGAT_DAM_ID,
    start: Optional[datetime.date] = None,
    cache_dir: Path = CACHE_DIR,
 ) -> Optional[pd.DataFrame]:
    """Daily dam rows indexed by date. DB first, on-disk cache as fallback."""
    cache_path = Path(cache_dir) / CACHE_FILE
    resolved = resolve_db_url(db_url)
    if resolved:
        try:
            from sqlalchemy import create_engine, text
            query = (
                "SELECT date, storage_pct, inflow_mcm, outflow_mcm "
                "FROM rid_reservoir_daily WHERE dam_id = :dam_id"
            )
            params = {"dam_id": dam_id}
            if start is not None:
                query += " AND date >= :start"
                params["start"] = start
            engine = create_engine(resolved, pool_pre_ping=True)
            with engine.connect() as conn:
                daily = pd.read_sql(text(query + " ORDER BY date"), conn, params=params)
            daily["date"] = pd.to_datetime(daily["date"])
            daily = daily.set_index("date")
            for col in DAM_COLUMNS:
                daily[col] = pd.to_numeric(daily[col], errors="coerce")
            # Only full, NON-EMPTY loads refresh the cache: a truncated or
            # freshly-recreated table must not wipe a good fallback archive.
            if start is None and not daily.empty:
                cache_path.parent.mkdir(parents=True, exist_ok=True)
                daily.to_csv(cache_path, compression="gzip")
            return daily
        except Exception as error:
            logger.warning(f"dam series DB load failed: {error}")
    if cache_path.exists():
        logger.warning("falling back to on-disk cache for the dam series")
        return pd.read_csv(cache_path, index_col=0, parse_dates=True)
    return None
 def hourly_frame(daily: Optional[pd.DataFrame]) -> Optional[pd.DataFrame]:
    """Step the daily rows onto an hourly grid, each valid from D 07:00."""
    if daily is None or daily.empty:
        return None
    frame = daily.copy()
    frame.index = pd.to_datetime(frame.index) + pd.Timedelta(hours=REPORT_HOUR)
    frame = frame[~frame.index.duplicated(keep="last")].sort_index()
    hourly_index = pd.date_range(
        frame.index.min(),
        frame.index.max() + pd.Timedelta(hours=FFILL_LIMIT_H),
        freq="h",
    )
    return frame.reindex(hourly_index).ffill(limit=FFILL_LIMIT_H)
 def load_history(db_url: Optional[str] = None) -> Optional[pd.DataFrame]:
    """Full hourly Mae Ngat history for training; None when unavailable."""
    return hourly_frame(load_daily(db_url))
 def serving_frame(
    db_url: Optional[str] = None, days: int = 21
 ) -> Optional[pd.DataFrame]:
    """Recent hourly dam state for inference (covers the 336 h feature window
    plus the 72 h storage-delta lag)."""
    start = datetime.date.today() - datetime.timedelta(days=days)
    return hourly_frame(load_daily(db_url, start=start))
@@ -23,7 +23,9 @@ from .features import UPSTREAM_LEADS
 logger = logging.getLogger(__name__)
-DEFAULT_API_URL = "http://100.81.167.42:8000"
+# Public dashboard. Override with --api-url for a local instance; the
 # Tailscale address of the server is deliberately not the default here.
 DEFAULT_API_URL = "https://water.buildfor.life"
 # Anchored to the repo root so training/prediction work from any CWD; a relative
 # path here silently produced 0 rows when the CLI ran outside the repo root.
 CACHE_DIR = Path(__file__).resolve().parents[2] / "models" / "cache"
@@ -97,6 +99,126 @@ def _fetch_from_db(
    return _normalize_long(df)
 # Stations whose HII mirror is the SAME telemetry (corr ≈ 1.000, median diff
 # == station offset exactly — validated 2026-08-11) plus P.81, where the HII
 # twin reads the same river with a bias (corr 0.906, MAE 19 cm) that the
 # dynamic overlap offset corrects. P.76/P.77/P.85/P.87 HII twins are DIFFERENT
 # physical sensors (corr 0.25-0.62) and must never be merged into RID series.
 HII_FILL_STATIONS = (
    "P.1",
    "P.103",
    "P.20",
    "P.4A",
    "P.67",
    "P.75",
    "P.82",
    "P.84",
    "P.92",
    "P.81",
 )
 _HII_EXACT_MIRRORS = frozenset(HII_FILL_STATIONS) - {"P.81"}
 _HII_MIN_OVERLAP_HOURS = 168
 def _fetch_hii_levels(
    db_url: str,
    stations: List[str],
    start: Optional[datetime.datetime],
    end: Optional[datetime.datetime],
 ) -> pd.DataFrame:
    engine = create_engine(db_url, pool_pre_ping=True)
    query = (
        "SELECT m.timestamp, s.rid_code AS station_code, m.wl_msl, m.discharge "
        "FROM hii_waterlevel m JOIN hii_wl_stations s ON s.id = m.station_id "
        "WHERE s.rid_code IS NOT NULL"
    )
    params: Dict = {}
    if start is not None:
        query += " AND m.timestamp >= :start_time"
        params["start_time"] = start
    if end is not None:
        query += " AND m.timestamp <= :end_time"
        params["end_time"] = end
    placeholders = ", ".join(f":station_{i}" for i in range(len(stations)))
    query += f" AND s.rid_code IN ({placeholders})"
    for i, code in enumerate(stations):
        params[f"station_{i}"] = code
    with engine.connect() as connection:
        df = pd.read_sql(text(query), connection, params=params)
    df = df.dropna(subset=["wl_msl"])
    if df.empty:
        return df
    df["timestamp"] = pd.to_datetime(df["timestamp"]).dt.floor("h")
    df["wl_msl"] = pd.to_numeric(df["wl_msl"], errors="coerce")
    df["discharge"] = pd.to_numeric(df["discharge"], errors="coerce")
    df = df.sort_values("timestamp").drop_duplicates(
        subset=["station_code", "timestamp"], keep="last"
    )
    return df
 def fill_from_hii(
    df: pd.DataFrame,
    db_url: str,
    start: Optional[datetime.datetime] = None,
    end: Optional[datetime.datetime] = None,
    stations: Optional[List[str]] = None,
    min_overlap_hours: int = _HII_MIN_OVERLAP_HOURS,
 ) -> pd.DataFrame:
    """Fill missing (station, hour) rows from the HII mirror telemetry.
    In-memory only — water_measurements is never written. Each station's
    MSL→gauge offset is derived from the overlap between the two series
    (median of wl_msl − water_level over ≥ `min_overlap_hours` shared hours),
    which reproduces the published offset for exact mirrors and bias-corrects
    P.81. Discharge is copied only for exact mirrors; P.81 fills get NaN
    discharge (its discharge bias was never validated). Failures degrade to
    returning `df` unchanged, so DBs without hii_* tables keep working.
    """
    codes = [c for c in (stations or HII_FILL_STATIONS) if c in set(df["station_code"])]
    if not codes:
        return df
    try:
        hii = _fetch_hii_levels(db_url, codes, start, end)
    except Exception as error:
        logger.warning(f"HII gap-fill skipped (fetch failed): {error}")
        return df
    if hii.empty:
        return df
    fills = []
    for code, mirror in hii.groupby("station_code"):
        base = df[df["station_code"] == code]
        overlap = base.merge(
            mirror[["timestamp", "wl_msl"]], on="timestamp", how="inner"
        ).dropna(subset=["water_level", "wl_msl"])
        if len(overlap) < min_overlap_hours:
            continue
        offset = (overlap["wl_msl"] - overlap["water_level"]).median()
        # Hours the RID series lacks entirely OR carries only a NaN level;
        # _normalize_long keeps the later (fill) row on collision.
        present = base.loc[base["water_level"].notna(), "timestamp"]
        missing = mirror[~mirror["timestamp"].isin(present)]
        if missing.empty:
            continue
        fill = pd.DataFrame(
            {
                "timestamp": missing["timestamp"],
                "station_code": code,
                "water_level": missing["wl_msl"] - offset,
                "discharge": (
                    missing["discharge"] if code in _HII_EXACT_MIRRORS else float("nan")
                ),
            }
        )
        fills.append(fill)
        logger.info(f"HII gap-fill {code}: +{len(fill)} hours (offset {offset:.3f} m)")
    if not fills:
        return df
    return _normalize_long(pd.concat([df] + fills, ignore_index=True))
 def _fetch_station_from_api(
    api_url: str, station_code: str, hours: int, limit: int = 100000
 ) -> pd.DataFrame:
@@ -160,6 +282,10 @@ def _read_cache(cache_dir: Path, stations: Optional[List[str]]) -> pd.DataFrame:
    frames = []
    for path in sorted(cache_dir.glob("*.csv.gz")):
        code = path.name[: -len(".csv.gz")]
        # The dir is shared with rain.py / dam.py caches (rain_openmeteo,
        # dam_<id>): only station files (P.<n>) are measurements.
        if not code.startswith("P."):
            continue
        if stations and code not in stations:
            continue
        with gzip.open(path, "rt", encoding="utf-8") as handle:
@@ -177,18 +303,23 @@ def load_measurements(
    use_cache: bool = True,
    cache_dir: Path = CACHE_DIR,
    api_url: str = DEFAULT_API_URL,
    hii_fill: bool = True,
 ) -> pd.DataFrame:
    """Load the long-format [timestamp, station_code, water_level, discharge] history.
    Tries PostgreSQL first, then the HTTP API, then the on-disk cache as a last
    resort. A successful DB/API fetch refreshes the cache; the cache itself is
-    never treated as a source of fresh data.
+    never treated as a source of fresh data. With `hii_fill` (DB path only),
    gaps are patched in memory from the HII mirror telemetry — training and
    serving both flow through here, so the two sides see identical series.
    """
    resolved_db_url = resolve_db_url(db_url)
    if resolved_db_url:
        try:
            df = _fetch_from_db(resolved_db_url, stations, start, end)
            if hii_fill:
                df = fill_from_hii(df, resolved_db_url, start=start, end=end)
            if use_cache:
                _write_cache(
                    df, cache_dir, source="postgres", discharge_maybe_synthetic=False
@@ -0,0 +1,515 @@
 """Rolling-origin, event-aware evaluation of forecast-model variants.
 Replaces the single fixed holdout (which contained only ~4 warning events)
 with one fold per monsoon season: train on everything through 30 April of the
 season's year (labels' rescue statistics bounded to the same cutoff, and label
 windows cannot reach the June+ test span, so the folds are leak-free), test on
 June-November. Metrics are event-level — first-alert lead versus each warning
 crossing, peak error at 24 h — plus pointwise MAE and false-alarm episodes,
 because pointwise PR-AUC alone hid the things that matter operationally.
 Variants under test target the two failures documented in
 docs/FLOOD_FORECASTING.md's re-examination note: absolute-level regression
 cannot extrapolate past its training maximum, and the flat sigma miscalibrates
 probabilities.
 """
 import json
 import logging
 from typing import Callable, Dict, List, Optional, Tuple
 import numpy as np
 import pandas as pd
 from scipy.special import erf
 from . import data, features
 from .train import HGB_PARAMS, _make_regressor
 logger = logging.getLogger(__name__)
 HORIZON = 24
 SEASONS = (2021, 2022, 2023, 2024, 2025)
 TEST_MONTHS = ("06-01", "11-30")
 TRAIN_END_MD = "04-30"
 ALERT_P = 0.5
 FIXED_SIGMA = 0.15
 EVENT_GAP_H = 24  # merge >=thr runs closer than this into one event
 FALSE_ALARM_GRACE_H = 48
 def _phi(z: np.ndarray) -> np.ndarray:
    return 0.5 * (1.0 + erf(z / np.sqrt(2.0)))
 def _quantile_regressor(q: float):
    from sklearn.ensemble import HistGradientBoostingRegressor
    return HistGradientBoostingRegressor(loss="quantile", quantile=q, **HGB_PARAMS)
 def _flood_weights(y_abs: pd.Series) -> np.ndarray:
    """Upweight the flood regime: 1x below 2.5 m ramping to 5x at >= 3.7 m."""
    return 1.0 + 4.0 * np.clip((y_abs.to_numpy() - 2.5) / 1.2, 0.0, 1.0)
 # Experimental forward-48h rain sum, built in evaluate_station (not in
 # features.build_features) so the served feature set is untouched until the
 # harness says it helps. Serving could supply it: fetch_forecast() already
 # pulls forecast_days=2.
 EXTRA_RAIN_FEATURES = ("rain_fc48",)
 class Variant:
    """A trainable candidate producing (pred_abs, sigma_per_row) on test rows."""
    def __init__(
        self,
        name: str,
        target: str,
        weighted: bool = False,
        quantile: bool = False,
        use_rain: bool = False,
        use_dam: bool = False,
        use_fc48: bool = False,
        qsigma: bool = False,
    ):
        self.name = name
        self.target = target  # 'abs' or 'rise'
        self.weighted = weighted
        self.quantile = quantile
        self.use_rain = use_rain
        self.use_dam = use_dam
        self.use_fc48 = use_fc48
        # Hybrid: L2 head for the point prediction (keeps the lead-time
        # behaviour of the deployed model exactly, since p>=0.5 alerts are
        # sigma-independent) and quantile heads ONLY for a per-row sigma.
        self.qsigma = qsigma
    def fit_predict(self, X_tr, y_abs_tr, X_te) -> Tuple[np.ndarray, np.ndarray]:
        if not self.use_rain:
            drop = [c for c in features.RAIN_FEATURES if c in X_tr.columns]
            X_tr = X_tr.drop(columns=drop)
            X_te = X_te.drop(columns=drop)
        elif "rain_24h" not in X_tr.columns:
            raise ValueError(
                f"{self.name} requires the rain series (run without --no-rain)"
            )
        if not self.use_dam:
            drop = [c for c in features.DAM_FEATURES if c in X_tr.columns]
            X_tr = X_tr.drop(columns=drop)
            X_te = X_te.drop(columns=drop)
        elif "dam_storage_pct" not in X_tr.columns:
            raise ValueError(
                f"{self.name} requires the dam series (rid_reservoir_daily backfilled)"
            )
        if not self.use_fc48:
            drop = [c for c in EXTRA_RAIN_FEATURES if c in X_tr.columns]
            X_tr = X_tr.drop(columns=drop)
            X_te = X_te.drop(columns=drop)
        elif "rain_fc48" not in X_tr.columns:
            raise ValueError(f"{self.name} requires the rain series")
        level_tr = X_tr["level"]
        level_te = X_te["level"].to_numpy()
        y_tr = (y_abs_tr - level_tr) if self.target == "rise" else y_abs_tr
        weights = _flood_weights(y_abs_tr) if self.weighted else None
        if self.quantile:
            q50 = _quantile_regressor(0.5).fit(X_tr, y_tr, sample_weight=weights)
            q90 = _quantile_regressor(0.9).fit(X_tr, y_tr, sample_weight=weights)
            p50 = q50.predict(X_te)
            spread = np.maximum(q90.predict(X_te) - p50, 0.0)
            sigma = np.maximum(spread / 1.2816, 0.05)
            pred = p50
        else:
            reg = _make_regressor().fit(X_tr, y_tr, sample_weight=weights)
            pred = reg.predict(X_te)
            if self.qsigma:
                q50 = _quantile_regressor(0.5).fit(X_tr, y_tr, sample_weight=weights)
                q90 = _quantile_regressor(0.9).fit(X_tr, y_tr, sample_weight=weights)
                spread = np.maximum(q90.predict(X_te) - q50.predict(X_te), 0.0)
                sigma = np.maximum(spread / 1.2816, 0.05)
            else:
                sigma = np.full(len(X_te), FIXED_SIGMA)
        pred_abs = pred + level_te if self.target == "rise" else pred
        pred_abs = np.maximum(pred_abs, level_te)  # peak >= current, as served
        return pred_abs, sigma
 VARIANTS: Dict[str, Variant] = {
    "baseline_abs": Variant("baseline_abs", target="abs"),
    "rise": Variant("rise", target="rise"),
    "rise_weighted": Variant("rise_weighted", target="rise", weighted=True),
    "rise_quantile": Variant(
        "rise_quantile", target="rise", weighted=True, quantile=True
    ),
    "rise_rain": Variant("rise_rain", target="rise", use_rain=True),
    "rise_rain_dam": Variant(
        "rise_rain_dam", target="rise", use_rain=True, use_dam=True
    ),
    "rise_dam": Variant("rise_dam", target="rise", use_dam=True),
    # 2026-09-12 experiments on top of the deployed rise_rain configuration:
    # per-row sigma from quantile heads (the served sigma sits on the 0.15
    # floor at every P.1 horizon, so stage probabilities are constant-
    # calibrated), and a longer forecast-rain window for the 24 h horizon.
    "rise_rain_quantile": Variant(
        "rise_rain_quantile", target="rise", weighted=True, quantile=True, use_rain=True
    ),
    "rise_rain_quantile_uw": Variant(
        "rise_rain_quantile_uw", target="rise", quantile=True, use_rain=True
    ),
    "rise_rain_fc48": Variant(
        "rise_rain_fc48", target="rise", use_rain=True, use_fc48=True
    ),
    "rise_rain_qsigma": Variant(
        "rise_rain_qsigma", target="rise", use_rain=True, qsigma=True
    ),
 }
 # Dam variants are opt-in by name: they require dam columns that only exist
 # for features.DAM_STATIONS and only when the reservoir series loaded, and
 # the 2026-08-13 ablation concluded them a negative result. The 2026-09-12
 # experiments are opt-in too (see their results in docs/FLOOD_FORECASTING.md).
 DEFAULT_VARIANTS = [
    k
    for k, v in VARIANTS.items()
    if not v.use_dam
    and not v.use_fc48
    and not v.qsigma
    and not (v.quantile and v.use_rain)
 ]
 def _find_events(observed: pd.Series, thr: float) -> List[dict]:
    """Contiguous >=thr episodes (gaps under EVENT_GAP_H merged)."""
    above = observed[observed >= thr]
    if above.empty:
        return []
    events = []
    start = prev = above.index[0]
    for ts in above.index[1:]:
        if (ts - prev) > pd.Timedelta(hours=EVENT_GAP_H):
            events.append((start, prev))
            start = ts
        prev = ts
    events.append((start, prev))
    out = []
    for begin, end in events:
        window = observed.loc[begin:end]
        out.append(
            {
                "crossing": begin,
                "end": end,
                "peak_ts": window.idxmax(),
                "peak_level": float(window.max()),
            }
        )
    return out
 def _first_alert_lead(
    p: pd.Series,
    crossing: pd.Timestamp,
    window_start_floor: Optional[pd.Timestamp] = None,
 ) -> Optional[float]:
    """Hours between the first SUSTAINED alert near the crossing and the
    crossing. Positive = warned in advance; negative = late.
    Sustained = two consecutive hourly samples with p >= ALERT_P (a single
    noisy spike gets no credit). The lookback never reaches past
    ``window_start_floor`` (the previous event's end), so one event's tail
    cannot be credited as early warning for the next crossing.
    """
    start = crossing - pd.Timedelta(hours=72)
    if window_start_floor is not None and window_start_floor > start:
        start = window_start_floor
    window = p.loc[start : crossing + pd.Timedelta(hours=24)]
    if len(window) < 2:
        return None
    alert = (
        (window >= ALERT_P)
        & (window.shift(-1) >= ALERT_P)
        & (
            (window.index.to_series().shift(-1) - window.index.to_series())
            <= pd.Timedelta(hours=2)
        )
    )
    hits = window.index[alert.fillna(False)]
    if len(hits) == 0:
        return None
    return float((crossing - hits[0]).total_seconds() / 3600.0)
 def _false_alarm_episodes(p: pd.Series, observed: pd.Series, thr: float) -> int:
    """Alert episodes with no observed >=thr within +/- FALSE_ALARM_GRACE_H."""
    alert_hours = p[p >= ALERT_P].index
    if len(alert_hours) == 0:
        return 0
    grace = pd.Timedelta(hours=FALSE_ALARM_GRACE_H)
    exceed_times = observed[observed >= thr].index
    episodes = 0
    episode_start = None
    prev = None
    for ts in alert_hours:
        # 12h gap tolerance: a data hole mid-alarm must not double-count it
        if prev is None or (ts - prev) > pd.Timedelta(hours=12):
            if episode_start is not None:
                episodes += _is_false(episode_start, prev, exceed_times, grace)
            episode_start = ts
        prev = ts
    episodes += _is_false(episode_start, prev, exceed_times, grace)
    return episodes
 def _is_false(start, end, exceed_times, grace) -> int:
    if len(exceed_times) == 0:
        return 1
    near = (exceed_times >= start - grace) & (exceed_times <= end + grace)
    return 0 if near.any() else 1
 def evaluate_station(
    df_long: pd.DataFrame,
    station: str,
    variants: Optional[List[str]] = None,
    seasons: Tuple[int, ...] = SEASONS,
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> Dict:
    """Run every fold x variant for one station; returns the results tree."""
    warn_thr, _ = features.get_thresholds(station)
    grid = features.make_hourly_grid(df_long)
    X_all = features.build_features(grid, station, rain=rain, dam=dam)
    if rain is not None:
        # forward sum over (t, t+48]; same construction as rain_fc24
        r = rain.reindex(X_all.index)
        X_all["rain_fc48"] = (
            r.shift(-1).iloc[::-1].rolling(48, min_periods=1).sum().iloc[::-1]
        )
    observed = grid.observed[(station, "water_level")]
    keep = X_all["obs_age_h"].notna()
    train_start = features.TRAIN_START.get(station)
    if train_start:
        keep &= X_all.index >= pd.Timestamp(train_start)
    X_all = X_all.loc[keep]
    chosen = {k: VARIANTS[k] for k in (variants or DEFAULT_VARIANTS)}
    results: Dict = {"station": station, "warn_thr": warn_thr, "folds": []}
    for year in seasons:
        train_end = pd.Timestamp(f"{year}-{TRAIN_END_MD}")
        test_lo = pd.Timestamp(f"{year}-{TEST_MONTHS[0]}")
        test_hi = pd.Timestamp(f"{year}-{TEST_MONTHS[1]} 23:00")
        # Labels rebuilt per fold so rescue statistics stop at the cutoff
        Y = features.build_labels(
            grid, station, (HORIZON,), stats_end=train_end.isoformat()
        ).loc[X_all.index]
        y_abs = Y[f"max_level_{HORIZON}"]
        tr = (X_all.index <= train_end) & y_abs.notna()
        te = (X_all.index >= test_lo) & (X_all.index <= test_hi)
        if tr.sum() < 5000 or te.sum() < 500:
            logger.info(
                f"{station} {year}: skipped (train {tr.sum()}, test {te.sum()})"
            )
            continue
        X_tr, X_te = X_all.loc[tr], X_all.loc[te]
        y_tr = y_abs.loc[tr]
        y_te = y_abs.loc[te]
        obs_test = observed.loc[test_lo:test_hi].dropna()
        events = _find_events(obs_test, warn_thr)
        fold: Dict = {
            "year": year,
            "n_train": int(tr.sum()),
            "n_test": int(te.sum()),
            "events": [
                {
                    "crossing": e["crossing"].isoformat(),
                    "peak_ts": e["peak_ts"].isoformat(),
                    "peak_level": e["peak_level"],
                }
                for e in events
            ],
            "variants": {},
        }
        for name, variant in chosen.items():
            try:
                pred_abs, sigma = variant.fit_predict(X_tr, y_tr, X_te)
            except ValueError as error:
                # A variant whose required feature family is absent (e.g. a
                # dam variant on a non-DAM_STATIONS target) skips this fold
                # instead of killing the whole run and its finished results.
                logger.warning(f"{station} {year} {name}: skipped ({error})")
                continue
            pred_series = pd.Series(pred_abs, index=X_te.index)
            p_warn = pd.Series(
                1.0 - _phi((warn_thr - pred_abs) / sigma), index=X_te.index
            )
            labeled = y_te.notna()
            errors = (pred_series[labeled] - y_te[labeled]).abs()
            high = y_te[labeled] >= warn_thr - 1.2  # flood-regime rows
            # Brier score on within-24h warning exceedance: unlike the p>=0.5
            # alert metrics (where sigma cancels algebraically), this actually
            # exercises each variant's uncertainty model.
            exceed = Y[f"exceed_warn_{HORIZON}"].loc[te]
            scored = exceed.notna()
            brier = (
                float(((p_warn[scored] - exceed[scored]) ** 2).mean())
                if scored.any()
                else None
            )
            event_rows = []
            for i, event in enumerate(events):
                floor = events[i - 1]["end"] if i > 0 else None
                lead = _first_alert_lead(p_warn, event["crossing"], floor)
                issue_ts = event["peak_ts"] - pd.Timedelta(hours=HORIZON)
                peak_pred = None
                if len(pred_series):
                    nearest = pred_series.index.get_indexer(
                        [issue_ts], method="nearest"
                    )[0]
                    matched_ts = pred_series.index[nearest]
                    # Tolerance: a "24h-ahead" prediction matched to a row
                    # hours away (data outage) is not that prediction at all.
                    if abs(matched_ts - issue_ts) <= pd.Timedelta(hours=3):
                        peak_pred = float(pred_series.iloc[nearest])
                event_rows.append(
                    {
                        "crossing": event["crossing"].isoformat(),
                        "lead_h": lead,
                        "peak_level": event["peak_level"],
                        "peak_pred_24h_before": peak_pred,
                    }
                )
            fold["variants"][name] = {
                "mae": float(errors.mean()) if len(errors) else None,
                "mae_above_2p5": (float(errors[high].mean()) if high.any() else None),
                "brier_warn": brier,
                "events": event_rows,
                "false_alarm_episodes": _false_alarm_episodes(
                    p_warn, obs_test, warn_thr
                ),
            }
        results["folds"].append(fold)
    return results
 def summarize(results: Dict) -> str:
    """Compact comparison table across folds for one station."""
    lines = [f"\n=== {results['station']} (warn {results['warn_thr']:.2f} m) ==="]
    header = (
        f"{'variant':16} {'year':>5} {'MAE':>6} {'MAE_hi':>7} {'Brier':>7} "
        f"{'FA':>3}  events (lead h | peak err m)"
    )
    lines.append(header)
    for fold in results["folds"]:
        for name, m in fold["variants"].items():
            events = (
                "  ".join(
                    f"[{e['crossing'][:10]}: "
                    f"{'—' if e['lead_h'] is None else format(e['lead_h'], '+.0f')}h"
                    + (
                        f" | {e['peak_pred_24h_before'] - e['peak_level']:+.2f}"
                        if e["peak_pred_24h_before"] is not None
                        else ""
                    )
                    + "]"
                    for e in m["events"]
                )
                or "no events"
            )
            lines.append(
                f"{name:16} {fold['year']:>5} "
                f"{m['mae'] if m['mae'] is not None else float('nan'):6.3f} "
                f"{m['mae_above_2p5'] if m['mae_above_2p5'] is not None else float('nan'):7.3f} "
                f"{m['brier_warn'] if m.get('brier_warn') is not None else float('nan'):7.4f} "
                f"{m['false_alarm_episodes']:>3}  {events}"
            )
    return "\n".join(lines)
 def main(argv=None) -> int:
    import argparse
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--stations", default="P.1")
    parser.add_argument("--db-url", default=None)
    parser.add_argument("--variants", default=None, help="comma list; default all")
    parser.add_argument("--out", default="models/eval_variants.json")
    parser.add_argument(
        "--no-rain", action="store_true", help="skip loading the Open-Meteo rain series"
    )
    parser.add_argument(
        "--no-dam",
        action="store_true",
        help="skip loading the Mae Ngat reservoir series",
    )
    parser.add_argument(
        "--from-cache",
        action="store_true",
        help="offline: read models/cache/ only (no DB, no API, "
        "no Open-Meteo refresh) -- reproducible reruns",
    )
    args = parser.parse_args(argv)
    logging.basicConfig(
        level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
    )
    if args.from_cache:
        df = data._read_cache(data.CACHE_DIR, None)
        logger.info(f"measurements from cache: {len(df)} rows")
    else:
        df = data.load_measurements(db_url=args.db_url)
    if df.empty:
        logger.error("no measurement data")
        return 1
    rain_series = None
    if not args.no_rain:
        from . import rain as rain_mod
        rain_series = rain_mod.catchment_mean(
            rain_mod.load_history(refresh=not args.from_cache)
        )
        if rain_series is None:
            logger.warning("rain history unavailable; rain features will be NaN")
        else:
            logger.info(
                f"rain series loaded: {rain_series.index.min()} .. "
                f"{rain_series.index.max()}"
            )
    dam_frame = None
    if not args.no_dam and not args.from_cache:
        from . import dam as dam_mod
        dam_frame = dam_mod.load_history(db_url=args.db_url)
        if dam_frame is None:
            logger.warning("dam history unavailable; dam features will be absent")
        else:
            logger.info(
                f"dam series loaded: {dam_frame.index.min()} .. "
                f"{dam_frame.index.max()}"
            )
    variant_names = args.variants.split(",") if args.variants else None
    all_results = []
    for station in args.stations.split(","):
        station = station.strip()
        logger.info(f"Evaluating {station}...")
        results = evaluate_station(
            df, station, variant_names, rain=rain_series, dam=dam_frame
        )
        all_results.append(results)
        print(summarize(results))
    with open(args.out, "w", encoding="utf-8") as fh:
        json.dump(all_results, fh, indent=1)
    logger.info(f"results written to {args.out}")
    return 0
@@ -37,9 +37,17 @@ THRESHOLDS: Dict[str, Tuple[float, float]] = {
    "P.4A": (3.40, 3.90),
    "P.5": (4.55, 4.95),
    "P.67": (2.45, 2.90),
-    "P.75": (2.75, 3.50),
+    # P.75: 2024 (the only year with a full flood record, 191% capacity peak)
    # puts 75-85% at 3.45 m and 95-105% at 3.72 m; 2018/2022 agree within
    # 0.15 m. The 2026-08 value (2.75) alerted on 15 quiet-season hours.
    "P.75": (3.20, 3.65),
    "P.76": (5.35, 5.45),
-    "P.77": (2.85, 3.35),
+    # P.77: recalibrated 2026-09-12. The 2026-08 value (2.85) sat below the
    # gauge's own dry-season baseline (2.6-2.7 m at 8-14% capacity), so the
    # first ntfy cycle fired a "warning" at 22% capacity. Across 2018-2024,
    # 75-85% capacity reads 3.35-4.57 m and 95-105% 4.27-5.08 m; 2024 (the
    # best-sampled flood year) gives 4.57 / 5.08. Slightly conservative:
    "P.77": (4.30, 4.90),
    "P.81": (5.15, 6.30),
    # P.82 never reached 100% capacity in the record (max level 3.78, max 96.4%);
    # danger sits just below the observed maximum so the head can actually train.
@@ -200,8 +208,38 @@ def _hours_since_observed(mask_col: pd.Series) -> pd.Series:
 # ---------------------------------------------------------------------------
-def build_features(grid: HourlyGrid, station: str) -> pd.DataFrame:
+RAIN_FEATURES = ("rain_6h", "rain_24h", "rain_72h", "rain_fc24")
-    """Build the deterministic-order feature matrix for one target station."""
+
 DAM_FEATURES = ("dam_storage_pct", "dam_storage_pct_d3", "dam_inflow", "dam_outflow")
 # Stations hydrologically downstream of the Mae Ngat confluence (Ping mainstem
 # at/below Mae Taeng) — the only ones where reservoir state is causal. West-
 # tributary and upper-mainstem stations never receive dam columns.
 DAM_STATIONS = frozenset({"P.1", "P.103", "P.67", "P.21", "P.5", "P.81"})
 def build_features(
    grid: HourlyGrid,
    station: str,
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> pd.DataFrame:
    """Build the deterministic-order feature matrix for one target station.
    ``rain`` is the hourly catchment-average precipitation series (Open-Meteo,
    src/ml/rain.py). Rain columns are added only when a series is passed:
    HistGradientBoosting REJECTS all-NaN columns at fit time, so training
    without rain must omit the columns entirely (bundles record their
    feature_names, and serving subsets to them). At serving, pass an empty
    series rather than None so the columns exist (as NaN) for rain-trained
    bundles even when the live fetch fails. rain_fc24 is the forward 24 h
    sum: the archived forecast series at training time, a real weather
    forecast at serving time; it never contains river data.
    ``dam`` is the hourly Mae Ngat reservoir frame (src/ml/dam.py; columns
    storage_pct/inflow_mcm/outflow_mcm, already leakage-shifted to 07:00
    report time). Same contract as rain: None omits the columns, an empty
    frame yields NaN columns; only DAM_STATIONS receive them.
    """
    idx = grid.observed.index
    cols: Dict[str, pd.Series] = {}
@@ -254,6 +292,28 @@ def build_features(grid: HourlyGrid, station: str) -> pd.DataFrame:
    cols["doy_cos"] = np.cos(2 * np.pi * doy / 365.25)
    cols["is_monsoon"] = idx.to_series().dt.month.isin(MONSOON_MONTHS).astype(float)
    if rain is not None:
        r = rain.reindex(idx)
        cols["rain_6h"] = r.rolling(6, min_periods=1).sum()
        cols["rain_24h"] = r.rolling(24, min_periods=1).sum()
        cols["rain_72h"] = r.rolling(72, min_periods=1).sum()
        # forward sum over (t, t+24]: shift(-1) starts the window at t+1
        cols["rain_fc24"] = (
            r.shift(-1).iloc[::-1].rolling(24, min_periods=1).sum().iloc[::-1]
        )
    if dam is not None and station in DAM_STATIONS:
        d = dam.reindex(idx)
        def _dam_col(name: str) -> pd.Series:
            return d[name] if name in d.columns else pd.Series(np.nan, index=idx)
        storage = _dam_col("storage_pct")
        cols["dam_storage_pct"] = storage
        cols["dam_storage_pct_d3"] = storage - storage.shift(72)
        cols["dam_inflow"] = _dam_col("inflow_mcm")
        cols["dam_outflow"] = _dam_col("outflow_mcm")
    return pd.DataFrame(cols, index=idx)
@@ -272,9 +332,17 @@ def _future_window_stats(col: pd.Series, horizon_h: int) -> Tuple[pd.Series, pd.
 def build_labels(
-    grid: HourlyGrid, station: str, horizons: Tuple[int, ...] = (6, 12, 24)
+    grid: HourlyGrid,
    station: str,
    horizons: Tuple[int, ...] = (6, 12, 24),
    stats_end: Optional[str] = None,
 ) -> pd.DataFrame:
-    """Build max-level and threshold-exceedance labels for one target station."""
+    """Build max-level and threshold-exceedance labels for one target station.
    ``stats_end`` bounds the data used for label-construction statistics (the
    rescue quantile below): pass the training cutoff during evaluation so
    test-period extremes cannot influence which training rows receive labels.
    """
    idx = grid.observed.index
    observed_level = _series(grid.observed, station, "water_level", idx)
    warn_thr, danger_thr = get_thresholds(station)
@@ -283,10 +351,11 @@ def build_labels(
    # NOT to warn_thr: coupling it to the configurable threshold made raising a
    # station's threshold silently shrink its regression training set (P.5 lost
    # 34% of rows and +46% MAE when its warning went 3.0 -> 4.55).
    stats_level = (
        observed_level.loc[: pd.Timestamp(stats_end)] if stats_end else observed_level
    )
    rescue_thr = (
-        float(observed_level.quantile(0.975))
+        float(stats_level.quantile(0.975)) if stats_level.notna().any() else np.inf
        if observed_level.notna().any()
        else np.inf
    )
    out: Dict[str, pd.Series] = {}
@@ -321,11 +390,14 @@ def build_matrix(
    df_long: pd.DataFrame,
    station: str,
    horizons: Tuple[int, ...] = (6, 12, 24),
    stats_end: Optional[str] = None,
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> Tuple[pd.DataFrame, pd.DataFrame, dict]:
    """Build (X, Y, meta) training/inference matrices for one station."""
    grid = make_hourly_grid(df_long)
-    X = build_features(grid, station)
+    X = build_features(grid, station, rain=rain, dam=dam)
-    Y = build_labels(grid, station, horizons)
+    Y = build_labels(grid, station, horizons, stats_end=stats_end)
    keep = X["obs_age_h"].notna()
    train_start = TRAIN_START.get(station)
@@ -0,0 +1,122 @@
 """Catchment-mean hourly rain from the HII/ThaiWater gauge network.
 Independent of Open-Meteo (src/ml/rain.py): those are model-analysis values,
 these are what the gauges measured. The `hii_rainfall` table has been filled
 by the hourly collector since 2026-08-11 and there is NO archive behind it
 (the api-v3 rain_24h_graph endpoint ignores its date range, see
 docs/DATA_SOURCES.md 2.1), so this series cannot yet be a training feature:
 every training row before 2026-08 would be NaN and HistGradientBoosting
 would learn nothing from the column. It becomes a candidate once a full
 monsoon season of gauge rows exists in the rolling-origin harness's test
 span -- the 2027 fold (train through 2027-04-30, test Jun-Nov 2027) is the
 first that could show anything.
 Until then it serves two purposes:
  * a live cross-check of the Open-Meteo catchment mean (/api/hii/rainfall
    already exposes the raw gauges; this gives the comparable aggregate);
  * accumulating the comparison so the eventual feature evaluation has a
    documented bias/variance relationship between the two sources.
 """
 import logging
 from typing import Optional, Sequence, Tuple
 import pandas as pd
 from .data import resolve_db_url
 logger = logging.getLogger(__name__)
 # Same footprint as rain.CATCHMENT_POINTS: the upper Ping above P.1. Gauges
 # inside this box are averaged; there are ~130 with recent data (DWR, FOP,
 # HII, RID, TMD), far denser than the five Open-Meteo points.
 CATCHMENT_BOX: Tuple[float, float, float, float] = (18.75, 19.60, 98.60, 99.30)
 # A gauge that reports the same rain_24h for many hours is stuck; drop hours
 # where fewer than this many gauges reported at all.
 MIN_GAUGES_PER_HOUR = 5
 def load_gauge_mean(
    db_url: Optional[str] = None,
    start: Optional[pd.Timestamp] = None,
    end: Optional[pd.Timestamp] = None,
    box: Sequence[float] = CATCHMENT_BOX,
    engine=None,
 ) -> Optional[pd.Series]:
    """Hourly catchment-mean rain_1h (mm) across HII gauges in `box`.
    Pass `engine` (the API's HII store engine) to reuse a pool; otherwise a
    connection is resolved from db_url / config. Returns None if the DB is
    unavailable or the table is empty. Hours with fewer than
    MIN_GAUGES_PER_HOUR reporting gauges are NaN.
    """
    if engine is None:
        resolved = resolve_db_url(db_url)
        if not resolved:
            return None
    lat_lo, lat_hi, lon_lo, lon_hi = box
    try:
        from sqlalchemy import create_engine, text
        query = (
            "SELECT m.timestamp, COUNT(m.rain_1h) AS n, AVG(m.rain_1h) AS rain_1h "
            "FROM hii_rainfall m JOIN hii_rain_stations s ON s.id = m.station_id "
            "WHERE s.latitude BETWEEN :lat_lo AND :lat_hi "
            "AND s.longitude BETWEEN :lon_lo AND :lon_hi "
            "AND m.rain_1h IS NOT NULL"
        )
        params = {
            "lat_lo": lat_lo,
            "lat_hi": lat_hi,
            "lon_lo": lon_lo,
            "lon_hi": lon_hi,
        }
        if start is not None:
            query += " AND m.timestamp >= :start"
            params["start"] = pd.Timestamp(start).to_pydatetime()
        if end is not None:
            query += " AND m.timestamp <= :end"
            params["end"] = pd.Timestamp(end).to_pydatetime()
        query += " GROUP BY m.timestamp ORDER BY m.timestamp"
        if engine is None:
            engine = create_engine(resolved, pool_pre_ping=True)
        with engine.connect() as conn:
            frame = pd.read_sql(text(query), conn, params=params)
    except Exception as error:
        logger.warning(f"HII gauge rain load failed: {error}")
        return None
    if frame.empty:
        return None
    frame["timestamp"] = pd.to_datetime(frame["timestamp"]).dt.floor("h")
    frame = frame.groupby("timestamp").agg(n=("n", "sum"), rain_1h=("rain_1h", "mean"))
    series = pd.to_numeric(frame["rain_1h"], errors="coerce")
    series[frame["n"] < MIN_GAUGES_PER_HOUR] = float("nan")
    series.name = "hii_gauge_mean"
    return series
 def compare_with_openmeteo(
    gauge: pd.Series, openmeteo: pd.Series, window_h: int = 24
 ) -> dict:
    """Bias/correlation of Open-Meteo against the gauges over the overlap.
    Both are summed over trailing `window_h` so single-hour timing offsets
    (gauges report at :00, the model's hour is an interval) do not dominate.
    """
    joined = pd.concat({"gauge": gauge, "openmeteo": openmeteo}, axis=1).dropna()
    if joined.empty:
        return {"overlap_hours": 0}
    g = joined["gauge"].rolling(window_h, min_periods=window_h).sum()
    o = joined["openmeteo"].rolling(window_h, min_periods=window_h).sum()
    both = pd.concat({"g": g, "o": o}, axis=1).dropna()
    if both.empty:
        return {"overlap_hours": int(len(joined))}
    return {
        "overlap_hours": int(len(joined)),
        "window_h": window_h,
        "gauge_mean_mm": float(both["g"].mean()),
        "openmeteo_mean_mm": float(both["o"].mean()),
        "bias_mm": float((both["o"] - both["g"]).mean()),
        "mae_mm": float((both["o"] - both["g"]).abs().mean()),
        "corr": float(both["g"].corr(both["o"])),
    }
@@ -127,6 +127,8 @@ def _model_forecast(
    bundle: dict,
    as_of: pd.Timestamp,
    current_level: float,
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> List[dict]:
    warn_thr = bundle["thresholds"]["warning"]
    danger_thr = bundle["thresholds"]["danger"]
@@ -145,7 +147,9 @@ def _model_forecast(
        )
        warn_thr, danger_thr = cfg_warn, cfg_danger
-    feature_row = features.build_features(grid, station_code).loc[[as_of]]
+    feature_row = features.build_features(grid, station_code, rain=rain, dam=dam).loc[
        [as_of]
    ]
    expected_columns = bundle["feature_names"]
    missing = [c for c in expected_columns if c not in feature_row.columns]
    if missing:
@@ -161,24 +165,35 @@ def _model_forecast(
        if reg is None:
            results.append(None)
            continue
-        predicted_max = max(float(reg.predict(feature_row)[0]), current_level)
+        raw_prediction = float(reg.predict(feature_row)[0])
        if bundle.get("regression_target") == "rise":
            # v2 bundles predict the rise over the current level
            raw_prediction += current_level
        predicted_max = max(raw_prediction, current_level)
        sigma_h = bundle["sigma"].get(horizon_h, HEURISTIC_SIGMA)
        # Belt-and-braces: the classifier head OR the regression-sigmoid path,
        # whichever is more alarmed. The 2026-08-11 backtest showed a trained
        # classifier staying silent through the 2024 record flood while the
        # regression head tracked it — alerting must never be worse than the
        # regression fallback.
        warn_head = (
            None if thresholds_stale else bundle["heads"].get(f"warn_{horizon_h}")
        )
        p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
        if warn_head is not None:
-            p_warning = float(warn_head.predict_proba(feature_row)[0][1])
+            p_warning = max(
-        else:
+                p_warning, float(warn_head.predict_proba(feature_row)[0][1])
-            p_warning = _sigmoid_probability(predicted_max, warn_thr, sigma_h)
+            )
        danger_head = (
            None if thresholds_stale else bundle["heads"].get(f"danger_{horizon_h}")
        )
        p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
        if danger_head is not None:
-            p_danger = float(danger_head.predict_proba(feature_row)[0][1])
+            p_danger = max(
-        else:
+                p_danger, float(danger_head.predict_proba(feature_row)[0][1])
-            p_danger = _sigmoid_probability(predicted_max, danger_thr, sigma_h)
+            )
        p_warning = _clip_probability(p_warning)
        p_danger = min(_clip_probability(p_danger), p_warning)
@@ -222,6 +237,8 @@ def _forecast_station(
    models_dir: Path,
    now: pd.Timestamp,
    horizons: Tuple[int, ...],
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> List[dict]:
    level_col = (station_code, "water_level")
    if level_col not in grid.observed.columns:
@@ -257,7 +274,9 @@ def _forecast_station(
        )
    bundle = _load_bundle(bundle_path)
-    model_results = _model_forecast(station_code, grid, bundle, as_of, current_level)
+    model_results = _model_forecast(
        station_code, grid, bundle, as_of, current_level, rain=rain, dam=dam
    )
    if model_results is None:
        return _heuristic_forecast(
            station_code,
@@ -294,6 +313,8 @@ def get_forecasts(
    readings_by_station: Dict[str, List[dict]],
    models_dir: Union[str, Path] = DEFAULT_MODELS_DIR,
    now: Optional[Union[datetime.datetime, str]] = None,
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> List[dict]:
    """Produce flood forecasts for every station present in `readings_by_station`.
@@ -316,7 +337,15 @@ def get_forecasts(
    for station_code in readings_by_station.keys():
        try:
            results.extend(
-                _forecast_station(station_code, grid, models_dir, now, DEFAULT_HORIZONS)
+                _forecast_station(
                    station_code,
                    grid,
                    models_dir,
                    now,
                    DEFAULT_HORIZONS,
                    rain=rain,
                    dam=dam,
                )
            )
        except Exception as error:
            logger.error(f"Forecast failed for station {station_code}: {error}")
@@ -352,4 +381,27 @@ def get_latest_forecasts(
            f"No recent data for station {missing_station}; omitting from forecasts"
        )
-    return get_forecasts(readings_by_station, models_dir=models_dir)
+    # Live rain: trailing days + next-48h forecast. On fetch failure pass an
    # EMPTY series (not None) so rain-trained bundles still find their columns
    # (as NaN) and serve model output instead of tripping the feature guard.
    from .rain import serving_series
    rain = serving_series()
    if rain is None:
        logger.warning("live rain unavailable; rain features will be NaN")
        rain = pd.Series(dtype=float)
    # Recent Mae Ngat reservoir state; same empty-not-None contract so
    # dam-trained bundles keep their columns (NaN) when the DB read fails.
    from . import dam as dam_mod
    try:
        dam = dam_mod.serving_frame(db_url=db_url)
    except Exception as error:
        logger.warning(f"dam serving frame failed: {error}")
        dam = None
    if dam is None:
        logger.warning("dam state unavailable; dam features will be NaN")
        dam = pd.DataFrame()
    return get_forecasts(readings_by_station, models_dir=models_dir, rain=rain, dam=dam)
@@ -0,0 +1,232 @@
 """Open-Meteo rainfall series for the upper Ping catchment.
 One consistent source for training AND serving: the Open-Meteo forecast-model
 archive (historical-forecast-api, 2021-03 onward) supplies hourly
 precipitation at five catchment points above P.1; the live forecast endpoint
 supplies the same series for recent days plus the next 48 h. Timestamps are
 Asia/Bangkok local, matching the measurement grid. Rows before 2021-03 simply
 have no rain data — HistGradientBoosting handles the NaNs natively.
 The forward-looking sum built from this series is a legitimate *forecast*
 feature, not label leakage: the series never contains river observations, and
 at serving time the future values come from an actual weather forecast.
 """
 import datetime
 import logging
 from pathlib import Path
 from typing import Iterable, List, Optional, Tuple
 import pandas as pd
 import requests
 logger = logging.getLogger(__name__)
 # (name, lat, lon) — upper Ping catchment above P.1, headwaters to city
 CATCHMENT_POINTS: Tuple[Tuple[str, float, float], ...] = (
    ("chiang_dao", 19.37, 98.97),
    ("mae_taeng", 19.12, 98.94),
    ("mae_ngat", 19.17, 99.05),
    ("mae_rim", 18.92, 98.92),
    ("chiang_mai", 18.79, 99.00),
 )
 HISTORY_URL = "https://historical-forecast-api.open-meteo.com/v1/forecast"
 FORECAST_URL = "https://api.open-meteo.com/v1/forecast"
 HISTORY_START = "2021-03-23"  # archive begins here
 CACHE_FILE = "rain_openmeteo.csv.gz"
 def _points_params() -> dict:
    return {
        "latitude": ",".join(str(lat) for _, lat, _ in CATCHMENT_POINTS),
        "longitude": ",".join(str(lon) for _, _, lon in CATCHMENT_POINTS),
        "hourly": "precipitation",
        "timezone": "Asia/Bangkok",
    }
 def _parse_multi(payload, columns: Iterable[str]) -> pd.DataFrame:
    """Open-Meteo returns a list when multiple coordinates are requested."""
    results = payload if isinstance(payload, list) else [payload]
    frames = {}
    for name, result in zip(columns, results):
        hourly = result.get("hourly", {})
        idx = pd.to_datetime(hourly.get("time", []))
        frames[name] = pd.Series(hourly.get("precipitation", []), index=idx)
    df = pd.DataFrame(frames)
    df.index.name = "timestamp"
    return df
 def fetch_history(
    start: str, end: str, session: Optional[requests.Session] = None
 ) -> pd.DataFrame:
    """Hourly precipitation for all catchment points over [start, end]."""
    session = session or requests.Session()
    response = session.get(
        HISTORY_URL,
        params={**_points_params(), "start_date": start, "end_date": end},
        timeout=120,
    )
    response.raise_for_status()
    return _parse_multi(response.json(), [p[0] for p in CATCHMENT_POINTS])
 def fetch_forecast(
    past_days: int = 5,
    forecast_days: int = 2,
    session: Optional[requests.Session] = None,
 ) -> pd.DataFrame:
    """Recent + next-48h precipitation from the live forecast endpoint."""
    session = session or requests.Session()
    response = session.get(
        FORECAST_URL,
        params={
            **_points_params(),
            "past_days": past_days,
            "forecast_days": forecast_days,
        },
        timeout=60,
    )
    response.raise_for_status()
    return _parse_multi(response.json(), [p[0] for p in CATCHMENT_POINTS])
 def load_history(
    cache_dir: Path = Path("models/cache"),
    end: Optional[datetime.date] = None,
    refresh: bool = True,
 ) -> Optional[pd.DataFrame]:
    """Cached catchment rain history from 2021-03 to ~today.
    Fetches year-sized chunks on first use (~6 requests), then only extends
    the tail. Returns None when the API is unreachable and no cache exists.
    """
    cache_dir.mkdir(parents=True, exist_ok=True)
    cache_path = cache_dir / CACHE_FILE
    end = end or datetime.date.today()
    cached: Optional[pd.DataFrame] = None
    if cache_path.exists():
        cached = pd.read_csv(cache_path, index_col=0, parse_dates=True)
    fetch_from = pd.Timestamp(HISTORY_START)
    if cached is not None and len(cached):
        fetch_from = cached.index.max() - pd.Timedelta(days=2)  # re-fetch tail
    if not refresh and cached is not None:
        return cached
    chunks: List[pd.DataFrame] = []
    cursor = fetch_from.date()
    try:
        while cursor <= end:
            chunk_end = min(datetime.date(cursor.year, 12, 31), end)
            chunks.append(fetch_history(cursor.isoformat(), chunk_end.isoformat()))
            cursor = datetime.date(cursor.year + 1, 1, 1)
    except Exception as error:
        logger.warning(f"Open-Meteo history fetch failed: {error}")
        if not chunks and cached is None:
            return None
    if chunks:
        fresh = pd.concat(chunks)
        combined = (
            pd.concat([cached[cached.index < fresh.index.min()], fresh])
            if cached is not None
            else fresh
        )
        combined = combined[~combined.index.duplicated(keep="last")].sort_index()
        combined.to_csv(cache_path, compression="gzip")
        return combined
    return cached
 def catchment_mean(df: Optional[pd.DataFrame]) -> Optional[pd.Series]:
    """Single catchment-average hourly rain series (mm)."""
    if df is None or df.empty:
        return None
    return df.mean(axis=1)
 def serving_series() -> Optional[pd.Series]:
    """Catchment rain for inference: trailing days + the next 48 h forecast."""
    try:
        return catchment_mean(fetch_forecast())
    except Exception as error:
        logger.warning(f"Open-Meteo forecast fetch failed: {error}")
        return None
 def backfill_db(engine, db_type: str, chunk_rows: int = 5000) -> int:
    """Push the full Open-Meteo history (2021+) into openmeteo_rain.
    Loads (or fetches) the archive cache and upserts in chunks; idempotent,
    safe to re-run, and safe alongside the hourly live writer.
    """
    history = load_history()
    if history is None or history.empty:
        logger.error("no rain history available to backfill")
        return 0
    total = 0
    for start in range(0, len(history), chunk_rows):
        part = history.iloc[start : start + chunk_rows]
        total += save_to_db(part, engine, db_type)
        logger.info(f"openmeteo_rain backfill: {total}/{len(history)} rows")
    return total
 def save_to_db(df: pd.DataFrame, engine, db_type: str) -> int:
    """Upsert per-point + catchment-mean hourly rain into openmeteo_rain.
    Called by the leader worker's hourly precompute with the live forecast
    frame, so the DB accumulates both what fell (past rows are the model
    analysis) and what was forecast (future rows, overwritten as they become
    past). The ML training path reads Open-Meteo's own archive, not this
    table — this is for dashboards, SQL analysis, and source independence.
    """
    if df is None or df.empty:
        return 0
    from sqlalchemy import text
    point_cols = [p[0] for p in CATCHMENT_POINTS]
    ddl_cols = ", ".join(f"{c} NUMERIC(6,2)" for c in point_cols)
    ddl = (
        "CREATE TABLE IF NOT EXISTS openmeteo_rain ("
        "timestamp TIMESTAMP PRIMARY KEY, "
        f"{ddl_cols}, catchment_mean NUMERIC(6,2), "
        "created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP)"
    )
    cols = ["timestamp"] + point_cols + ["catchment_mean"]
    placeholders = ", ".join(f":{c}" for c in cols)
    updates = ", ".join(
        f"{c} = " + (f"VALUES({c})" if db_type == "mysql" else f"EXCLUDED.{c}")
        for c in cols[1:]
    )
    if db_type == "mysql":
        sql = (
            f"INSERT INTO openmeteo_rain ({', '.join(cols)}) VALUES ({placeholders}) "
            f"ON DUPLICATE KEY UPDATE {updates}"
        )
    else:
        sql = (
            f"INSERT INTO openmeteo_rain ({', '.join(cols)}) VALUES ({placeholders}) "
            f"ON CONFLICT (timestamp) DO UPDATE SET {updates}"
        )
    mean = df.mean(axis=1)
    params = [
        {
            "timestamp": ts.to_pydatetime(),
            **{c: (None if pd.isna(row[c]) else float(row[c])) for c in point_cols},
            "catchment_mean": None if pd.isna(mean.loc[ts]) else float(mean.loc[ts]),
        }
        for ts, row in df.iterrows()
    ]
    try:
        with engine.begin() as conn:
            conn.execute(text(ddl))
            conn.execute(text(sql), params)
        return len(params)
    except Exception as error:
        logger.error(f"openmeteo_rain save failed: {error}")
        return 0
@@ -0,0 +1,170 @@
 """Live forecast skill: what the deployed model said versus what the river did.
 Every hour the precompute stores the issued 24 h forecast (forecast_history);
 water_measurements holds what actually happened. Joining the two gives a
 verification that needs no retraining and answers the question the dashboard
 is asked most: "is the model getting better?" — per model version, on the
 hours that version was actually serving.
 Metrics per version and horizon:
  n           verified forecasts (issued, and the horizon has since elapsed)
  mae         |predicted_max - observed_max| over the horizon window, metres
  bias        mean(predicted - observed): >0 over-predicts the peak
  persistence MAE of the trivial "peak = current level" forecast on the
              same rows; a model is only useful if it beats this
  skill       1 - mae/persistence  (0 = no better than persistence, 1 = perfect)
  above_2m    same MAE restricted to rows where the observed peak >= 2 m,
              i.e. the flood-relevant regime
 Only the P.1 gauge is verified by default: it is the one the city threshold
 is keyed to, and one station keeps the query cheap enough to run on request.
 """
 import datetime
 import logging
 from typing import Dict, List, Optional
 logger = logging.getLogger(__name__)
 DEFAULT_STATION = "P.1"
 DEFAULT_HORIZON = 24
 MIN_VERIFIED = 24  # fewer than a day of verified hours is not a number
 def _sql_for(db_type: str) -> str:
    """Join each issued forecast to the observed max over (as_of, as_of + h]."""
    if db_type == "postgresql":
        window_end = "f.as_of + (f.horizon_hours || ' hours')::interval"
    elif db_type == "mysql":
        window_end = "DATE_ADD(f.as_of, INTERVAL f.horizon_hours HOUR)"
    else:  # sqlite
        window_end = "datetime(f.as_of, '+' || f.horizon_hours || ' hours')"
    return f"""
        SELECT f.as_of, f.model_version, f.predicted_max_level, f.current_level,
               (SELECT MAX(m.water_level) FROM water_measurements m
                 JOIN stations s ON s.id = m.station_id
                WHERE s.station_code = f.station_code
                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_max,
               (SELECT COUNT(m.water_level) FROM water_measurements m
                 JOIN stations s ON s.id = m.station_id
                WHERE s.station_code = f.station_code
                  AND m.timestamp > f.as_of AND m.timestamp <= {window_end}) AS observed_n
        FROM forecast_history f
        WHERE f.station_code = :code AND f.horizon_hours = :horizon
          AND f.source = 'model' AND f.predicted_max_level IS NOT NULL
          AND f.as_of <= :verifiable_before
        ORDER BY f.as_of
    """
 def compute_skill(
    engine,
    db_type: str,
    station_code: str = DEFAULT_STATION,
    horizon_hours: int = DEFAULT_HORIZON,
    now: Optional[datetime.datetime] = None,
 ) -> Dict:
    """Per-model-version verification of issued forecasts against observations."""
    from sqlalchemy import text
    now = now or datetime.datetime.now()
    verifiable_before = now - datetime.timedelta(hours=horizon_hours)
    with engine.connect() as conn:
        rows = [
            dict(r._mapping)
            for r in conn.execute(
                text(_sql_for(db_type)),
                {
                    "code": station_code,
                    "horizon": horizon_hours,
                    "verifiable_before": verifiable_before,
                },
            )
        ]
    def _ts(value):
        # sqlite hands back strings; postgres/mysql give datetimes
        if isinstance(value, datetime.datetime):
            return value
        return datetime.datetime.fromisoformat(str(value).replace(" ", "T"))
    by_version: Dict[str, List[dict]] = {}
    for r in rows:
        r["as_of"] = _ts(r["as_of"])
        # need most of the window observed, or the "max" is not the peak
        if r["observed_max"] is None or (r["observed_n"] or 0) < horizon_hours * 0.75:
            continue
        by_version.setdefault(r["model_version"] or "unknown", []).append(r)
    versions = []
    for version, vrows in by_version.items():
        pred = [float(r["predicted_max_level"]) for r in vrows]
        obs = [float(r["observed_max"]) for r in vrows]
        cur = [
            float(r["current_level"]) if r["current_level"] is not None else None
            for r in vrows
        ]
        err = [p - o for p, o in zip(pred, obs)]
        mae = sum(abs(e) for e in err) / len(err)
        bias = sum(err) / len(err)
        pers_rows = [(c, o) for c, o in zip(cur, obs) if c is not None]
        persistence = (
            sum(abs(c - o) for c, o in pers_rows) / len(pers_rows)
            if pers_rows
            else None
        )
        high = [(p, o) for p, o in zip(pred, obs) if o >= 2.0]
        versions.append(
            {
                "model_version": version,
                "first_issued": min(r["as_of"] for r in vrows).isoformat(),
                "last_issued": max(r["as_of"] for r in vrows).isoformat(),
                "n": len(vrows),
                "mae_m": round(mae, 3),
                "bias_m": round(bias, 3),
                "persistence_mae_m": (
                    None if persistence is None else round(persistence, 3)
                ),
                "skill": (
                    None if not persistence else round(1.0 - mae / persistence, 3)
                ),
                "above_2m_n": len(high),
                "above_2m_mae_m": (
                    round(sum(abs(p - o) for p, o in high) / len(high), 3)
                    if high
                    else None
                ),
                "enough_data": len(vrows) >= MIN_VERIFIED,
            }
        )
    versions.sort(key=lambda v: v["first_issued"])
    # Headline: current version vs the previous one that had enough data
    current = versions[-1] if versions else None
    previous = (
        next((v for v in reversed(versions[:-1]) if v["enough_data"]), None)
        if versions
        else None
    )
    trend = None
    if current and previous and current["enough_data"]:
        trend = {
            "previous_version": previous["model_version"],
            "mae_delta_m": round(current["mae_m"] - previous["mae_m"], 3),
            "skill_delta": (
                None
                if current["skill"] is None or previous["skill"] is None
                else round(current["skill"] - previous["skill"], 3)
            ),
            "better": current["mae_m"] < previous["mae_m"],
        }
    return {
        "station_code": station_code,
        "horizon_hours": horizon_hours,
        "verified_until": verifiable_before.isoformat(),
        "min_verified": MIN_VERIFIED,
        "versions": versions,
        "current": current,
        "trend": trend,
    }
@@ -45,6 +45,15 @@ MIN_SIGMA = 0.15
 MIN_ROWS_TO_TRAIN = 200
 MIN_ROWS_FOR_HEAD = 50
 class RainUnavailableError(RuntimeError):
    """Raised when a rain-enabled training run cannot obtain the rain series.
    Training would otherwise fall through to gauge-only (v2) bundles and
    overwrite the deployed v3 artifacts without anyone noticing.
    """
 HGB_PARAMS = {
    "max_iter": 300,
    "learning_rate": 0.06,
@@ -126,7 +135,8 @@ def _p_warning_series(
    """Model score if a classifier head exists, else the sigmoid-derived fallback probability."""
    if head is not None:
        return pd.Series(head.predict_proba(X)[:, 1], index=X.index)
-    predicted_max = pd.Series(reg.predict(X), index=X.index)
+    # reg predicts the RISE over current level; add the level back
    predicted_max = pd.Series(reg.predict(X), index=X.index) + X["level"]
    return 1.0 / (1.0 + np.exp(-(predicted_max - threshold) / sigma))
@@ -202,9 +212,11 @@ def train_station(
    split_train_end: str = SPLIT_B_TRAIN_END,
    split_test_start: str = SPLIT_B_TEST_START,
    split_test_end: str = SPLIT_B_TEST_END,
    rain: Optional[pd.Series] = None,
    dam: Optional[pd.DataFrame] = None,
 ) -> Tuple[Optional[dict], dict]:
    """Train every head for one station. Returns (bundle_or_None, station_metrics)."""
-    X, Y, meta = features.build_matrix(df_long, station, horizons)
+    X, Y, meta = features.build_matrix(df_long, station, horizons, rain=rain, dam=dam)
    if meta["n_rows"] < MIN_ROWS_TO_TRAIN:
        return None, {
            "status": "failed",
@@ -240,14 +252,22 @@ def train_station(
        )
        horizon_metrics: dict = {}
-        # --- regression head (max level) ---
+        # --- regression head (rise to future max) ---
        # Target = future max MINUS current level ("rise"). Rises are far more
        # stationary than absolute stages, which softens the cannot-exceed-
        # training-max ceiling: on the rolling-origin harness (2026-08-12) the
        # rise target moved P.1 first-alert leads from +0h to +6/+46h and cut
        # the 2024 record-peak underprediction. Prediction = rise + level.
        reg_labeled = eval_Y[max_col].notna()
        reg = None
        if reg_labeled.sum() >= MIN_ROWS_FOR_HEAD:
            rise_target = (
                eval_Y.loc[reg_labeled, max_col] - eval_X.loc[reg_labeled, "level"]
            )
            reg = _safe_fit(
                _make_regressor(hgb_overrides),
                eval_X.loc[reg_labeled],
-                eval_Y.loc[reg_labeled, max_col],
+                rise_target,
                f"max_{h}",
                skipped_heads,
            )
@@ -259,7 +279,10 @@ def train_station(
            test_labeled = Y_test[max_col].notna()
            if test_labeled.sum() > 0:
                y_true = Y_test.loc[test_labeled, max_col]
-                y_pred = reg.predict(X_test.loc[test_labeled])
+                y_pred = (
                    reg.predict(X_test.loc[test_labeled])
                    + X_test.loc[test_labeled, "level"].to_numpy()
                )
                residuals = y_true.to_numpy() - y_pred
                sigma_h = max(float(np.std(residuals)), MIN_SIGMA)
                horizon_metrics["n_test"] = int(test_labeled.sum())
@@ -297,9 +320,9 @@ def train_station(
                    skipped_heads,
                )
            else:
-                skipped_heads[
+                skipped_heads[head_key] = (
-                    head_key
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
            heads[head_key] = clf
            if not skip_eval:
@@ -367,7 +390,7 @@ def train_station(
            reg = _safe_fit(
                _make_regressor(hgb_overrides),
                X.loc[labeled],
-                Y.loc[labeled, max_col],
+                Y.loc[labeled, max_col] - X.loc[labeled, "level"],  # rise target
                head_key,
                skipped_heads,
            )
@@ -394,14 +417,24 @@ def train_station(
                if clf is not None:
                    skipped_heads.pop(head_key, None)
            else:
-                skipped_heads[
+                skipped_heads[head_key] = (
-                    head_key
+                    f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
-                ] = f"only {n_pos} positives in train span (< {MIN_POSITIVES_FOR_CLASSIFIER})"
+                )
                final_heads[head_key] = None
    # v4 = + Mae Ngat dam features; v3 = rise + rain; v2 = rise target only
    if "dam_storage_pct" in feature_names:
        version_prefix = "hgb-v4"
    elif "rain_24h" in feature_names:
        version_prefix = "hgb-v3"
    else:
        version_prefix = "hgb-v2"
    bundle = {
        "station_code": station,
-        "model_version": f"hgb-v1+{_git_short_sha()}",
+        "model_version": f"{version_prefix}+{_git_short_sha()}",
        # v2+: regression heads predict the RISE over the current level; the
        # serving side must add the level back. Old v1 bundles lack this key.
        "regression_target": "rise",
        "trained_at": datetime.datetime.now().isoformat(),
        "sklearn_version": sklearn.__version__,
        "feature_names": feature_names,
@@ -424,11 +457,80 @@ def train_all(
    models_dir: Path = Path("models"),
    skip_eval: bool = False,
    hgb_overrides: Optional[dict] = None,
    use_rain: bool = True,
    use_dam: bool = False,
    db_url: Optional[str] = None,
 ) -> dict:
    """Train and save every requested station's models. Returns the metrics.json payload."""
    models_dir = Path(models_dir)
    models_dir.mkdir(parents=True, exist_ok=True)
-    model_version = f"hgb-v1+{_git_short_sha()}"
+
    # Catchment rain (Open-Meteo archive, 2021+). A rain-less run produces v2
    # bundles that serve fine but have measurably less flood lead (the 2024
    # record flood: 13 h early with rain vs 18 h late without). The 2026-09-01
    # server retrain hit exactly that -- the archive fetch failed on a checkout
    # with no models/cache/ and the run quietly wrote v2 over v3. So the
    # downgrade is now an error unless the caller opts out with use_rain=False
    # (the --no-rain flag), which is the only way to get v2 deliberately.
    rain_series = None
    if use_rain:
        try:
            from . import rain as rain_mod
            rain_series = rain_mod.catchment_mean(rain_mod.load_history())
        except Exception as error:
            raise RainUnavailableError(
                f"rain history unavailable ({error}); refusing to silently "
                "downgrade to v2 bundles -- fix Open-Meteo access or restore "
                "models/cache/rain_openmeteo.csv.gz, or pass --no-rain to "
                "train gauge-only bundles on purpose"
            ) from error
        if rain_series is None:
            raise RainUnavailableError(
                "rain history unavailable (Open-Meteo archive unreachable and "
                "no models/cache/rain_openmeteo.csv.gz); refusing to silently "
                "downgrade to v2 bundles -- fix access, restore the cache file, "
                "or pass --no-rain to train gauge-only bundles on purpose"
            )
    if rain_series is not None:
        logger.info(
            f"rain series: {rain_series.index.min()} .. {rain_series.index.max()}"
        )
    # Mae Ngat reservoir state (rid_reservoir_daily, 2018+). OFF by default:
    # the 2026-08-13 backtest ablation showed every dam-feature subset COSTS
    # 1-3 h of first-alert lead on the 2024 record flood (the daily report
    # lags up to 31 h, so during fast onset the columns describe yesterday's
    # benign reservoir and damp the alarm). Kept as an opt-in for post-monsoon
    # re-evaluation once the 2026 season adds dam-era flood events.
    dam_frame = None
    if use_dam:
        try:
            from . import dam as dam_mod
            dam_frame = dam_mod.load_history(db_url=db_url)
        except Exception as error:
            logger.warning(f"dam history unavailable, training without it: {error}")
        if dam_frame is None:
            # load_history returns None (no raise) when both DB and cache
            # miss — an explicitly requested experiment must say so loudly.
            logger.warning(
                "--dam requested but no dam history available; "
                "training v3-style bundles WITHOUT dam features"
            )
    if dam_frame is not None:
        logger.info(f"dam series: {dam_frame.index.min()} .. {dam_frame.index.max()}")
    # Run-level version: v4 only if some requested station actually receives
    # dam columns (they are gated to DAM_STATIONS; per-bundle versions are
    # derived from each station's own feature_names and remain authoritative).
    if dam_frame is not None and any(s in features.DAM_STATIONS for s in stations):
        version_prefix = "hgb-v4"
    elif rain_series is not None:
        version_prefix = "hgb-v3"
    else:
        version_prefix = "hgb-v2"
    model_version = f"{version_prefix}+{_git_short_sha()}"
    station_results: Dict[str, dict] = {}
    for station in stations:
@@ -444,6 +546,8 @@ def train_all(
                horizons,
                skip_eval=skip_eval,
                hgb_overrides=hgb_overrides,
                rain=rain_series,
                dam=dam_frame,
            )
            if bundle is None:
                logger.warning(f"{station}: failed ({station_metrics.get('reason')})")
@@ -501,6 +605,19 @@ def main(argv: Optional[List[str]] = None) -> None:
    parser.add_argument(
        "--end", default=None, help="ISO date; latest measurement to load"
    )
    parser.add_argument(
        "--no-rain",
        action="store_true",
        help="DELIBERATELY train without the Open-Meteo rain features "
        "(v2-style bundles). Without this flag a missing rain series aborts "
        "the run instead of quietly downgrading the deployed model",
    )
    parser.add_argument(
        "--dam",
        action="store_true",
        help="EXPERIMENTAL: include Mae Ngat reservoir features (v4 bundles); "
        "the 2026-08 ablation showed they cost 1-3 h of alert lead",
    )
    args = parser.parse_args(argv)
    if args.stations == "all":
@@ -524,15 +641,33 @@ def main(argv: Optional[List[str]] = None) -> None:
    )
    metrics_payload = train_all(
-        df_long, stations, models_dir=Path(args.models_dir), skip_eval=args.skip_eval
+        df_long,
        stations,
        models_dir=Path(args.models_dir),
        skip_eval=args.skip_eval,
        use_rain=not args.no_rain,
        use_dam=args.dam,
        db_url=resolve_db_url(args.db_url),
    )
    trained = sum(
        1 for s in metrics_payload["stations"].values() if s["status"] == "trained"
    )
    logger.info(
-        f"Done: {trained}/{len(stations)} stations trained. metrics.json written to {args.models_dir}"
+        f"Done: {trained}/{len(stations)} stations trained "
        f"({metrics_payload['model_version']}). "
        f"metrics.json written to {args.models_dir}"
    )
 def cli() -> int:
    """Console entry: RainUnavailableError becomes a one-line error, exit 2."""
    try:
        main()
    except RainUnavailableError as error:
        logger.error(str(error))
        return 2
    return 0
 if __name__ == "__main__":
-    main()
+    raise SystemExit(cli())
@@ -0,0 +1,454 @@
 """Public flood notifications over ntfy.
 Runs once per collection cycle inside the API process (leader only), right
 after the forecast precompute, so it sees the same readings and forecasts the
 dashboard shows. Publishes to a self-hosted ntfy server; anyone subscribes to
 a topic from the free app or a browser, no account needed.
 Topics (all under one configurable prefix, default "ping"):
  {prefix}-{station}-warning     observed level crossed the station's warning threshold
  {prefix}-{station}-danger      observed level crossed the danger threshold
  {prefix}-warning               any station crossed warning (basin-wide digest)
  {prefix}-danger                any station crossed danger
  {prefix}-p1-outlook            model early warning for Chiang Mai city: P.1's 24 h
                                 warning probability crossed the alert level (opt-in;
                                 the forecast is experimental and says so)
  {prefix}-status                feed/monitor health: data stale, recovered
 Each notification is a TRANSITION, not a state: crossing UP into a level sends
 one message; dropping back below (with hysteresis) sends an all-clear. While
 the river sits above a threshold nothing is repeated, so a subscriber in a
 flood gets a handful of messages, not one an hour. The per-topic state is
 persisted (notification_state table) so a restart never re-sends.
 Everything is fail-safe: ntfy unreachable, table missing, malformed
 reading -> a logged warning, never an exception into the collection loop.
 """
 import datetime
 import logging
 from dataclasses import dataclass
 from typing import Dict, Iterable, List, Optional
 import requests
 from .ml import features
 logger = logging.getLogger(__name__)
 # Hysteresis: an all-clear needs the level this far BELOW the threshold, so a
 # river bobbing around 3.70 m does not toggle warning/clear every hour.
 CLEAR_MARGIN_M = 0.10
 # Capacity guard. The level thresholds in features.THRESHOLDS were calibrated
 # from RID's discharge_percent (% of channel capacity); if RID re-rates a
 # gauge or moves its datum, the level crosses while capacity says the channel
 # is nearly empty (P.77, 2026-09: 3.0 m "warning" at 22 %). A crossing is
 # only announced when the reported capacity agrees that the river is high.
 # P.1 is exempt: its stages come from the municipal inundation map, not from
 # capacity. Readings without a capacity figure fall back to level only.
 CAPACITY_GUARD_MIN_PCT = 60.0
 CAPACITY_GUARD_EXEMPT = {"P.1"}
 # Outlook alert fires when p_warning(24h) rises through ON, clears below OFF.
 OUTLOOK_ON = 0.50
 OUTLOOK_OFF = 0.25
 # Below this the outlook is not announced at all (avoid "5 % chance" noise).
 OUTLOOK_HORIZON = 24
 STATION_NAMES: Dict[str, str] = {
    "P.1": "Nawarat Bridge, Chiang Mai city",
    "P.103": "Ring Road Bridge 3, Chiang Mai",
    "P.67": "Ban Tae (Mae Taeng)",
    "P.21": "Ban Rim Tai (Mae Rim)",
    "P.75": "Ban Chai Lat",
    "P.92": "Ban Muang Aut",
    "P.20": "Ban Chiang Dao",
    "P.4A": "Ban Mae Taeng",
    "P.5": "Tha Nang Bridge (downstream)",
    "P.81": "Ban Pong (downstream)",
    "P.82": "Ban Sob Win",
    "P.84": "Ban Panton",
    "P.87": "Ban Pa Sang",
    "P.77": "Ban Sop Mae Sapuat",
    "P.85": "Ban Lai Kaew",
    "P.76": "Ban Mae I Hai",
 }
 def _slug(code: str) -> str:
    return code.lower().replace(".", "")
@dataclass
 class Notification:
    topic: str
    title: str
    message: str
    priority: int = 3  # ntfy: 1 min .. 5 max
    tags: Optional[List[str]] = None
    click: Optional[str] = None
 class NtfyPublisher:
    def __init__(
        self,
        server: str,
        prefix: str = "ping",
        token: Optional[str] = None,
        dashboard_url: str = "https://water.buildfor.life/",
        timeout: int = 10,
    ):
        self.server = server.rstrip("/")
        self.prefix = prefix
        self.token = token
        self.dashboard_url = dashboard_url
        self.timeout = timeout
    def topic(self, *parts: str) -> str:
        return "-".join([self.prefix, *parts])
    def publish(self, n: Notification) -> bool:
        headers = {
            "Title": n.title,
            "Priority": str(n.priority),
            "Click": n.click or self.dashboard_url,
            "Actions": f"view, Open dashboard, {n.click or self.dashboard_url}",
        }
        if n.tags:
            headers["Tags"] = ",".join(n.tags)
        if self.token:
            headers["Authorization"] = f"Bearer {self.token}"
        try:
            r = requests.post(
                f"{self.server}/{n.topic}",
                data=n.message.encode("utf-8"),
                headers=headers,
                timeout=self.timeout,
            )
            if r.status_code >= 300:
                logger.warning(f"ntfy {n.topic}: HTTP {r.status_code} {r.text[:120]}")
                return False
            return True
        except Exception as error:
            logger.warning(f"ntfy {n.topic}: {error}")
            return False
 class NotificationState:
    """Per-key last-sent state, in the monitor's own SQL database."""
    def __init__(self, engine, db_type: str):
        self.engine = engine
        self.db_type = db_type
        self._ensure()
    def _ensure(self) -> None:
        from sqlalchemy import text
        ddl = (
            "CREATE TABLE IF NOT EXISTS notification_state ("
            "key VARCHAR(64) PRIMARY KEY, state VARCHAR(16) NOT NULL, "
            "value NUMERIC(8,3), updated_at TIMESTAMP NOT NULL)"
        )
        try:
            with self.engine.begin() as conn:
                conn.execute(text(ddl))
        except Exception as error:
            # Postgres: two sessions racing CREATE TABLE IF NOT EXISTS can
            # both pass the existence check; the loser fails with a unique
            # violation on pg_type. The table exists either way; verify.
            with self.engine.connect() as conn:
                conn.execute(text("SELECT 1 FROM notification_state WHERE 1=0"))
            logger.debug(f"notification_state DDL raced, table present: {error}")
    def get(self, key: str) -> Optional[str]:
        from sqlalchemy import text
        with self.engine.connect() as conn:
            row = conn.execute(
                text("SELECT state FROM notification_state WHERE key = :k"), {"k": key}
            ).fetchone()
        return row[0] if row else None
    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
        from sqlalchemy import text
        now = datetime.datetime.now()
        with self.engine.begin() as conn:
            if self.db_type == "mysql":
                sql = (
                    "INSERT INTO notification_state (key, state, value, updated_at) "
                    "VALUES (:k, :s, :v, :t) ON DUPLICATE KEY UPDATE "
                    "state = VALUES(state), value = VALUES(value), updated_at = VALUES(updated_at)"
                )
            else:
                sql = (
                    "INSERT INTO notification_state (key, state, value, updated_at) "
                    "VALUES (:k, :s, :v, :t) ON CONFLICT (key) DO UPDATE SET "
                    "state = EXCLUDED.state, value = EXCLUDED.value, updated_at = EXCLUDED.updated_at"
                )
            conn.execute(text(sql), {"k": key, "s": state, "v": value, "t": now})
 class InMemoryState(NotificationState):
    """For tests and when no SQL engine is available (loses state on restart)."""
    def __init__(self):  # noqa: D107 - intentionally skips the SQL parent
        self._d: Dict[str, str] = {}
    def get(self, key: str) -> Optional[str]:
        return self._d.get(key)
    def set(self, key: str, state: str, value: Optional[float] = None) -> None:
        self._d[key] = state
 def _level_state(level: float, warn: float, danger: float, prev: Optional[str]) -> str:
    """'clear' | 'warning' | 'danger', with hysteresis on the way down."""
    if level >= danger:
        return "danger"
    if level >= warn:
        # from danger: stay 'danger' until below danger - margin
        if prev == "danger" and level >= danger - CLEAR_MARGIN_M:
            return "danger"
        return "warning"
    if prev in ("warning", "danger") and level >= warn - CLEAR_MARGIN_M:
        return "warning"
    return "clear"
 def evaluate(
    readings: Iterable[dict],
    forecasts: Iterable[dict],
    state: NotificationState,
    publisher: NtfyPublisher,
    stale_after_h: float = 3.0,
    now: Optional[datetime.datetime] = None,
 ) -> List[Notification]:
    """Compare current readings/forecasts with last-sent state; publish transitions.
    readings: rows with station_code, water_level, timestamp (latest per station)
    forecasts: /forecast rows (station_code, horizon_hours, p_warning, predicted_max_level)
    Returns the notifications that were published (for logs/tests).
    """
    now = now or datetime.datetime.now()
    sent: List[Notification] = []
    def emit(n: Notification) -> bool:
        ok = publisher.publish(n)
        if ok:
            sent.append(n)
        return ok
    # ---- observed levels, per station, plus basin-wide fan-out
    basin_changes: Dict[str, List[str]] = {"warning": [], "danger": [], "clear": []}
    latest_ts: Optional[datetime.datetime] = None
    for r in readings:
        code = r.get("station_code")
        level = r.get("water_level")
        if not code or level is None:
            continue
        try:
            level = float(level)
        except (TypeError, ValueError):
            continue
        ts = r.get("timestamp")
        if isinstance(ts, str):
            try:
                ts = datetime.datetime.fromisoformat(ts)
            except ValueError:
                ts = None
        if isinstance(ts, datetime.datetime) and (latest_ts is None or ts > latest_ts):
            latest_ts = ts
        warn, danger = features.get_thresholds(code)
        key = f"level:{code}"
        prev = state.get(key) or "clear"
        cur = _level_state(level, warn, danger, prev)
        pct = r.get("discharge_percent")
        if (
            cur != "clear"
            and prev == "clear"
            and code not in CAPACITY_GUARD_EXEMPT
            and pct is not None
        ):
            try:
                if float(pct) < CAPACITY_GUARD_MIN_PCT:
                    logger.info(
                        f"{code}: level {level:.2f} m >= {warn:.2f} but only "
                        f"{float(pct):.0f}% capacity; threshold looks stale, not alerting"
                    )
                    continue
            except (TypeError, ValueError):
                pass
        if cur == prev:
            continue
        name = STATION_NAMES.get(code, code)
        slug = _slug(code)
        when = (
            ts.strftime("%d %b %H:%M") if isinstance(ts, datetime.datetime) else "now"
        )
        if cur == "danger":
            ok = emit(
                Notification(
                    publisher.topic(slug, "danger"),
                    f"DANGER level at {code}",
                    f"{name}: {level:.2f} m at {when}, above the danger level of {danger:.2f} m.",
                    priority=5,
                    tags=["rotating_light", code],
                )
            )
            basin_changes["danger"].append(f"{code} {level:.2f} m")
        elif cur == "warning":
            if prev == "danger":
                ok = emit(
                    Notification(
                        publisher.topic(slug, "danger"),
                        f"{code} back below danger level",
                        f"{name}: {level:.2f} m at {when}; still above the warning level of {warn:.2f} m.",
                        priority=3,
                        tags=["arrow_down", code],
                    )
                )
                basin_changes["clear"].append(f"{code} below danger ({level:.2f} m)")
            else:
                ok = emit(
                    Notification(
                        publisher.topic(slug, "warning"),
                        f"Warning level at {code}",
                        f"{name}: {level:.2f} m at {when}, above the warning level of {warn:.2f} m.",
                        priority=4,
                        tags=["warning", code],
                    )
                )
                basin_changes["warning"].append(f"{code} {level:.2f} m")
        else:  # clear
            ok = emit(
                Notification(
                    publisher.topic(slug, "warning"),
                    f"{code} back to normal",
                    f"{name}: {level:.2f} m at {when}, below the warning level of {warn:.2f} m.",
                    priority=2,
                    tags=["white_check_mark", code],
                )
            )
            basin_changes["clear"].append(f"{code} normal ({level:.2f} m)")
        # Only remember the transition once it was actually delivered: if ntfy
        # was down, the next cycle retries instead of silently swallowing a
        # flood crossing.
        if ok:
            state.set(key, cur, level)
    if basin_changes["danger"]:
        emit(
            Notification(
                publisher.topic("danger"),
                "Ping River: danger level reached",
                "; ".join(basin_changes["danger"]),
                priority=5,
                tags=["rotating_light"],
            )
        )
    if basin_changes["warning"]:
        emit(
            Notification(
                publisher.topic("warning"),
                "Ping River: warning level reached",
                "; ".join(basin_changes["warning"]),
                priority=4,
                tags=["warning"],
            )
        )
    if basin_changes["clear"]:
        emit(
            Notification(
                publisher.topic("warning"),
                "Ping River: levels falling",
                "; ".join(basin_changes["clear"]),
                priority=2,
                tags=["white_check_mark"],
            )
        )
    # ---- model outlook for the city gauge (opt-in topic, experimental)
    p1 = next(
        (
            f
            for f in forecasts
            if f.get("station_code") == "P.1"
            and f.get("horizon_hours") == OUTLOOK_HORIZON
            and f.get("source") == "model"
        ),
        None,
    )
    if p1 and p1.get("p_warning") is not None:
        p = float(p1["p_warning"])
        key = "outlook:P.1"
        prev = state.get(key) or "off"
        cur = (
            "on" if (p >= OUTLOOK_ON or (prev == "on" and p >= OUTLOOK_OFF)) else "off"
        )
        if cur != prev:
            peak = p1.get("predicted_max_level")
            warn, _ = features.get_thresholds("P.1")
            if cur == "on":
                ok = emit(
                    Notification(
                        publisher.topic("p1-outlook"),
                        "Early warning: Chiang Mai flood risk rising",
                        f"The forecast model gives a {p * 100:.0f}% chance that Nawarat Bridge (P.1) "
                        f"reaches {warn:.2f} m within 24 h"
                        + (
                            f" (expected peak {float(peak):.2f} m)"
                            if peak is not None
                            else ""
                        )
                        + ". Experimental model output, not an official warning; "
                        "follow ThaiWater/TMD for official alerts.",
                        priority=4,
                        tags=["crystal_ball"],
                    )
                )
            else:
                ok = emit(
                    Notification(
                        publisher.topic("p1-outlook"),
                        "Chiang Mai flood risk easing",
                        f"The model's 24 h probability of reaching {warn:.2f} m at P.1 has dropped to {p * 100:.0f}%.",
                        priority=2,
                        tags=["crystal_ball"],
                    )
                )
            if ok:
                state.set(key, cur, p)
    # ---- feed health
    if latest_ts is not None:
        age_h = (now - latest_ts).total_seconds() / 3600.0
        key = "feed"
        prev = state.get(key) or "ok"
        cur = "stale" if age_h >= stale_after_h else "ok"
        if cur != prev:
            if cur == "stale":
                ok = emit(
                    Notification(
                        publisher.topic("status"),
                        "Ping River monitor: gauge feed stale",
                        f"No new readings for {age_h:.0f} h (last {latest_ts:%d %b %H:%M}). "
                        "Levels and forecasts on the dashboard are not current.",
                        priority=3,
                        tags=["hourglass"],
                    )
                )
            else:
                ok = emit(
                    Notification(
                        publisher.topic("status"),
                        "Ping River monitor: feed recovered",
                        f"Readings are current again (latest {latest_ts:%d %b %H:%M}).",
                        priority=2,
                        tags=["white_check_mark"],
                    )
                )
            if ok:
                state.set(key, cur, age_h)
    return sent
@@ -51,8 +51,7 @@ class PostgresHistory:
        if start >= end:
            raise ValueError("start must be before end")
-        query = text(
+        query = text("""
            """
            SELECT m.timestamp, s.station_code, m.water_level,
                   m.discharge, m.discharge_percent
            FROM water_measurements m
@@ -62,8 +61,7 @@ class PostgresHistory:
              AND m.timestamp <= :end_time
            ORDER BY m.timestamp ASC
            LIMIT :limit
-            """
+            """)
        )
        with self.engine.connect() as connection:
            rows = connection.execute(
                query,
@@ -91,9 +89,9 @@ class PostgresHistory:
                        "station_code": station_code,
                        "water_level": water_level,
                        "discharge": discharge,
-                        "discharge_percent": float(row[4])
+                        "discharge_percent": (
-                        if row[4] is not None
+                            float(row[4]) if row[4] is not None else None
-                        else None,
+                        ),
                    }
                )
            return result
--- a/Show More
+++ b/Show More
		`@@ -1,2 +0,0 @@`
			`DB_TYPE=postgresql`
			`POSTGRES_CONNECTION_STRING=postgresql://postgres:password@localhost:5432/water_monitoring`