Initial commit: Northern Thailand Ping River Monitor v3.1.0

Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!
2025-08-12 15:37:09 +07:00
commit af62cfef0b
60 changed files with 13267 additions and 0 deletions
--- a/docs/HTTPS_CONFIGURATION.md
+++ b/docs/HTTPS_CONFIGURATION.md
@@ -0,0 +1,389 @@
+# HTTPS VictoriaMetrics Configuration Guide
+
+This guide explains how to configure the Thailand Water Monitor to connect to VictoriaMetrics through HTTPS and reverse proxies.
+
+## Configuration Options
+
+### 1. Environment Variables for HTTPS
+
+```bash
+# Option 1: Full HTTPS URL (Recommended)
+export DB_TYPE=victoriametrics
+export VM_HOST=https://vm.example.com
+export VM_PORT=443
+
+# Option 2: Host and port separately
+export DB_TYPE=victoriametrics
+export VM_HOST=vm.example.com
+export VM_PORT=443
+
+# Option 3: Custom port with HTTPS
+export DB_TYPE=victoriametrics
+export VM_HOST=https://vm.example.com
+export VM_PORT=8443
+```
+
+### 2. Windows PowerShell Configuration
+
+```powershell
+# Set environment variables for HTTPS
+$env:DB_TYPE="victoriametrics"
+$env:VM_HOST="https://vm.example.com"
+$env:VM_PORT="443"
+
+# Run the water monitor
+python water_scraper_v3.py
+```
+
+### 3. Linux/Mac Configuration
+
+```bash
+# Set environment variables for HTTPS
+export DB_TYPE=victoriametrics
+export VM_HOST=https://vm.example.com
+export VM_PORT=443
+
+# Run the water monitor
+python water_scraper_v3.py
+```
+
+## Reverse Proxy Examples
+
+### 1. Nginx Reverse Proxy
+
+```nginx
+server {
+    listen 443 ssl http2;
+    server_name vm.example.com;
+    
+    # SSL Configuration
+    ssl_certificate /path/to/certificate.crt;
+    ssl_certificate_key /path/to/private.key;
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
+    
+    # Security headers
+    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+    add_header X-Frame-Options DENY always;
+    add_header X-Content-Type-Options nosniff always;
+    
+    # Optional: Basic authentication
+    # auth_basic "VictoriaMetrics";
+    # auth_basic_user_file /etc/nginx/.htpasswd;
+    
+    location / {
+        proxy_pass http://localhost:8428;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        
+        # WebSocket support (if needed)
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        
+        # Timeouts
+        proxy_connect_timeout 60s;
+        proxy_send_timeout 60s;
+        proxy_read_timeout 60s;
+    }
+}
+
+# Redirect HTTP to HTTPS
+server {
+    listen 80;
+    server_name vm.example.com;
+    return 301 https://$server_name$request_uri;
+}
+```
+
+### 2. Apache Reverse Proxy
+
+```apache
+<VirtualHost *:443>
+    ServerName vm.example.com
+    
+    # SSL Configuration
+    SSLEngine on
+    SSLCertificateFile /path/to/certificate.crt
+    SSLCertificateKeyFile /path/to/private.key
+    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
+    SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
+    
+    # Security headers
+    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
+    Header always set X-Frame-Options DENY
+    Header always set X-Content-Type-Options nosniff
+    
+    # Reverse proxy configuration
+    ProxyPreserveHost On
+    ProxyPass / http://localhost:8428/
+    ProxyPassReverse / http://localhost:8428/
+    
+    # Optional: Basic authentication
+    # AuthType Basic
+    # AuthName "VictoriaMetrics"
+    # AuthUserFile /etc/apache2/.htpasswd
+    # Require valid-user
+</VirtualHost>
+
+<VirtualHost *:80>
+    ServerName vm.example.com
+    Redirect permanent / https://vm.example.com/
+</VirtualHost>
+```
+
+### 3. Traefik Reverse Proxy
+
+```yaml
+# docker-compose.yml with Traefik
+version: '3.8'
+
+services:
+  traefik:
+    image: traefik:v2.10
+    command:
+      - --api.dashboard=true
+      - --entrypoints.web.address=:80
+      - --entrypoints.websecure.address=:443
+      - --providers.docker=true
+      - --certificatesresolvers.letsencrypt.acme.tlschallenge=true
+      - --certificatesresolvers.letsencrypt.acme.email=admin@example.com
+      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - letsencrypt:/letsencrypt
+    labels:
+      - traefik.http.routers.api.rule=Host(`traefik.example.com`)
+      - traefik.http.routers.api.tls.certresolver=letsencrypt
+
+  victoriametrics:
+    image: victoriametrics/victoria-metrics:latest
+    command:
+      - '--storageDataPath=/victoria-metrics-data'
+      - '--retentionPeriod=2y'
+      - '--httpListenAddr=:8428'
+    volumes:
+      - vm_data:/victoria-metrics-data
+    labels:
+      - traefik.enable=true
+      - traefik.http.routers.vm.rule=Host(`vm.example.com`)
+      - traefik.http.routers.vm.tls.certresolver=letsencrypt
+      - traefik.http.services.vm.loadbalancer.server.port=8428
+
+volumes:
+  vm_data:
+  letsencrypt:
+```
+
+## Testing HTTPS Configuration
+
+### 1. Test Connection
+
+```bash
+# Test HTTPS connection
+curl -k https://vm.example.com/health
+
+# Test with specific port
+curl -k https://vm.example.com:8443/health
+
+# Test API endpoint
+curl -k "https://vm.example.com/api/v1/query?query=up"
+```
+
+### 2. Test with Water Monitor
+
+```bash
+# Set environment variables
+export DB_TYPE=victoriametrics
+export VM_HOST=https://vm.example.com
+export VM_PORT=443
+
+# Test with demo script
+python demo_databases.py victoriametrics
+
+# Run full water monitor
+python water_scraper_v3.py
+```
+
+### 3. Verify SSL Certificate
+
+```bash
+# Check SSL certificate
+openssl s_client -connect vm.example.com:443 -servername vm.example.com
+
+# Check certificate expiration
+echo | openssl s_client -connect vm.example.com:443 2>/dev/null | openssl x509 -noout -dates
+```
+
+## Configuration Examples
+
+### 1. Production HTTPS Setup
+
+```bash
+# Environment variables for production
+export DB_TYPE=victoriametrics
+export VM_HOST=https://metrics.company.com
+export VM_PORT=443
+export LOG_LEVEL=INFO
+export SCRAPING_INTERVAL_HOURS=1
+
+# Run water monitor
+python water_scraper_v3.py
+```
+
+### 2. Development with Self-Signed Certificate
+
+```bash
+# For development with self-signed certificates
+export DB_TYPE=victoriametrics
+export VM_HOST=https://dev-vm.local
+export VM_PORT=443
+export PYTHONHTTPSVERIFY=0  # Disable SSL verification (dev only)
+
+python water_scraper_v3.py
+```
+
+### 3. Custom Port Configuration
+
+```bash
+# Custom HTTPS port
+export DB_TYPE=victoriametrics
+export VM_HOST=https://vm.example.com
+export VM_PORT=8443
+
+python water_scraper_v3.py
+```
+
+## Troubleshooting HTTPS Issues
+
+### 1. SSL Certificate Errors
+
+```bash
+# Error: SSL certificate verify failed
+# Solution: Check certificate validity
+openssl x509 -in certificate.crt -text -noout
+
+# Temporary workaround (not recommended for production)
+export PYTHONHTTPSVERIFY=0
+```
+
+### 2. Connection Timeout
+
+```bash
+# Error: Connection timeout
+# Check firewall and network connectivity
+telnet vm.example.com 443
+nc -zv vm.example.com 443
+```
+
+### 3. DNS Resolution Issues
+
+```bash
+# Error: Name resolution failed
+# Check DNS resolution
+nslookup vm.example.com
+dig vm.example.com
+```
+
+### 4. Proxy Configuration Issues
+
+```bash
+# Check proxy logs
+# Nginx
+tail -f /var/log/nginx/error.log
+
+# Apache
+tail -f /var/log/apache2/error.log
+
+# Test direct connection to backend
+curl http://localhost:8428/health
+```
+
+## Security Best Practices
+
+### 1. SSL/TLS Configuration
+
+- Use TLS 1.2 or higher
+- Disable weak ciphers
+- Enable HSTS headers
+- Use strong SSL certificates
+
+### 2. Authentication
+
+```nginx
+# Basic authentication in Nginx
+auth_basic "VictoriaMetrics Access";
+auth_basic_user_file /etc/nginx/.htpasswd;
+
+# Create password file
+htpasswd -c /etc/nginx/.htpasswd username
+```
+
+### 3. Network Security
+
+- Use firewall rules to restrict access
+- Consider VPN for internal access
+- Implement rate limiting
+- Monitor access logs
+
+### 4. Certificate Management
+
+```bash
+# Auto-renewal with Let's Encrypt
+certbot renew --dry-run
+
+# Certificate monitoring
+echo | openssl s_client -connect vm.example.com:443 2>/dev/null | \
+openssl x509 -noout -dates | grep notAfter
+```
+
+## Docker Configuration for HTTPS
+
+### 1. Docker Compose with HTTPS
+
+```yaml
+version: '3.8'
+
+services:
+  water-monitor:
+    build: .
+    environment:
+      - DB_TYPE=victoriametrics
+      - VM_HOST=https://vm.example.com
+      - VM_PORT=443
+    restart: unless-stopped
+    depends_on:
+      - victoriametrics
+
+  victoriametrics:
+    image: victoriametrics/victoria-metrics:latest
+    ports:
+      - "8428:8428"
+    volumes:
+      - vm_data:/victoria-metrics-data
+    command:
+      - '--storageDataPath=/victoria-metrics-data'
+      - '--retentionPeriod=2y'
+      - '--httpListenAddr=:8428'
+
+volumes:
+  vm_data:
+```
+
+### 2. Environment File (.env)
+
+```bash
+# .env file
+DB_TYPE=victoriametrics
+VM_HOST=https://vm.example.com
+VM_PORT=443
+LOG_LEVEL=INFO
+SCRAPING_INTERVAL_HOURS=1
+```
+
+This configuration guide provides comprehensive instructions for setting up HTTPS connectivity to VictoriaMetrics through reverse proxies, ensuring secure and reliable data transmission for the Thailand Water Monitor.