Initial commit: Northern Thailand Ping River Monitor v3.1.0

Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!
2025-08-12 15:37:09 +07:00
commit af62cfef0b
60 changed files with 13267 additions and 0 deletions
--- a/docs/VICTORIAMETRICS_SETUP.md
+++ b/docs/VICTORIAMETRICS_SETUP.md
@@ -0,0 +1,443 @@
+# VictoriaMetrics Setup Guide for Thailand Water Monitor
+
+This guide provides comprehensive instructions for setting up VictoriaMetrics as the time-series database backend for the Thailand Water Monitor.
+
+## Why VictoriaMetrics?
+
+VictoriaMetrics is an excellent choice for water monitoring data because:
+
+- **High Performance**: Up to 10x faster than InfluxDB
+- **Low Resource Usage**: Uses 10x less RAM than Prometheus
+- **Better Compression**: 70x better compression than Prometheus
+- **Prometheus Compatible**: Drop-in replacement for Prometheus
+- **Easy to Deploy**: Single binary, no dependencies
+- **Cost Effective**: Open source with commercial support available
+
+## Quick Start
+
+### 1. Environment Variables
+
+Set these environment variables to configure VictoriaMetrics:
+
+```bash
+# Windows (PowerShell)
+$env:DB_TYPE="victoriametrics"
+$env:VM_HOST="localhost"
+$env:VM_PORT="8428"
+
+# Linux/Mac
+export DB_TYPE=victoriametrics
+export VM_HOST=localhost
+export VM_PORT=8428
+```
+
+### 2. Start VictoriaMetrics with Docker
+
+```bash
+# Simple setup
+docker run -d \
+  --name victoriametrics \
+  -p 8428:8428 \
+  -v victoria-metrics-data:/victoria-metrics-data \
+  victoriametrics/victoria-metrics:latest \
+  --storageDataPath=/victoria-metrics-data \
+  --retentionPeriod=2y \
+  --httpListenAddr=:8428
+
+# Verify it's running
+curl http://localhost:8428/health
+```
+
+### 3. Run the Water Monitor
+
+```bash
+python water_scraper_v3.py
+```
+
+### 4. Access Grafana Dashboard
+
+```bash
+# Start with Docker Compose (includes Grafana)
+docker-compose -f docker-compose.victoriametrics.yml up -d
+
+# Access Grafana at http://localhost:3000
+# Username: admin
+# Password: admin_password
+```
+
+## Production Setup
+
+### Docker Compose Configuration
+
+Use the provided `docker-compose.victoriametrics.yml` file:
+
+```bash
+# Start the complete stack
+docker-compose -f docker-compose.victoriametrics.yml up -d
+
+# Check status
+docker-compose -f docker-compose.victoriametrics.yml ps
+
+# View logs
+docker-compose -f docker-compose.victoriametrics.yml logs -f
+```
+
+### Manual VictoriaMetrics Configuration
+
+#### High-Performance Configuration
+
+```bash
+docker run -d \
+  --name victoriametrics \
+  -p 8428:8428 \
+  -v victoria-metrics-data:/victoria-metrics-data \
+  victoriametrics/victoria-metrics:latest \
+  --storageDataPath=/victoria-metrics-data \
+  --retentionPeriod=2y \
+  --httpListenAddr=:8428 \
+  --maxConcurrentInserts=32 \
+  --search.maxQueryDuration=60s \
+  --search.maxConcurrentRequests=16 \
+  --dedup.minScrapeInterval=30s \
+  --memory.allowedPercent=80 \
+  --loggerLevel=INFO \
+  --loggerFormat=json \
+  --search.maxSeries=1000000 \
+  --search.maxPointsPerTimeseries=100000
+```
+
+#### Configuration Parameters Explained
+
+| Parameter | Description | Recommended Value |
+|-----------|-------------|-------------------|
+| `--storageDataPath` | Data storage directory | `/victoria-metrics-data` |
+| `--retentionPeriod` | How long to keep data | `2y` (2 years) |
+| `--httpListenAddr` | HTTP listen address | `:8428` |
+| `--maxConcurrentInserts` | Max concurrent inserts | `32` |
+| `--search.maxQueryDuration` | Max query duration | `60s` |
+| `--search.maxConcurrentRequests` | Max concurrent queries | `16` |
+| `--dedup.minScrapeInterval` | Deduplication interval | `30s` |
+| `--memory.allowedPercent` | Max memory usage | `80` |
+| `--loggerLevel` | Log level | `INFO` |
+| `--search.maxSeries` | Max time series | `1000000` |
+
+### Kubernetes Deployment
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: victoriametrics
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: victoriametrics
+  template:
+    metadata:
+      labels:
+        app: victoriametrics
+    spec:
+      containers:
+      - name: victoriametrics
+        image: victoriametrics/victoria-metrics:latest
+        ports:
+        - containerPort: 8428
+        args:
+        - --storageDataPath=/victoria-metrics-data
+        - --retentionPeriod=2y
+        - --httpListenAddr=:8428
+        - --maxConcurrentInserts=32
+        volumeMounts:
+        - name: storage
+          mountPath: /victoria-metrics-data
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "500m"
+          limits:
+            memory: "2Gi"
+            cpu: "2000m"
+      volumes:
+      - name: storage
+        persistentVolumeClaim:
+          claimName: victoriametrics-pvc
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: victoriametrics
+spec:
+  selector:
+    app: victoriametrics
+  ports:
+  - port: 8428
+    targetPort: 8428
+  type: ClusterIP
+```
+
+## Data Queries
+
+### HTTP API Queries
+
+VictoriaMetrics provides a Prometheus-compatible HTTP API:
+
+```bash
+# Current water levels for all stations
+curl "http://localhost:8428/api/v1/query?query=water_level"
+
+# Water levels for specific station
+curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
+
+# Average discharge over last hour
+curl "http://localhost:8428/api/v1/query?query=avg_over_time(water_discharge[1h])"
+
+# High discharge alerts (>80%)
+curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
+
+# Time range query (last 6 hours)
+START=$(date -d '6 hours ago' +%s)
+END=$(date +%s)
+curl "http://localhost:8428/api/v1/query_range?query=water_level&start=${START}&end=${END}&step=300"
+```
+
+### PromQL Examples
+
+```promql
+# Current water levels
+water_level
+
+# Water level trends (last 24h)
+water_level[24h]
+
+# Discharge rates by station
+water_discharge{station_code="P.1"}
+
+# Average discharge across all stations
+avg(water_discharge)
+
+# Stations with high discharge (>80%)
+water_discharge_percent > 80
+
+# Rate of change in water level
+rate(water_level[5m])
+
+# Maximum water level in last hour
+max_over_time(water_level[1h])
+
+# Stations with increasing water levels
+increase(water_level[1h]) > 0
+```
+
+## Grafana Integration
+
+### Data Source Configuration
+
+1. **Add VictoriaMetrics as Prometheus Data Source**:
+   - URL: `http://localhost:8428` (or `http://victoriametrics:8428` in Docker)
+   - Access: Server (default)
+   - HTTP Method: POST
+
+2. **Import Dashboard**:
+   - Use the provided `water-monitoring-dashboard.json`
+   - Or create custom dashboards with the queries above
+
+### Dashboard Panels
+
+The included dashboard provides:
+
+- **Time Series**: Water levels and discharge over time
+- **Table**: Current status of all stations
+- **Pie Chart**: Discharge percentage distribution
+- **Gauge**: Average discharge percentage
+- **Variables**: Filter by station
+
+## Monitoring and Maintenance
+
+### Health Checks
+
+```bash
+# Check VictoriaMetrics health
+curl http://localhost:8428/health
+
+# Check metrics endpoint
+curl http://localhost:8428/metrics
+
+# Check configuration
+curl http://localhost:8428/api/v1/status/config
+```
+
+### Performance Monitoring
+
+```bash
+# Query performance stats
+curl http://localhost:8428/api/v1/status/tsdb
+
+# Memory usage
+curl http://localhost:8428/api/v1/status/runtime
+
+# Active queries
+curl http://localhost:8428/api/v1/status/active_queries
+```
+
+### Backup and Restore
+
+```bash
+# Create backup
+docker exec victoriametrics /usr/bin/vmbackup \
+  -storageDataPath=/victoria-metrics-data \
+  -dst=fs:///backup/$(date +%Y%m%d)
+
+# Restore from backup
+docker exec victoriametrics /usr/bin/vmrestore \
+  -src=fs:///backup/20250724 \
+  -storageDataPath=/victoria-metrics-data
+```
+
+### Log Analysis
+
+```bash
+# View logs
+docker logs victoriametrics
+
+# Follow logs
+docker logs -f victoriametrics
+
+# Search for errors
+docker logs victoriametrics 2>&1 | grep ERROR
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Connection Refused**:
+   ```bash
+   # Check if VictoriaMetrics is running
+   docker ps | grep victoriametrics
+   
+   # Check port binding
+   netstat -tlnp | grep 8428
+   ```
+
+2. **High Memory Usage**:
+   ```bash
+   # Reduce memory limit
+   docker run ... --memory.allowedPercent=60 ...
+   ```
+
+3. **Slow Queries**:
+   ```bash
+   # Increase query timeout
+   docker run ... --search.maxQueryDuration=120s ...
+   ```
+
+4. **Data Not Appearing**:
+   ```bash
+   # Check if data is being written
+   curl "http://localhost:8428/api/v1/query?query=up"
+   
+   # Check water monitor logs
+   tail -f water_monitor.log
+   ```
+
+### Performance Tuning
+
+1. **For High Write Load**:
+   ```bash
+   --maxConcurrentInserts=64
+   --insert.maxQueueDuration=60s
+   ```
+
+2. **For High Query Load**:
+   ```bash
+   --search.maxConcurrentRequests=32
+   --search.maxQueryDuration=120s
+   ```
+
+3. **For Large Datasets**:
+   ```bash
+   --search.maxSeries=10000000
+   --search.maxPointsPerTimeseries=1000000
+   ```
+
+## Security
+
+### Authentication
+
+VictoriaMetrics doesn't have built-in authentication. Use a reverse proxy:
+
+```nginx
+server {
+    listen 80;
+    server_name victoriametrics.example.com;
+    
+    auth_basic "VictoriaMetrics";
+    auth_basic_user_file /etc/nginx/.htpasswd;
+    
+    location / {
+        proxy_pass http://localhost:8428;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+    }
+}
+```
+
+### TLS/SSL
+
+```bash
+# Use nginx or traefik for TLS termination
+# Or use VictoriaMetrics with TLS:
+docker run ... \
+  -v /path/to/cert.pem:/cert.pem \
+  -v /path/to/key.pem:/key.pem \
+  victoriametrics/victoria-metrics:latest \
+  --tls \
+  --tlsCertFile=/cert.pem \
+  --tlsKeyFile=/key.pem
+```
+
+## Scaling
+
+### Cluster Setup
+
+For high availability and horizontal scaling:
+
+```bash
+# Start multiple VictoriaMetrics instances
+docker run -d --name vm1 -p 8428:8428 victoriametrics/victoria-metrics:latest
+docker run -d --name vm2 -p 8429:8428 victoriametrics/victoria-metrics:latest
+
+# Use load balancer to distribute queries
+# Use vminsert/vmselect/vmstorage for true clustering
+```
+
+### Resource Requirements
+
+| Data Points/Hour | RAM | CPU | Storage/Day |
+|------------------|-----|-----|-------------|
+| 1,000 | 100MB | 0.1 CPU | 10MB |
+| 10,000 | 500MB | 0.5 CPU | 100MB |
+| 100,000 | 2GB | 1 CPU | 1GB |
+| 1,000,000 | 8GB | 2 CPU | 10GB |
+
+## Migration
+
+### From InfluxDB
+
+```bash
+# Export from InfluxDB
+influx -database water_monitoring -execute "SELECT * FROM water_data" -format csv > data.csv
+
+# Import to VictoriaMetrics (convert to Prometheus format first)
+# Use vmctl tool for migration
+```
+
+### From Prometheus
+
+```bash
+# Use vmctl for direct migration
+vmctl prometheus --prom-snapshot=/path/to/prometheus/data --vm-addr=http://localhost:8428
+```
+
+This comprehensive setup guide should help you configure VictoriaMetrics for optimal performance with the Thailand Water Monitor system.