Features: - Real-time water level monitoring for Ping River Basin (16 stations) - Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand - FastAPI web interface with interactive dashboard and station management - Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics) - Comprehensive monitoring with health checks and metrics collection - Docker deployment with Grafana integration - Production-ready architecture with enterprise-grade observability CI/CD & Automation: - Complete Gitea Actions workflows for CI/CD, security, and releases - Multi-Python version testing (3.9-3.12) - Multi-architecture Docker builds (amd64, arm64) - Daily security scanning and dependency monitoring - Automated documentation generation - Performance testing and validation Production Ready: - Type safety with Pydantic models and comprehensive type hints - Data validation layer with range checking and error handling - Rate limiting and request tracking for API protection - Enhanced logging with rotation, colors, and performance metrics - Station management API for dynamic CRUD operations - Comprehensive documentation and deployment guides Technical Stack: - Python 3.9+ with FastAPI and Pydantic - Multi-database architecture with adapter pattern - Docker containerization with multi-stage builds - Grafana dashboards for visualization - Gitea Actions for CI/CD automation - Enterprise monitoring and alerting Ready for deployment to B4L infrastructure!
10 KiB
VictoriaMetrics Setup Guide for Thailand Water Monitor
This guide provides comprehensive instructions for setting up VictoriaMetrics as the time-series database backend for the Thailand Water Monitor.
Why VictoriaMetrics?
VictoriaMetrics is an excellent choice for water monitoring data because:
- High Performance: Up to 10x faster than InfluxDB
- Low Resource Usage: Uses 10x less RAM than Prometheus
- Better Compression: 70x better compression than Prometheus
- Prometheus Compatible: Drop-in replacement for Prometheus
- Easy to Deploy: Single binary, no dependencies
- Cost Effective: Open source with commercial support available
Quick Start
1. Environment Variables
Set these environment variables to configure VictoriaMetrics:
# Windows (PowerShell)
$env:DB_TYPE="victoriametrics"
$env:VM_HOST="localhost"
$env:VM_PORT="8428"
# Linux/Mac
export DB_TYPE=victoriametrics
export VM_HOST=localhost
export VM_PORT=8428
2. Start VictoriaMetrics with Docker
# Simple setup
docker run -d \
--name victoriametrics \
-p 8428:8428 \
-v victoria-metrics-data:/victoria-metrics-data \
victoriametrics/victoria-metrics:latest \
--storageDataPath=/victoria-metrics-data \
--retentionPeriod=2y \
--httpListenAddr=:8428
# Verify it's running
curl http://localhost:8428/health
3. Run the Water Monitor
python water_scraper_v3.py
4. Access Grafana Dashboard
# Start with Docker Compose (includes Grafana)
docker-compose -f docker-compose.victoriametrics.yml up -d
# Access Grafana at http://localhost:3000
# Username: admin
# Password: admin_password
Production Setup
Docker Compose Configuration
Use the provided docker-compose.victoriametrics.yml
file:
# Start the complete stack
docker-compose -f docker-compose.victoriametrics.yml up -d
# Check status
docker-compose -f docker-compose.victoriametrics.yml ps
# View logs
docker-compose -f docker-compose.victoriametrics.yml logs -f
Manual VictoriaMetrics Configuration
High-Performance Configuration
docker run -d \
--name victoriametrics \
-p 8428:8428 \
-v victoria-metrics-data:/victoria-metrics-data \
victoriametrics/victoria-metrics:latest \
--storageDataPath=/victoria-metrics-data \
--retentionPeriod=2y \
--httpListenAddr=:8428 \
--maxConcurrentInserts=32 \
--search.maxQueryDuration=60s \
--search.maxConcurrentRequests=16 \
--dedup.minScrapeInterval=30s \
--memory.allowedPercent=80 \
--loggerLevel=INFO \
--loggerFormat=json \
--search.maxSeries=1000000 \
--search.maxPointsPerTimeseries=100000
Configuration Parameters Explained
Parameter | Description | Recommended Value |
---|---|---|
--storageDataPath |
Data storage directory | /victoria-metrics-data |
--retentionPeriod |
How long to keep data | 2y (2 years) |
--httpListenAddr |
HTTP listen address | :8428 |
--maxConcurrentInserts |
Max concurrent inserts | 32 |
--search.maxQueryDuration |
Max query duration | 60s |
--search.maxConcurrentRequests |
Max concurrent queries | 16 |
--dedup.minScrapeInterval |
Deduplication interval | 30s |
--memory.allowedPercent |
Max memory usage | 80 |
--loggerLevel |
Log level | INFO |
--search.maxSeries |
Max time series | 1000000 |
Kubernetes Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: victoriametrics
spec:
replicas: 1
selector:
matchLabels:
app: victoriametrics
template:
metadata:
labels:
app: victoriametrics
spec:
containers:
- name: victoriametrics
image: victoriametrics/victoria-metrics:latest
ports:
- containerPort: 8428
args:
- --storageDataPath=/victoria-metrics-data
- --retentionPeriod=2y
- --httpListenAddr=:8428
- --maxConcurrentInserts=32
volumeMounts:
- name: storage
mountPath: /victoria-metrics-data
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2000m"
volumes:
- name: storage
persistentVolumeClaim:
claimName: victoriametrics-pvc
---
apiVersion: v1
kind: Service
metadata:
name: victoriametrics
spec:
selector:
app: victoriametrics
ports:
- port: 8428
targetPort: 8428
type: ClusterIP
Data Queries
HTTP API Queries
VictoriaMetrics provides a Prometheus-compatible HTTP API:
# Current water levels for all stations
curl "http://localhost:8428/api/v1/query?query=water_level"
# Water levels for specific station
curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
# Average discharge over last hour
curl "http://localhost:8428/api/v1/query?query=avg_over_time(water_discharge[1h])"
# High discharge alerts (>80%)
curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
# Time range query (last 6 hours)
START=$(date -d '6 hours ago' +%s)
END=$(date +%s)
curl "http://localhost:8428/api/v1/query_range?query=water_level&start=${START}&end=${END}&step=300"
PromQL Examples
# Current water levels
water_level
# Water level trends (last 24h)
water_level[24h]
# Discharge rates by station
water_discharge{station_code="P.1"}
# Average discharge across all stations
avg(water_discharge)
# Stations with high discharge (>80%)
water_discharge_percent > 80
# Rate of change in water level
rate(water_level[5m])
# Maximum water level in last hour
max_over_time(water_level[1h])
# Stations with increasing water levels
increase(water_level[1h]) > 0
Grafana Integration
Data Source Configuration
-
Add VictoriaMetrics as Prometheus Data Source:
- URL:
http://localhost:8428
(orhttp://victoriametrics:8428
in Docker) - Access: Server (default)
- HTTP Method: POST
- URL:
-
Import Dashboard:
- Use the provided
water-monitoring-dashboard.json
- Or create custom dashboards with the queries above
- Use the provided
Dashboard Panels
The included dashboard provides:
- Time Series: Water levels and discharge over time
- Table: Current status of all stations
- Pie Chart: Discharge percentage distribution
- Gauge: Average discharge percentage
- Variables: Filter by station
Monitoring and Maintenance
Health Checks
# Check VictoriaMetrics health
curl http://localhost:8428/health
# Check metrics endpoint
curl http://localhost:8428/metrics
# Check configuration
curl http://localhost:8428/api/v1/status/config
Performance Monitoring
# Query performance stats
curl http://localhost:8428/api/v1/status/tsdb
# Memory usage
curl http://localhost:8428/api/v1/status/runtime
# Active queries
curl http://localhost:8428/api/v1/status/active_queries
Backup and Restore
# Create backup
docker exec victoriametrics /usr/bin/vmbackup \
-storageDataPath=/victoria-metrics-data \
-dst=fs:///backup/$(date +%Y%m%d)
# Restore from backup
docker exec victoriametrics /usr/bin/vmrestore \
-src=fs:///backup/20250724 \
-storageDataPath=/victoria-metrics-data
Log Analysis
# View logs
docker logs victoriametrics
# Follow logs
docker logs -f victoriametrics
# Search for errors
docker logs victoriametrics 2>&1 | grep ERROR
Troubleshooting
Common Issues
-
Connection Refused:
# Check if VictoriaMetrics is running docker ps | grep victoriametrics # Check port binding netstat -tlnp | grep 8428
-
High Memory Usage:
# Reduce memory limit docker run ... --memory.allowedPercent=60 ...
-
Slow Queries:
# Increase query timeout docker run ... --search.maxQueryDuration=120s ...
-
Data Not Appearing:
# Check if data is being written curl "http://localhost:8428/api/v1/query?query=up" # Check water monitor logs tail -f water_monitor.log
Performance Tuning
-
For High Write Load:
--maxConcurrentInserts=64 --insert.maxQueueDuration=60s
-
For High Query Load:
--search.maxConcurrentRequests=32 --search.maxQueryDuration=120s
-
For Large Datasets:
--search.maxSeries=10000000 --search.maxPointsPerTimeseries=1000000
Security
Authentication
VictoriaMetrics doesn't have built-in authentication. Use a reverse proxy:
server {
listen 80;
server_name victoriametrics.example.com;
auth_basic "VictoriaMetrics";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass http://localhost:8428;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
TLS/SSL
# Use nginx or traefik for TLS termination
# Or use VictoriaMetrics with TLS:
docker run ... \
-v /path/to/cert.pem:/cert.pem \
-v /path/to/key.pem:/key.pem \
victoriametrics/victoria-metrics:latest \
--tls \
--tlsCertFile=/cert.pem \
--tlsKeyFile=/key.pem
Scaling
Cluster Setup
For high availability and horizontal scaling:
# Start multiple VictoriaMetrics instances
docker run -d --name vm1 -p 8428:8428 victoriametrics/victoria-metrics:latest
docker run -d --name vm2 -p 8429:8428 victoriametrics/victoria-metrics:latest
# Use load balancer to distribute queries
# Use vminsert/vmselect/vmstorage for true clustering
Resource Requirements
Data Points/Hour | RAM | CPU | Storage/Day |
---|---|---|---|
1,000 | 100MB | 0.1 CPU | 10MB |
10,000 | 500MB | 0.5 CPU | 100MB |
100,000 | 2GB | 1 CPU | 1GB |
1,000,000 | 8GB | 2 CPU | 10GB |
Migration
From InfluxDB
# Export from InfluxDB
influx -database water_monitoring -execute "SELECT * FROM water_data" -format csv > data.csv
# Import to VictoriaMetrics (convert to Prometheus format first)
# Use vmctl tool for migration
From Prometheus
# Use vmctl for direct migration
vmctl prometheus --prom-snapshot=/path/to/prometheus/data --vm-addr=http://localhost:8428
This comprehensive setup guide should help you configure VictoriaMetrics for optimal performance with the Thailand Water Monitor system.