Northern-Thailand-Ping-River-Monitor/docs/VICTORIAMETRICS_SETUP.md

VictoriaMetrics Setup Guide for Thailand Water Monitor

This guide provides comprehensive instructions for setting up VictoriaMetrics as the time-series database backend for the Thailand Water Monitor.

Why VictoriaMetrics?

VictoriaMetrics is an excellent choice for water monitoring data because:

• High Performance: Up to 10x faster than InfluxDB
• Low Resource Usage: Uses 10x less RAM than Prometheus
• Better Compression: 70x better compression than Prometheus
• Prometheus Compatible: Drop-in replacement for Prometheus
• Easy to Deploy: Single binary, no dependencies
• Cost Effective: Open source with commercial support available

Quick Start

1. Environment Variables

Set these environment variables to configure VictoriaMetrics:

# Windows (PowerShell)
$env:DB_TYPE="victoriametrics"
$env:VM_HOST="localhost"
$env:VM_PORT="8428"

# Linux/Mac
export DB_TYPE=victoriametrics
export VM_HOST=localhost
export VM_PORT=8428

2. Start VictoriaMetrics with Docker

# Simple setup
docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428

# Verify it's running
curl http://localhost:8428/health

3. Run the Water Monitor

python water_scraper_v3.py

4. Access Grafana Dashboard

# Start with Docker Compose (includes Grafana)
docker-compose -f docker-compose.victoriametrics.yml up -d

# Access Grafana at http://localhost:3000
# Username: admin
# Password: admin_password

Production Setup

Docker Compose Configuration

Use the provided docker-compose.victoriametrics.yml file:

# Start the complete stack
docker-compose -f docker-compose.victoriametrics.yml up -d

# Check status
docker-compose -f docker-compose.victoriametrics.yml ps

# View logs
docker-compose -f docker-compose.victoriametrics.yml logs -f

Manual VictoriaMetrics Configuration

High-Performance Configuration

docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428 \
  --maxConcurrentInserts=32 \
  --search.maxQueryDuration=60s \
  --search.maxConcurrentRequests=16 \
  --dedup.minScrapeInterval=30s \
  --memory.allowedPercent=80 \
  --loggerLevel=INFO \
  --loggerFormat=json \
  --search.maxSeries=1000000 \
  --search.maxPointsPerTimeseries=100000

Configuration Parameters Explained

Parameter	Description	Recommended Value
--storageDataPath	Data storage directory	/victoria-metrics-data
--retentionPeriod	How long to keep data	2y (2 years)
--httpListenAddr	HTTP listen address	:8428
--maxConcurrentInserts	Max concurrent inserts	32
--search.maxQueryDuration	Max query duration	60s
--search.maxConcurrentRequests	Max concurrent queries	16
--dedup.minScrapeInterval	Deduplication interval	30s
--memory.allowedPercent	Max memory usage	80
--loggerLevel	Log level	INFO
--search.maxSeries	Max time series	1000000

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: victoriametrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: victoriametrics
  template:
    metadata:
      labels:
        app: victoriametrics
    spec:
      containers:
      - name: victoriametrics
        image: victoriametrics/victoria-metrics:latest
        ports:
        - containerPort: 8428
        args:
        - --storageDataPath=/victoria-metrics-data
        - --retentionPeriod=2y
        - --httpListenAddr=:8428
        - --maxConcurrentInserts=32
        volumeMounts:
        - name: storage
          mountPath: /victoria-metrics-data
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: victoriametrics-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: victoriametrics
spec:
  selector:
    app: victoriametrics
  ports:
  - port: 8428
    targetPort: 8428
  type: ClusterIP

Data Queries

HTTP API Queries

VictoriaMetrics provides a Prometheus-compatible HTTP API:

# Current water levels for all stations
curl "http://localhost:8428/api/v1/query?query=water_level"

# Water levels for specific station
curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"

# Average discharge over last hour
curl "http://localhost:8428/api/v1/query?query=avg_over_time(water_discharge[1h])"

# High discharge alerts (>80%)
curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"

# Time range query (last 6 hours)
START=$(date -d '6 hours ago' +%s)
END=$(date +%s)
curl "http://localhost:8428/api/v1/query_range?query=water_level&start=${START}&end=${END}&step=300"

PromQL Examples

# Current water levels
water_level

# Water level trends (last 24h)
water_level[24h]

# Discharge rates by station
water_discharge{station_code="P.1"}

# Average discharge across all stations
avg(water_discharge)

# Stations with high discharge (>80%)
water_discharge_percent > 80

# Rate of change in water level
rate(water_level[5m])

# Maximum water level in last hour
max_over_time(water_level[1h])

# Stations with increasing water levels
increase(water_level[1h]) > 0

Grafana Integration

Data Source Configuration

1. Add VictoriaMetrics as Prometheus Data Source:

   • URL: http://localhost:8428 (or http://victoriametrics:8428 in Docker)
   • Access: Server (default)
   • HTTP Method: POST

2. Import Dashboard:

   • Use the provided water-monitoring-dashboard.json
   • Or create custom dashboards with the queries above

Dashboard Panels

The included dashboard provides:

• Time Series: Water levels and discharge over time
• Table: Current status of all stations
• Pie Chart: Discharge percentage distribution
• Gauge: Average discharge percentage
• Variables: Filter by station

Monitoring and Maintenance

Health Checks

# Check VictoriaMetrics health
curl http://localhost:8428/health

# Check metrics endpoint
curl http://localhost:8428/metrics

# Check configuration
curl http://localhost:8428/api/v1/status/config

Performance Monitoring

# Query performance stats
curl http://localhost:8428/api/v1/status/tsdb

# Memory usage
curl http://localhost:8428/api/v1/status/runtime

# Active queries
curl http://localhost:8428/api/v1/status/active_queries

Backup and Restore

# Create backup
docker exec victoriametrics /usr/bin/vmbackup \
  -storageDataPath=/victoria-metrics-data \
  -dst=fs:///backup/$(date +%Y%m%d)

# Restore from backup
docker exec victoriametrics /usr/bin/vmrestore \
  -src=fs:///backup/20250724 \
  -storageDataPath=/victoria-metrics-data

Log Analysis

# View logs
docker logs victoriametrics

# Follow logs
docker logs -f victoriametrics

# Search for errors
docker logs victoriametrics 2>&1 | grep ERROR

Troubleshooting

Common Issues

1. Connection Refused:

   # Check if VictoriaMetrics is running
   docker ps | grep victoriametrics
   
   # Check port binding
   netstat -tlnp | grep 8428
   

2. High Memory Usage:

   # Reduce memory limit
   docker run ... --memory.allowedPercent=60 ...
   

3. Slow Queries:

   # Increase query timeout
   docker run ... --search.maxQueryDuration=120s ...
   

4. Data Not Appearing:

   # Check if data is being written
   curl "http://localhost:8428/api/v1/query?query=up"
   
   # Check water monitor logs
   tail -f water_monitor.log
   

Performance Tuning

1. For High Write Load:

   --maxConcurrentInserts=64
   --insert.maxQueueDuration=60s
   

2. For High Query Load:

   --search.maxConcurrentRequests=32
   --search.maxQueryDuration=120s
   

3. For Large Datasets:

   --search.maxSeries=10000000
   --search.maxPointsPerTimeseries=1000000
   

Security

Authentication

VictoriaMetrics doesn't have built-in authentication. Use a reverse proxy:

server {
    listen 80;
    server_name victoriametrics.example.com;
    
    auth_basic "VictoriaMetrics";
    auth_basic_user_file /etc/nginx/.htpasswd;
    
    location / {
        proxy_pass http://localhost:8428;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

TLS/SSL

# Use nginx or traefik for TLS termination
# Or use VictoriaMetrics with TLS:
docker run ... \
  -v /path/to/cert.pem:/cert.pem \
  -v /path/to/key.pem:/key.pem \
  victoriametrics/victoria-metrics:latest \
  --tls \
  --tlsCertFile=/cert.pem \
  --tlsKeyFile=/key.pem

Scaling

Cluster Setup

For high availability and horizontal scaling:

# Start multiple VictoriaMetrics instances
docker run -d --name vm1 -p 8428:8428 victoriametrics/victoria-metrics:latest
docker run -d --name vm2 -p 8429:8428 victoriametrics/victoria-metrics:latest

# Use load balancer to distribute queries
# Use vminsert/vmselect/vmstorage for true clustering

Resource Requirements

Data Points/Hour	RAM	CPU	Storage/Day
1,000	100MB	0.1 CPU	10MB
10,000	500MB	0.5 CPU	100MB
100,000	2GB	1 CPU	1GB
1,000,000	8GB	2 CPU	10GB

Migration

From InfluxDB

# Export from InfluxDB
influx -database water_monitoring -execute "SELECT * FROM water_data" -format csv > data.csv

# Import to VictoriaMetrics (convert to Prometheus format first)
# Use vmctl tool for migration

From Prometheus

# Use vmctl for direct migration
vmctl prometheus --prom-snapshot=/path/to/prometheus/data --vm-addr=http://localhost:8428

This comprehensive setup guide should help you configure VictoriaMetrics for optimal performance with the Thailand Water Monitor system.