Northern-Thailand-Ping-River-Monitor/docs/VICTORIAMETRICS_SETUP.md

# VictoriaMetrics Setup Guide for Thailand Water Monitor

This guide provides comprehensive instructions for setting up VictoriaMetrics as the time-series database backend for the Thailand Water Monitor.

## Why VictoriaMetrics?

VictoriaMetrics is an excellent choice for water monitoring data because:

- **High Performance**: Up to 10x faster than InfluxDB
- **Low Resource Usage**: Uses 10x less RAM than Prometheus
- **Better Compression**: 70x better compression than Prometheus
- **Prometheus Compatible**: Drop-in replacement for Prometheus
- **Easy to Deploy**: Single binary, no dependencies
- **Cost Effective**: Open source with commercial support available

## Quick Start

### 1. Environment Variables

Set these environment variables to configure VictoriaMetrics:

```bash
# Windows (PowerShell)
$env:DB_TYPE="victoriametrics"
$env:VM_HOST="localhost"
$env:VM_PORT="8428"

# Linux/Mac
export DB_TYPE=victoriametrics
export VM_HOST=localhost
export VM_PORT=8428
```

### 2. Start VictoriaMetrics with Docker

```bash
# Simple setup
docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428

# Verify it's running
curl http://localhost:8428/health
```

### 3. Run the Water Monitor

```bash
python water_scraper_v3.py
```

### 4. Access Grafana Dashboard

```bash
# Start with Docker Compose (includes Grafana)
docker-compose -f docker-compose.victoriametrics.yml up -d

# Access Grafana at http://localhost:3000
# Username: admin
# Password: admin_password
```

## Production Setup

### Docker Compose Configuration

Use the provided `docker-compose.victoriametrics.yml` file:

```bash
# Start the complete stack
docker-compose -f docker-compose.victoriametrics.yml up -d

# Check status
docker-compose -f docker-compose.victoriametrics.yml ps

# View logs
docker-compose -f docker-compose.victoriametrics.yml logs -f
```

### Manual VictoriaMetrics Configuration

#### High-Performance Configuration

```bash
docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428 \
  --maxConcurrentInserts=32 \
  --search.maxQueryDuration=60s \
  --search.maxConcurrentRequests=16 \
  --dedup.minScrapeInterval=30s \
  --memory.allowedPercent=80 \
  --loggerLevel=INFO \
  --loggerFormat=json \
  --search.maxSeries=1000000 \
  --search.maxPointsPerTimeseries=100000
```

#### Configuration Parameters Explained

| Parameter | Description | Recommended Value |
|-----------|-------------|-------------------|
| `--storageDataPath` | Data storage directory | `/victoria-metrics-data` |
| `--retentionPeriod` | How long to keep data | `2y` (2 years) |
| `--httpListenAddr` | HTTP listen address | `:8428` |
| `--maxConcurrentInserts` | Max concurrent inserts | `32` |
| `--search.maxQueryDuration` | Max query duration | `60s` |
| `--search.maxConcurrentRequests` | Max concurrent queries | `16` |
| `--dedup.minScrapeInterval` | Deduplication interval | `30s` |
| `--memory.allowedPercent` | Max memory usage | `80` |
| `--loggerLevel` | Log level | `INFO` |
| `--search.maxSeries` | Max time series | `1000000` |

### Kubernetes Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: victoriametrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: victoriametrics
  template:
    metadata:
      labels:
        app: victoriametrics
    spec:
      containers:
      - name: victoriametrics
        image: victoriametrics/victoria-metrics:latest
        ports:
        - containerPort: 8428
        args:
        - --storageDataPath=/victoria-metrics-data
        - --retentionPeriod=2y
        - --httpListenAddr=:8428
        - --maxConcurrentInserts=32
        volumeMounts:
        - name: storage
          mountPath: /victoria-metrics-data
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: victoriametrics-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: victoriametrics
spec:
  selector:
    app: victoriametrics
  ports:
  - port: 8428
    targetPort: 8428
  type: ClusterIP
```

## Data Queries

### HTTP API Queries

VictoriaMetrics provides a Prometheus-compatible HTTP API:

```bash
# Current water levels for all stations
curl "http://localhost:8428/api/v1/query?query=water_level"

# Water levels for specific station
curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"

# Average discharge over last hour
curl "http://localhost:8428/api/v1/query?query=avg_over_time(water_discharge[1h])"

# High discharge alerts (>80%)
curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"

# Time range query (last 6 hours)
START=$(date -d '6 hours ago' +%s)
END=$(date +%s)
curl "http://localhost:8428/api/v1/query_range?query=water_level&start=${START}&end=${END}&step=300"
```

### PromQL Examples

```promql
# Current water levels
water_level

# Water level trends (last 24h)
water_level[24h]

# Discharge rates by station
water_discharge{station_code="P.1"}

# Average discharge across all stations
avg(water_discharge)

# Stations with high discharge (>80%)
water_discharge_percent > 80

# Rate of change in water level
rate(water_level[5m])

# Maximum water level in last hour
max_over_time(water_level[1h])

# Stations with increasing water levels
increase(water_level[1h]) > 0
```

## Grafana Integration

### Data Source Configuration

1. **Add VictoriaMetrics as Prometheus Data Source**:
   - URL: `http://localhost:8428` (or `http://victoriametrics:8428` in Docker)
   - Access: Server (default)
   - HTTP Method: POST

2. **Import Dashboard**:
   - Use the provided `water-monitoring-dashboard.json`
   - Or create custom dashboards with the queries above

### Dashboard Panels

The included dashboard provides:

- **Time Series**: Water levels and discharge over time
- **Table**: Current status of all stations
- **Pie Chart**: Discharge percentage distribution
- **Gauge**: Average discharge percentage
- **Variables**: Filter by station

## Monitoring and Maintenance

### Health Checks

```bash
# Check VictoriaMetrics health
curl http://localhost:8428/health

# Check metrics endpoint
curl http://localhost:8428/metrics

# Check configuration
curl http://localhost:8428/api/v1/status/config
```

### Performance Monitoring

```bash
# Query performance stats
curl http://localhost:8428/api/v1/status/tsdb

# Memory usage
curl http://localhost:8428/api/v1/status/runtime

# Active queries
curl http://localhost:8428/api/v1/status/active_queries
```

### Backup and Restore

```bash
# Create backup
docker exec victoriametrics /usr/bin/vmbackup \
  -storageDataPath=/victoria-metrics-data \
  -dst=fs:///backup/$(date +%Y%m%d)

# Restore from backup
docker exec victoriametrics /usr/bin/vmrestore \
  -src=fs:///backup/20250724 \
  -storageDataPath=/victoria-metrics-data
```

### Log Analysis

```bash
# View logs
docker logs victoriametrics

# Follow logs
docker logs -f victoriametrics

# Search for errors
docker logs victoriametrics 2>&1 | grep ERROR
```

## Troubleshooting

### Common Issues

1. **Connection Refused**:
   ```bash
   # Check if VictoriaMetrics is running
   docker ps | grep victoriametrics
   
   # Check port binding
   netstat -tlnp | grep 8428
   ```

2. **High Memory Usage**:
   ```bash
   # Reduce memory limit
   docker run ... --memory.allowedPercent=60 ...
   ```

3. **Slow Queries**:
   ```bash
   # Increase query timeout
   docker run ... --search.maxQueryDuration=120s ...
   ```

4. **Data Not Appearing**:
   ```bash
   # Check if data is being written
   curl "http://localhost:8428/api/v1/query?query=up"
   
   # Check water monitor logs
   tail -f water_monitor.log
   ```

### Performance Tuning

1. **For High Write Load**:
   ```bash
   --maxConcurrentInserts=64
   --insert.maxQueueDuration=60s
   ```

2. **For High Query Load**:
   ```bash
   --search.maxConcurrentRequests=32
   --search.maxQueryDuration=120s
   ```

3. **For Large Datasets**:
   ```bash
   --search.maxSeries=10000000
   --search.maxPointsPerTimeseries=1000000
   ```

## Security

### Authentication

VictoriaMetrics doesn't have built-in authentication. Use a reverse proxy:

```nginx
server {
    listen 80;
    server_name victoriametrics.example.com;
    
    auth_basic "VictoriaMetrics";
    auth_basic_user_file /etc/nginx/.htpasswd;
    
    location / {
        proxy_pass http://localhost:8428;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```

### TLS/SSL

```bash
# Use nginx or traefik for TLS termination
# Or use VictoriaMetrics with TLS:
docker run ... \
  -v /path/to/cert.pem:/cert.pem \
  -v /path/to/key.pem:/key.pem \
  victoriametrics/victoria-metrics:latest \
  --tls \
  --tlsCertFile=/cert.pem \
  --tlsKeyFile=/key.pem
```

## Scaling

### Cluster Setup

For high availability and horizontal scaling:

```bash
# Start multiple VictoriaMetrics instances
docker run -d --name vm1 -p 8428:8428 victoriametrics/victoria-metrics:latest
docker run -d --name vm2 -p 8429:8428 victoriametrics/victoria-metrics:latest

# Use load balancer to distribute queries
# Use vminsert/vmselect/vmstorage for true clustering
```

### Resource Requirements

| Data Points/Hour | RAM | CPU | Storage/Day |
|------------------|-----|-----|-------------|
| 1,000 | 100MB | 0.1 CPU | 10MB |
| 10,000 | 500MB | 0.5 CPU | 100MB |
| 100,000 | 2GB | 1 CPU | 1GB |
| 1,000,000 | 8GB | 2 CPU | 10GB |

## Migration

### From InfluxDB

```bash
# Export from InfluxDB
influx -database water_monitoring -execute "SELECT * FROM water_data" -format csv > data.csv

# Import to VictoriaMetrics (convert to Prometheus format first)
# Use vmctl tool for migration
```

### From Prometheus

```bash
# Use vmctl for direct migration
vmctl prometheus --prom-snapshot=/path/to/prometheus/data --vm-addr=http://localhost:8428
```

This comprehensive setup guide should help you configure VictoriaMetrics for optimal performance with the Thailand Water Monitor system.