Northern-Thailand-Ping-River-Monitor/docs/DEBIAN_TROUBLESHOOTING.md

# Debian/Linux Troubleshooting Guide

This guide addresses common issues when running the Thailand Water Monitor on Debian and other Linux distributions.

## Fixed Issues

### SQLAlchemy Connection Error (RESOLVED)

**Error Message:**
```
2025-07-24 19:48:31,920 - ERROR - Failed to connect to SQLITE: 'Connection' object has no attribute 'commit'
2025-07-24 19:48:32,740 - ERROR - Error saving to SQLITE: 'Connection' object has no attribute 'commit'
```

**Root Cause:**
This error occurred due to incompatibility between the database adapter code and newer versions of SQLAlchemy. The code was calling `conn.commit()` on a connection object that doesn't have a `commit()` method in newer SQLAlchemy versions.

**Solution Applied:**
Changed from `engine.connect()` to `engine.begin()` context manager, which automatically handles transactions:

```python
# OLD (problematic) code:
with self.engine.connect() as conn:
    conn.execute(text(sql))
    conn.commit()  # This fails in newer SQLAlchemy

# NEW (fixed) code:
with self.engine.begin() as conn:
    conn.execute(text(sql))
    # Transaction automatically committed when context exits
```

**Status:** ✅ **FIXED** - The issue has been resolved in the current version.

## Installation on Debian/Ubuntu

### System Requirements

```bash
# Update package list
sudo apt update

# Install Python and pip
sudo apt install python3 python3-pip python3-venv

# Install system dependencies for database drivers
sudo apt install build-essential python3-dev

# For MySQL support (optional)
sudo apt install default-libmysqlclient-dev

# For PostgreSQL support (optional)
sudo apt install libpq-dev
```

### Python Environment Setup

```bash
# Create virtual environment
python3 -m venv water_monitor_env

# Activate virtual environment
source water_monitor_env/bin/activate

# Install requirements
pip install -r requirements.txt
```

### Running the Monitor

```bash
# Test run
python water_scraper_v3.py --test

# Run with specific database
export DB_TYPE=sqlite
python water_scraper_v3.py

# Run demo
python demo_databases.py
```

## Common Linux Issues

### 1. Permission Errors

**Error:**
```
PermissionError: [Errno 13] Permission denied: 'water_levels.db'
```

**Solution:**
```bash
# Check current directory permissions
ls -la

# Create data directory with proper permissions
mkdir -p data
chmod 755 data

# Set database path to data directory
export WATER_DB_PATH=data/water_levels.db
```

### 2. Missing System Dependencies

**Error:**
```
ImportError: No module named '_sqlite3'
```

**Solution:**
```bash
# Install SQLite development headers
sudo apt install libsqlite3-dev

# Reinstall Python if needed
sudo apt install python3-sqlite
```

### 3. Network/Firewall Issues

**Error:**
```
requests.exceptions.ConnectionError: HTTPSConnectionPool
```

**Solution:**
```bash
# Test network connectivity
curl -I https://hyd-app-db.rid.go.th/hydro1h.html

# Check firewall rules
sudo ufw status

# Allow outbound HTTPS if needed
sudo ufw allow out 443
```

### 4. Systemd Service Setup

Create service file `/etc/systemd/system/water-monitor.service`:

```ini
[Unit]
Description=Thailand Water Level Monitor
After=network.target

[Service]
Type=simple
User=water-monitor
Group=water-monitor
WorkingDirectory=/opt/water_level_monitor
Environment=PATH=/opt/water_level_monitor/venv/bin
Environment=DB_TYPE=sqlite
Environment=WATER_DB_PATH=/opt/water_level_monitor/data/water_levels.db
ExecStart=/opt/water_level_monitor/venv/bin/python water_scraper_v3.py
Restart=always
RestartSec=60

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/water_level_monitor/data
ReadWritePaths=/opt/water_level_monitor/logs

[Install]
WantedBy=multi-user.target
```

Enable and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable water-monitor.service
sudo systemctl start water-monitor.service
sudo systemctl status water-monitor.service
```

### 5. Log Rotation

Create `/etc/logrotate.d/water-monitor`:

```
/opt/water_level_monitor/water_monitor.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 water-monitor water-monitor
    postrotate
        systemctl reload water-monitor.service
    endscript
}
```

## Database-Specific Issues

### SQLite

**Issue:** Database locked
```bash
# Check for processes using the database
sudo lsof /path/to/water_levels.db

# Kill processes if needed
sudo pkill -f water_scraper_v3.py
```

### VictoriaMetrics with HTTPS

**Configuration:**
```bash
export DB_TYPE=victoriametrics
export VM_HOST=https://your-vm-server.com
export VM_PORT=443
```

**Test connection:**
```bash
curl -k https://your-vm-server.com/health
```

## Performance Optimization

### 1. System Tuning

```bash
# Increase file descriptor limits
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

# Optimize network settings
echo "net.core.rmem_max = 16777216" >> /etc/sysctl.conf
echo "net.core.wmem_max = 16777216" >> /etc/sysctl.conf
sysctl -p
```

### 2. Database Optimization

```bash
# For SQLite
export SQLITE_CACHE_SIZE=10000
export SQLITE_SYNCHRONOUS=NORMAL

# Monitor database size
du -h data/water_levels.db
```

## Monitoring and Maintenance

### Health Check Script

Create `health_check.sh`:

```bash
#!/bin/bash
LOG_FILE="/opt/water_level_monitor/water_monitor.log"
SERVICE_NAME="water-monitor"

# Check if service is running
if ! systemctl is-active --quiet $SERVICE_NAME; then
    echo "ERROR: $SERVICE_NAME is not running"
    systemctl restart $SERVICE_NAME
    exit 1
fi

# Check recent log entries
RECENT_ERRORS=$(tail -n 100 $LOG_FILE | grep -c "ERROR")
if [ $RECENT_ERRORS -gt 5 ]; then
    echo "WARNING: $RECENT_ERRORS errors found in recent logs"
    exit 1
fi

echo "OK: Service is healthy"
exit 0
```

### Cron Job for Health Checks

```bash
# Add to crontab
*/5 * * * * /opt/water_level_monitor/health_check.sh >> /var/log/water-monitor-health.log 2>&1
```

## Getting Help

### Debug Information

```bash
# System information
uname -a
python3 --version
pip list | grep -E "(sqlalchemy|requests|influxdb)"

# Service logs
journalctl -u water-monitor.service -f

# Application logs
tail -f water_monitor.log

# Database information
sqlite3 water_levels.db ".schema"
sqlite3 water_levels.db "SELECT COUNT(*) FROM water_measurements;"
```

### Common Commands

```bash
# Restart service
sudo systemctl restart water-monitor.service

# View logs
sudo journalctl -u water-monitor.service --since "1 hour ago"

# Test configuration
python config.py

# Test database connection
python demo_databases.py

# Manual data fetch
python water_scraper_v3.py --test
```

This troubleshooting guide should help resolve most common issues encountered when running the Thailand Water Monitor on Debian and other Linux distributions.