Northern-Thailand-Ping-River-Monitor/docs/DEBIAN_TROUBLESHOOTING.md

Debian/Linux Troubleshooting Guide

This guide addresses common issues when running the Thailand Water Monitor on Debian and other Linux distributions.

Fixed Issues

SQLAlchemy Connection Error (RESOLVED)

Error Message:

2025-07-24 19:48:31,920 - ERROR - Failed to connect to SQLITE: 'Connection' object has no attribute 'commit'
2025-07-24 19:48:32,740 - ERROR - Error saving to SQLITE: 'Connection' object has no attribute 'commit'

Root Cause: This error occurred due to incompatibility between the database adapter code and newer versions of SQLAlchemy. The code was calling conn.commit() on a connection object that doesn't have a commit() method in newer SQLAlchemy versions.

Solution Applied: Changed from engine.connect() to engine.begin() context manager, which automatically handles transactions:

# OLD (problematic) code:
with self.engine.connect() as conn:
    conn.execute(text(sql))
    conn.commit()  # This fails in newer SQLAlchemy

# NEW (fixed) code:
with self.engine.begin() as conn:
    conn.execute(text(sql))
    # Transaction automatically committed when context exits

Status: ✅ FIXED - The issue has been resolved in the current version.

Installation on Debian/Ubuntu

System Requirements

# Update package list
sudo apt update

# Install Python and pip
sudo apt install python3 python3-pip python3-venv

# Install system dependencies for database drivers
sudo apt install build-essential python3-dev

# For MySQL support (optional)
sudo apt install default-libmysqlclient-dev

# For PostgreSQL support (optional)
sudo apt install libpq-dev

Python Environment Setup

# Create virtual environment
python3 -m venv water_monitor_env

# Activate virtual environment
source water_monitor_env/bin/activate

# Install requirements
pip install -r requirements.txt

Running the Monitor

# Test run
python water_scraper_v3.py --test

# Run with specific database
export DB_TYPE=sqlite
python water_scraper_v3.py

# Run demo
python demo_databases.py

Common Linux Issues

1. Permission Errors

Error:

PermissionError: [Errno 13] Permission denied: 'water_levels.db'

Solution:

# Check current directory permissions
ls -la

# Create data directory with proper permissions
mkdir -p data
chmod 755 data

# Set database path to data directory
export WATER_DB_PATH=data/water_levels.db

2. Missing System Dependencies

Error:

ImportError: No module named '_sqlite3'

Solution:

# Install SQLite development headers
sudo apt install libsqlite3-dev

# Reinstall Python if needed
sudo apt install python3-sqlite

3. Network/Firewall Issues

Error:

requests.exceptions.ConnectionError: HTTPSConnectionPool

Solution:

# Test network connectivity
curl -I https://hyd-app-db.rid.go.th/hydro1h.html

# Check firewall rules
sudo ufw status

# Allow outbound HTTPS if needed
sudo ufw allow out 443

4. Systemd Service Setup

Create service file /etc/systemd/system/water-monitor.service:

[Unit]
Description=Thailand Water Level Monitor
After=network.target

[Service]
Type=simple
User=water-monitor
Group=water-monitor
WorkingDirectory=/opt/water_level_monitor
Environment=PATH=/opt/water_level_monitor/venv/bin
Environment=DB_TYPE=sqlite
Environment=WATER_DB_PATH=/opt/water_level_monitor/data/water_levels.db
ExecStart=/opt/water_level_monitor/venv/bin/python water_scraper_v3.py
Restart=always
RestartSec=60

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/water_level_monitor/data
ReadWritePaths=/opt/water_level_monitor/logs

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable water-monitor.service
sudo systemctl start water-monitor.service
sudo systemctl status water-monitor.service

5. Log Rotation

Create /etc/logrotate.d/water-monitor:

/opt/water_level_monitor/water_monitor.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 water-monitor water-monitor
    postrotate
        systemctl reload water-monitor.service
    endscript
}

Database-Specific Issues

SQLite

Issue: Database locked

# Check for processes using the database
sudo lsof /path/to/water_levels.db

# Kill processes if needed
sudo pkill -f water_scraper_v3.py

VictoriaMetrics with HTTPS

Configuration:

export DB_TYPE=victoriametrics
export VM_HOST=https://your-vm-server.com
export VM_PORT=443

Test connection:

curl -k https://your-vm-server.com/health

Performance Optimization

1. System Tuning

# Increase file descriptor limits
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

# Optimize network settings
echo "net.core.rmem_max = 16777216" >> /etc/sysctl.conf
echo "net.core.wmem_max = 16777216" >> /etc/sysctl.conf
sysctl -p

2. Database Optimization

# For SQLite
export SQLITE_CACHE_SIZE=10000
export SQLITE_SYNCHRONOUS=NORMAL

# Monitor database size
du -h data/water_levels.db

Monitoring and Maintenance

Health Check Script

Create health_check.sh:

#!/bin/bash
LOG_FILE="/opt/water_level_monitor/water_monitor.log"
SERVICE_NAME="water-monitor"

# Check if service is running
if ! systemctl is-active --quiet $SERVICE_NAME; then
    echo "ERROR: $SERVICE_NAME is not running"
    systemctl restart $SERVICE_NAME
    exit 1
fi

# Check recent log entries
RECENT_ERRORS=$(tail -n 100 $LOG_FILE | grep -c "ERROR")
if [ $RECENT_ERRORS -gt 5 ]; then
    echo "WARNING: $RECENT_ERRORS errors found in recent logs"
    exit 1
fi

echo "OK: Service is healthy"
exit 0

Cron Job for Health Checks

# Add to crontab
*/5 * * * * /opt/water_level_monitor/health_check.sh >> /var/log/water-monitor-health.log 2>&1

Getting Help

Debug Information

# System information
uname -a
python3 --version
pip list | grep -E "(sqlalchemy|requests|influxdb)"

# Service logs
journalctl -u water-monitor.service -f

# Application logs
tail -f water_monitor.log

# Database information
sqlite3 water_levels.db ".schema"
sqlite3 water_levels.db "SELECT COUNT(*) FROM water_measurements;"

Common Commands

# Restart service
sudo systemctl restart water-monitor.service

# View logs
sudo journalctl -u water-monitor.service --since "1 hour ago"

# Test configuration
python config.py

# Test database connection
python demo_databases.py

# Manual data fetch
python water_scraper_v3.py --test

This troubleshooting guide should help resolve most common issues encountered when running the Thailand Water Monitor on Debian and other Linux distributions.