Northern-Thailand-Ping-River-Monitor/SQLITE_MIGRATION.md

SQLite to PostgreSQL Migration Guide

This guide helps you migrate your existing SQLite water monitoring data to PostgreSQL.

Quick Migration

1. Analyze Your SQLite Database (Optional)

First, check what's in your SQLite database:

# Analyze without migrating
make analyze-sqlite

# Or specify a specific SQLite file
uv run python scripts/migrate_sqlite_to_postgres.py --dry-run /path/to/your/database.db

2. Run the Migration

# Auto-detect SQLite file and migrate
make migrate-sqlite

# Or specify a specific SQLite file
uv run python scripts/migrate_sqlite_to_postgres.py /path/to/your/database.db

The migration tool will:

• ✅ Connect to both databases
• ✅ Analyze your SQLite schema automatically
• ✅ Migrate station information
• ✅ Migrate all measurement data in batches
• ✅ Handle different SQLite table structures
• ✅ Verify the migration results
• ✅ Generate a detailed log file

What Gets Migrated

Station Data

• Station IDs and codes
• Thai and English names
• Coordinates (latitude/longitude)
• Geohash data (if available)
• Creation/update timestamps

Measurement Data

• Water level readings
• Discharge measurements
• Discharge percentages
• Timestamps
• Station associations
• Data quality status

Supported SQLite Schemas

The migration tool automatically detects and handles various SQLite table structures:

Modern Schema

-- Stations
stations: id, station_code, station_name_th, station_name_en, latitude, longitude, geohash

-- Measurements
water_measurements: timestamp, station_id, water_level, discharge, discharge_percent, status

Legacy Schema

-- Stations
water_stations: station_id, station_code, station_name, lat, lon

-- Measurements
measurements: timestamp, station_id, water_level, discharge, discharge_percent

Simple Schema

-- Any table with basic water level data
-- The tool will adapt and map columns automatically

Migration Process

Step 1: Database Connection

• Connects to your SQLite database
• Verifies PostgreSQL connection
• Validates configuration

Step 2: Schema Analysis

• Scans SQLite tables and columns
• Reports data counts
• Identifies table structures

Step 3: Station Migration

• Extracts station metadata
• Maps to PostgreSQL format
• Handles missing data gracefully

Step 4: Measurement Migration

• Processes data in batches (1000 records at a time)
• Converts timestamps correctly
• Preserves all measurement values
• Shows progress during migration

Step 5: Verification

• Compares record counts
• Validates data integrity
• Reports migration statistics

Command Options

# Basic migration (auto-detects SQLite file)
uv run python scripts/migrate_sqlite_to_postgres.py

# Specify SQLite database path
uv run python scripts/migrate_sqlite_to_postgres.py /path/to/database.db

# Dry run (analyze only, no migration)
uv run python scripts/migrate_sqlite_to_postgres.py --dry-run

# Custom batch size for large databases
uv run python scripts/migrate_sqlite_to_postgres.py --batch-size 5000

Auto-Detection

The tool automatically searches for SQLite files in common locations:

• water_levels.db
• water_monitoring.db
• database.db
• ../water_levels.db

Migration Output

The tool provides detailed logging:

========================================
SQLite to PostgreSQL Migration Tool
========================================
SQLite database: water_levels.db
PostgreSQL: postgresql

Step 1: Connecting to databases...
Connected to SQLite database: water_levels.db
Connected to PostgreSQL database

Step 2: Analyzing SQLite database structure...
Table 'stations': 8 columns, 25 rows
Table 'water_measurements': 7 columns, 15420 rows

Step 3: Migrating station data...
Migrated 25 stations

Step 4: Migrating measurement data...
Found 15420 measurements to migrate
Migrated 1000/15420 measurements
Migrated 2000/15420 measurements
...
Successfully migrated 15420 measurements

Step 5: Verifying migration...
SQLite stations: 25
SQLite measurements: 15420
PostgreSQL measurements retrieved: 15420
Migrated stations: 25
Migrated measurements: 15420

========================================
MIGRATION COMPLETED
========================================
Duration: 0:02:15
Stations migrated: 25
Measurements migrated: 15420
No errors encountered

Error Handling

The migration tool is robust and handles:

• Missing tables - Tries alternative table names
• Different column names - Maps common variations
• Missing data - Uses sensible defaults
• Invalid timestamps - Attempts multiple date formats
• Connection issues - Provides clear error messages
• Large datasets - Processes in batches to avoid memory issues

Log Files

Migration creates a detailed log file:

• migration.log - Complete migration log
• Shows all operations, errors, and statistics
• Useful for troubleshooting

Troubleshooting

Common Issues

1. SQLite file not found

SQLite database file not found. Please specify the path:
  python migrate_sqlite_to_postgres.py /path/to/database.db

Solution: Specify the correct path to your SQLite file

2. PostgreSQL not configured

Error: PostgreSQL not configured. Set DB_TYPE=postgresql in your .env file

Solution: Ensure your .env file has DB_TYPE=postgresql

3. Connection failed

Database connection error: connection refused

Solution: Check your PostgreSQL connection settings

4. No tables found

Could not analyze SQLite database structure

Solution: Verify your SQLite file contains water monitoring data

Performance Tips

• Large databases: Use --batch-size 5000 for faster processing
• Slow networks: Reduce batch size to --batch-size 100
• Memory issues: Process smaller batches

After Migration

Once migration is complete:

1. Verify data:

   make run-test
   make run-api
   

2. Check the web interface: Latest readings should show your migrated data

3. Backup your SQLite: Keep the original file as backup

4. Update configurations: Remove SQLite references from configs

Rollback

If you need to rollback:

1. Clear PostgreSQL data:

   DELETE FROM water_measurements;
   DELETE FROM stations;
   

2. Switch back to SQLite:

   # In .env file
   DB_TYPE=sqlite
   WATER_DB_PATH=water_levels.db
   

3. Test the rollback:

   make run-test
   

The migration tool is designed to be safe and can be run multiple times - it handles duplicates appropriately.

Next Steps

After successful migration:

• Set up automated backups for PostgreSQL
• Configure monitoring and alerting
• Consider data retention policies
• Update documentation references