Northern-Thailand-Ping-River-Monitor/SQLITE_MIGRATION.md

# SQLite to PostgreSQL Migration Guide

This guide helps you migrate your existing SQLite water monitoring data to PostgreSQL.

## Quick Migration

### 1. Analyze Your SQLite Database (Optional)

First, check what's in your SQLite database:

```bash
# Analyze without migrating
make analyze-sqlite

# Or specify a specific SQLite file
uv run python scripts/migrate_sqlite_to_postgres.py --dry-run /path/to/your/database.db
```

### 2. Run the Migration

```bash
# Auto-detect SQLite file and migrate
make migrate-sqlite

# Or specify a specific SQLite file
uv run python scripts/migrate_sqlite_to_postgres.py /path/to/your/database.db
```

The migration tool will:
- ✅ Connect to both databases
- ✅ Analyze your SQLite schema automatically
- ✅ Migrate station information
- ✅ Migrate all measurement data in batches
- ✅ Handle different SQLite table structures
- ✅ Verify the migration results
- ✅ Generate a detailed log file

## What Gets Migrated

### Station Data
- Station IDs and codes
- Thai and English names
- Coordinates (latitude/longitude)
- Geohash data (if available)
- Creation/update timestamps

### Measurement Data
- Water level readings
- Discharge measurements
- Discharge percentages
- Timestamps
- Station associations
- Data quality status

## Supported SQLite Schemas

The migration tool automatically detects and handles various SQLite table structures:

### Modern Schema
```sql
-- Stations
stations: id, station_code, station_name_th, station_name_en, latitude, longitude, geohash

-- Measurements
water_measurements: timestamp, station_id, water_level, discharge, discharge_percent, status
```

### Legacy Schema
```sql
-- Stations
water_stations: station_id, station_code, station_name, lat, lon

-- Measurements
measurements: timestamp, station_id, water_level, discharge, discharge_percent
```

### Simple Schema
```sql
-- Any table with basic water level data
-- The tool will adapt and map columns automatically
```

## Migration Process

### Step 1: Database Connection
- Connects to your SQLite database
- Verifies PostgreSQL connection
- Validates configuration

### Step 2: Schema Analysis
- Scans SQLite tables and columns
- Reports data counts
- Identifies table structures

### Step 3: Station Migration
- Extracts station metadata
- Maps to PostgreSQL format
- Handles missing data gracefully

### Step 4: Measurement Migration
- Processes data in batches (1000 records at a time)
- Converts timestamps correctly
- Preserves all measurement values
- Shows progress during migration

### Step 5: Verification
- Compares record counts
- Validates data integrity
- Reports migration statistics

## Command Options

```bash
# Basic migration (auto-detects SQLite file)
uv run python scripts/migrate_sqlite_to_postgres.py

# Specify SQLite database path
uv run python scripts/migrate_sqlite_to_postgres.py /path/to/database.db

# Dry run (analyze only, no migration)
uv run python scripts/migrate_sqlite_to_postgres.py --dry-run

# Custom batch size for large databases
uv run python scripts/migrate_sqlite_to_postgres.py --batch-size 5000
```

## Auto-Detection

The tool automatically searches for SQLite files in common locations:
- `water_levels.db`
- `water_monitoring.db`
- `database.db`
- `../water_levels.db`

## Migration Output

The tool provides detailed logging:

```
========================================
SQLite to PostgreSQL Migration Tool
========================================
SQLite database: water_levels.db
PostgreSQL: postgresql

Step 1: Connecting to databases...
Connected to SQLite database: water_levels.db
Connected to PostgreSQL database

Step 2: Analyzing SQLite database structure...
Table 'stations': 8 columns, 25 rows
Table 'water_measurements': 7 columns, 15420 rows

Step 3: Migrating station data...
Migrated 25 stations

Step 4: Migrating measurement data...
Found 15420 measurements to migrate
Migrated 1000/15420 measurements
Migrated 2000/15420 measurements
...
Successfully migrated 15420 measurements

Step 5: Verifying migration...
SQLite stations: 25
SQLite measurements: 15420
PostgreSQL measurements retrieved: 15420
Migrated stations: 25
Migrated measurements: 15420

========================================
MIGRATION COMPLETED
========================================
Duration: 0:02:15
Stations migrated: 25
Measurements migrated: 15420
No errors encountered
```

## Error Handling

The migration tool is robust and handles:
- **Missing tables** - Tries alternative table names
- **Different column names** - Maps common variations
- **Missing data** - Uses sensible defaults
- **Invalid timestamps** - Attempts multiple date formats
- **Connection issues** - Provides clear error messages
- **Large datasets** - Processes in batches to avoid memory issues

## Log Files

Migration creates a detailed log file:
- `migration.log` - Complete migration log
- Shows all operations, errors, and statistics
- Useful for troubleshooting

## Troubleshooting

### Common Issues

**1. SQLite file not found**
```
SQLite database file not found. Please specify the path:
  python migrate_sqlite_to_postgres.py /path/to/database.db
```
**Solution**: Specify the correct path to your SQLite file

**2. PostgreSQL not configured**
```
Error: PostgreSQL not configured. Set DB_TYPE=postgresql in your .env file
```
**Solution**: Ensure your .env file has `DB_TYPE=postgresql`

**3. Connection failed**
```
Database connection error: connection refused
```
**Solution**: Check your PostgreSQL connection settings

**4. No tables found**
```
Could not analyze SQLite database structure
```
**Solution**: Verify your SQLite file contains water monitoring data

### Performance Tips

- **Large databases**: Use `--batch-size 5000` for faster processing
- **Slow networks**: Reduce batch size to `--batch-size 100`
- **Memory issues**: Process smaller batches

## After Migration

Once migration is complete:

1. **Verify data**:
   ```bash
   make run-test
   make run-api
   ```

2. **Check the web interface**: Latest readings should show your migrated data

3. **Backup your SQLite**: Keep the original file as backup

4. **Update configurations**: Remove SQLite references from configs

## Rollback

If you need to rollback:

1. **Clear PostgreSQL data**:
   ```sql
   DELETE FROM water_measurements;
   DELETE FROM stations;
   ```

2. **Switch back to SQLite**:
   ```bash
   # In .env file
   DB_TYPE=sqlite
   WATER_DB_PATH=water_levels.db
   ```

3. **Test the rollback**:
   ```bash
   make run-test
   ```

The migration tool is designed to be safe and can be run multiple times - it handles duplicates appropriately.

## Next Steps

After successful migration:
- Set up automated backups for PostgreSQL
- Configure monitoring and alerting
- Consider data retention policies
- Update documentation references