Northern-Thailand-Ping-River-Monitor/docs/GAP_FILLING_GUIDE.md

Gap Filling and Data Integrity Guide

This guide explains the enhanced gap-filling functionality that addresses data gaps and missing timestamps in the Thailand Water Monitor.

✅ Issues Resolved

1. Data Gaps Problem

• Before: Tool only fetched current day data, leaving gaps in historical records
• After: Automatically detects and fills missing timestamps for the last 7 days

2. Missing Midnight Timestamps

• Before: Jump from 23:00 to 01:00 (missing 00:00 midnight data)
• After: Specifically checks for and fills midnight hour gaps

3. Changed Values

• Before: No mechanism to update existing data if values changed on the server
• After: Compares existing data with fresh API data and updates changed values

🔧 New Features

Command Line Interface

# Check for missing data gaps
python water_scraper_v3.py --check-gaps [days]

# Fill missing data gaps
python water_scraper_v3.py --fill-gaps [days]

# Update existing data with latest values
python water_scraper_v3.py --update-data [days]

# Run single test cycle
python water_scraper_v3.py --test

# Show help
python water_scraper_v3.py --help

Automatic Gap Detection

The system now automatically:

• Generates expected hourly timestamps for the specified time range
• Compares with existing database records
• Identifies missing timestamps
• Groups missing data by date for efficient API calls

Intelligent Gap Filling

• Historical Data Fetching: Retrieves data for specific dates to fill gaps
• Selective Insertion: Only inserts data for actually missing timestamps
• API Rate Limiting: Includes delays between API calls to be respectful
• Error Handling: Continues processing even if some dates fail

Data Update Mechanism

• Change Detection: Compares water levels, discharge rates, and percentages
• Precision Checking: Uses appropriate thresholds (0.001m for water level, 0.1 cms for discharge)
• Selective Updates: Only updates records where values have actually changed

📊 Test Results

Before Enhancement

Found 22 missing timestamps in the last 2 days:
  2025-07-23: Missing hours [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
  2025-07-24: Missing hours [0, 20, 21, 22, 23]
  2025-07-25: Missing hours [0, 9]

After Gap Filling

Gap filling completed. Filled 96 missing data points

Remaining gaps:
  2025-07-24: Missing hours [10]
  2025-07-25: Missing hours [0, 10]

Improvement: Reduced from 22 missing timestamps to 3 (86% improvement)

🚀 Enhanced Scraping Cycle

The regular scraping cycle now includes three phases:

Phase 1: Current Data Collection

# Fetch and save current data
water_data = self.fetch_water_data()
success = self.save_to_database(water_data)

Phase 2: Gap Filling (Last 7 Days)

# Check for and fill missing data
filled_count = self.fill_data_gaps(days_back=7)

Phase 3: Data Updates (Last 2 Days)

# Update existing data with latest values
updated_count = self.update_existing_data(days_back=2)

🔧 Technical Improvements

Database Connection Handling

• SQLite Optimization: Added timeout and thread safety parameters
• Retry Logic: Exponential backoff for database lock errors
• Transaction Management: Proper use of engine.begin() for automatic commits

Error Recovery

# Retry logic with exponential backoff
for attempt in range(max_retries):
    try:
        success = self.db_adapter.save_measurements(water_data)
        if success:
            return True
    except Exception as e:
        if "database is locked" in str(e).lower():
            time.sleep(2 ** attempt)  # 1s, 2s, 4s delays
            continue

Memory Efficiency

• Selective Data Processing: Only processes data for missing timestamps
• Batch Processing: Groups operations by date to minimize API calls
• Resource Management: Proper cleanup and connection handling

📋 Usage Examples

Daily Maintenance

# Check for gaps in the last week
python water_scraper_v3.py --check-gaps 7

# Fill any found gaps
python water_scraper_v3.py --fill-gaps 7

# Update recent data for accuracy
python water_scraper_v3.py --update-data 2

Historical Data Recovery

# Check for gaps in the last month
python water_scraper_v3.py --check-gaps 30

# Fill gaps for the last month (be patient, this takes time)
python water_scraper_v3.py --fill-gaps 30

Production Monitoring

# Quick test to ensure system is working
python water_scraper_v3.py --test

# Check for recent gaps
python water_scraper_v3.py --check-gaps 1

🔍 Monitoring and Alerts

Gap Detection Output

Found 22 missing timestamps:
  2025-07-23: Missing hours [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
  2025-07-24: Missing hours [0, 20, 21, 22, 23]
  2025-07-25: Missing hours [0, 9]

Gap Filling Progress

Fetching data for 2025-07-24 to fill 5 missing timestamps
Successfully fetched 368 data points from API for 2025-07-24
Filled 80 data points for 2025-07-24
Gap filling completed. Filled 96 missing data points

Update Detection

Checking for updates on 2025-07-24
Update needed for P.1 at 2025-07-24 15:00:00
Updated 5 measurements for 2025-07-24
Data update completed. Updated 5 measurements

⚙️ Configuration Options

Environment Variables

# Database configuration
export DB_TYPE=sqlite
export WATER_DB_PATH=water_monitoring.db

# Gap filling settings (can be added to config.py)
export GAP_FILL_DAYS=7        # Days to check for gaps
export UPDATE_DAYS=2          # Days to check for updates
export API_DELAY=1            # Seconds between API calls
export MAX_RETRIES=3          # Database retry attempts

Customizable Parameters

• Gap Check Period: Default 7 days, configurable via command line
• Update Period: Default 2 days, configurable via command line
• API Rate Limiting: 1-second delay between calls (configurable)
• Retry Logic: 3 attempts with exponential backoff (configurable)

🛠️ Troubleshooting

Common Issues

Database Locked Errors

ERROR - Error saving to SQLITE: database is locked

Solution: The retry logic now handles this automatically with exponential backoff.

API Rate Limiting

WARNING - Too many requests to API

Solution: Increase delay between API calls or reduce the number of days processed at once.

Missing Data Still Present

Found X missing timestamps after gap filling

Possible Causes:

• Data not available on the Thai government server for those timestamps
• Network issues during API calls
• API returned empty data for those specific times

Debug Commands

# Enable debug logging
export LOG_LEVEL=DEBUG
python water_scraper_v3.py --check-gaps 1

# Test specific date range
python water_scraper_v3.py --fill-gaps 1

# Check database directly
sqlite3 water_monitoring.db "SELECT COUNT(*) FROM water_measurements;"
sqlite3 water_monitoring.db "SELECT timestamp, COUNT(*) FROM water_measurements GROUP BY timestamp ORDER BY timestamp DESC LIMIT 10;"

📈 Performance Metrics

Gap Filling Efficiency

• API Calls: Grouped by date to minimize requests
• Processing Speed: ~100-400 data points per API call
• Success Rate: 86% gap reduction in test case
• Resource Usage: Minimal memory footprint with selective processing

Database Performance

• SQLite Optimization: Connection pooling and timeout handling
• Transaction Efficiency: Batch inserts with proper transaction management
• Retry Success: Automatic recovery from temporary lock conditions

🎯 Best Practices

Regular Maintenance

1. Daily: Run --check-gaps 1 to monitor recent data quality
2. Weekly: Run --fill-gaps 7 to catch any missed data
3. Monthly: Run --update-data 7 to ensure data accuracy

Production Deployment

1. Automated Scheduling: Use cron or systemd timers for regular gap checks
2. Monitoring: Set up alerts for excessive missing data
3. Backup: Regular database backups before major gap-filling operations

Data Quality Assurance

1. Validation: Check for reasonable value ranges after gap filling
2. Comparison: Compare filled data with nearby timestamps for consistency
3. Documentation: Log all gap-filling activities for audit trails

This enhanced gap-filling system ensures comprehensive and accurate water level monitoring with minimal data loss and automatic recovery capabilities.