Northern Thailand Ping River Monitor 🏔️

A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.

🌟 Features

📊 Real-time Data Collection

• 16 Monitoring Stations across Thailand
• 15-minute Collection Frequency with intelligent scheduling
• Automatic Gap Filling for missing historical data
• Data Validation and error recovery mechanisms
• Rate Limiting to prevent API abuse

🌐 Web API Interface (NEW!)

• FastAPI-powered REST API with interactive documentation
• Station Management - Add, update, and remove monitoring stations
• Real-time health monitoring and system status
• Manual data collection triggers via web interface
• Comprehensive metrics and performance monitoring
• CORS support for web applications

🗄️ Multi-Database Support

• VictoriaMetrics (Recommended) - High-performance time-series
• InfluxDB - Purpose-built time-series database
• PostgreSQL + TimescaleDB - Relational with time-series optimization
• MySQL - Traditional relational database
• SQLite - Local development and testing

🗺️ Geolocation Support

• Grafana Geomap integration ready
• GPS coordinates and geohash support
• Interactive mapping of water stations

📈 Visualization & Monitoring

• Pre-built Grafana dashboards
• Real-time alerts and notifications
• Historical trend analysis
• Built-in metrics collection (counters, gauges, histograms)
• Health checks for database, API, and system resources

🚀 Production Ready

• Docker containerization with multi-service support
• Systemd service configuration
• HTTPS support with SSL certificates
• Comprehensive logging with rotation and colored output
• Type safety with Pydantic models and type hints
• Custom exception handling for better error management

🚀 Quick Start

Prerequisites

• Python 3.9 or higher
• Internet connection for data fetching
• Database server (optional - SQLite works out of the box)

Installation

# Clone the repository
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
cd Northern-Thailand-Ping-River-Monitor

# Quick setup with Make
make dev-setup

# Or manual setup:
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env

Basic Usage

# Test run with SQLite (default)
make run-test
# or: python run.py --test

# Run continuous monitoring
make run
# or: python run.py

# Start web API server (NEW!)
make run-api
# or: python run.py --web-api

# Run all tests
make test

# Demo different databases
python src/demo_databases.py

🌐 Web API Interface (NEW!)

The system now includes a comprehensive FastAPI web interface:

# Start the web API
python run.py --web-api

# Access the API at:
# - Dashboard: http://localhost:8000
# - Interactive docs: http://localhost:8000/docs
# - Health check: http://localhost:8000/health
# - Latest data: http://localhost:8000/measurements/latest

Key API Endpoints:

• GET / - Web dashboard
• GET /health - System health status
• GET /metrics - Application metrics
• GET /stations - List all monitoring stations
• POST /stations - Add new monitoring station
• PUT /stations/{id} - Update station information
• DELETE /stations/{id} - Remove monitoring station
• GET /measurements/latest - Latest measurements
• GET /measurements/station/{code} - Station-specific data
• POST /scrape/trigger - Trigger manual data collection

📊 Station Information

The system monitors 16 water stations along the Ping River Basin in Northern Thailand:

Station	Thai Name	English Name	Location
P.1	สะพานนวรัฐ	Nawarat Bridge	Nakhon Sawan
P.5	สะพานท่านาง	Tha Nang Bridge	-
P.20	บ้านเชียงดาว	Ban Chiang Dao	Chiang Mai
P.21	บ้านริมใต้	Ban Rim Tai	-
P.4A	บ้านแม่แตง	Ban Mae Taeng	Chiang Mai
P.67	บ้านแม่แต	Ban Tae	-
P.75	บ้านช่อแล	Ban Chai Lat	-
P.76	บ้านแม่อีไฮ	Banb Mae I Hai	-
P.77	บ้านสบแม่สะป๊วด	Baan Sop Mae Sapuord	-
P.81	บ้านโป่ง	Ban Pong	-
P.82	บ้านสบวิน	Ban Sob win	-
P.84	บ้านพันตน	Ban Panton	-
P.85	บ้านหล่ายแก้ว	Baan Lai Kaew	-
P.87	บ้านป่าซาง	Ban Pa Sang	-
P.92	บ้านเมืองกึ๊ด	Ban Muang Aut	-
P.103	สะพานวงแหวนรอบ 3	Ring Bridge 3	Bangkok

Data Metrics

• Water Level: Measured in meters (m)
• Discharge: Flow rate in cubic meters per second (cms)
• Discharge Percentage: Relative to station capacity
• Timestamp: Thai time (UTC+7) with Buddhist calendar support

🗄️ Database Configuration

VictoriaMetrics (Recommended)

High-performance time-series database with excellent compression and query speed.

# Environment variables
export DB_TYPE=victoriametrics
export VM_HOST=localhost
export VM_PORT=8428

# Quick start with Docker
docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v victoria-metrics-data:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  --storageDataPath=/victoria-metrics-data \
  --retentionPeriod=2y \
  --httpListenAddr=:8428

Complete Stack with Grafana

# Start the complete monitoring stack
docker-compose -f docker-compose.victoriametrics.yml up -d

# Access Grafana at http://localhost:3000
# Username: admin, Password: admin_password

Other Database Options

InfluxDB Configuration

export DB_TYPE=influxdb
export INFLUX_HOST=localhost
export INFLUX_PORT=8086
export INFLUX_DATABASE=water_monitoring
export INFLUX_USERNAME=water_user
export INFLUX_PASSWORD=your_password

PostgreSQL Configuration

export DB_TYPE=postgresql
export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring

MySQL Configuration

export DB_TYPE=mysql
export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring

📈 Grafana Dashboards

Pre-built Dashboard Features

• Real-time water levels across all stations
• Historical trends and patterns
• Discharge monitoring with percentage indicators
• Station status and health monitoring
• Geomap visualization of station locations
• Alert thresholds for critical water levels

Sample Queries

VictoriaMetrics/Prometheus:

# Current water levels
water_level

# High discharge alerts
water_discharge_percent > 80

# Station-specific data
water_level{station_code="P.1"}

SQL Databases:

-- Latest readings from all stations
SELECT s.station_code, s.english_name, m.water_level, m.discharge
FROM stations s 
JOIN water_measurements m ON s.id = m.station_id
WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);

🚀 Production Deployment

Docker Deployment

# Build the image
docker build -t thailand-water-monitor .

# Run with environment variables
docker run -d \
  --name water-monitor \
  -e DB_TYPE=victoriametrics \
  -e VM_HOST=victoriametrics \
  thailand-water-monitor

Systemd Service (Linux)

# Copy service file
sudo cp scripts/water-monitor.service /etc/systemd/system/

# Enable and start
sudo systemctl enable water-monitor.service
sudo systemctl start water-monitor.service

Migration for Existing Systems

If you have an existing installation, use the migration script to add geolocation support:

# Stop the service
sudo systemctl stop water-monitor

# Run migration
python scripts/migrate_geolocation.py

# Restart the service
sudo systemctl start water-monitor

🔧 Command Line Tools

Main Application

python src/water_scraper_v3.py                    # Run continuous monitoring
python src/water_scraper_v3.py --test             # Single test cycle
python src/water_scraper_v3.py --help             # Show help

Data Management

python src/water_scraper_v3.py --check-gaps 7     # Check for missing data (7 days)
python src/water_scraper_v3.py --fill-gaps 7      # Fill missing data gaps
python src/water_scraper_v3.py --update-data 2    # Update existing data (2 days)

Database Testing

python src/demo_databases.py                      # SQLite demo
python src/demo_databases.py victoriametrics      # VictoriaMetrics demo
python src/demo_databases.py all                  # Test all databases

📚 Documentation

Core Documentation

• Installation Guide - Complete setup instructions
• Scheduler Guide - 15-minute scheduling system
• Geolocation Guide - Grafana geomap integration
• Gap Filling Guide - Data integrity management

Deployment Guides

• VictoriaMetrics Setup - High-performance deployment
• HTTPS Configuration - Secure deployment
• Debian Troubleshooting - Linux deployment issues

References

• Notable Documents - Official Thai government resources
• Migration Guide - Updating existing systems

🔍 Troubleshooting

Common Issues

Database Connection Errors:

# Check database status
python src/demo_databases.py

# Test specific database
python src/demo_databases.py victoriametrics

Missing Data:

# Check for gaps
python src/water_scraper_v3.py --check-gaps 7

# Fill missing data
python src/water_scraper_v3.py --fill-gaps 7

Service Issues:

# Check service status
sudo systemctl status water-monitor

# View logs
sudo journalctl -u water-monitor -f

Health Checks

# VictoriaMetrics health
curl http://localhost:8428/health

# Check latest data
curl "http://localhost:8428/api/v1/query?query=water_level"

# Application logs
tail -f water_monitor.log

🌐 API Integration

VictoriaMetrics API Examples

# Query current water levels
curl "http://localhost:8428/api/v1/query?query=water_level"

# Query discharge rates for last hour
curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"

# Query specific station
curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"

# High discharge alerts
curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"

📊 Performance

System Requirements

• CPU: 1-2 cores (minimal load)
• RAM: 512MB - 2GB (depending on database)
• Storage: 1GB+ (for historical data)
• Network: Stable internet connection

Performance Metrics

• Data Collection: ~300 data points every 15 minutes
• Database Write Speed: 1000+ points/second (VictoriaMetrics)
• Query Response: <100ms for recent data
• Storage Efficiency: 70x compression vs. raw data

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

Development Setup

# Clone your fork
git clone https://github.com/your-username/thailand-water-monitor.git
cd thailand-water-monitor

# Install development dependencies
pip install -r requirements.txt
pip install pytest black flake8

# Run tests
pytest

# Format code
black src/

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Royal Irrigation Department (RID) of Thailand for providing the data API
• VictoriaMetrics team for the excellent time-series database
• Grafana team for the visualization platform
• Python community for the amazing libraries and tools

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Project Wiki

---

📁 Project Structure

Northern-Thailand-Ping-River-Monitor/
├── src/                    # Main application code
├── tests/                  # Test suite
├── docs/                   # Documentation
├── grafana/               # Grafana dashboards
├── scripts/               # Utility scripts
├── docker-compose.yml     # Docker deployment
├── Makefile              # Development tasks
└── requirements.txt      # Dependencies

See docs/PROJECT_STRUCTURE.md for detailed architecture information.

🔄 CI/CD & Automation

The project includes comprehensive Gitea Actions workflows:

• 🧪 CI/CD Pipeline - Automated testing, building, and deployment
• 🔒 Security Scanning - Daily vulnerability and dependency checks
• 📚 Documentation - Automated API docs and validation
• 🚀 Release Management - Automated releases with multi-arch Docker builds

See docs/GITEA_WORKFLOWS.md for detailed workflow documentation.

🔗 Repository

• Main Repository: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
• Issues: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
• Actions: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
• Documentation: docs/

Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin