grabowski
17a716fcd0
Some checks failed
Release - Northern Thailand Ping River Monitor / Create Release (push) Successful in 7s
Security & Dependency Updates / Dependency Security Scan (push) Successful in 35s
Security & Dependency Updates / Check for Dependency Updates (push) Has been cancelled
Security & Dependency Updates / Code Quality Metrics (push) Has been cancelled
Security & Dependency Updates / Security Summary (push) Has been cancelled
Security & Dependency Updates / License Compliance (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.11) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.12) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.9) (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Build Release Images (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Security Scan (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Deploy Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Validate Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Notify Release (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Build (3.10) (push) Has been cancelled
Version Updates: - Core application: src/__init__.py, src/main.py, src/web_api.py - Package configuration: setup.py - Documentation: README.md, docs/GITEA_WORKFLOWS.md - Workflows: .gitea/workflows/docs.yml, .gitea/workflows/release.yml - Scripts: generate_badges.py, init_git scripts - Tests: test_integration.py - Deployment docs: GITEA_SETUP_SUMMARY.md, DEPLOYMENT_CHECKLIST.md Purpose: - Force new build process after workflow fixes - Test updated security.yml without YAML errors - Verify setup.py robustness improvements - Trigger clean CI/CD pipeline execution All version references synchronized at v3.1.3 Ready for new build and deployment testing
492 lines
15 KiB
Markdown
492 lines
15 KiB
Markdown
# Northern Thailand Ping River Monitor 🏔️
|
|
|
|
A comprehensive real-time water level monitoring system for the Ping River Basin in Northern Thailand, covering Royal Irrigation Department (RID) stations from Chiang Dao to Nakhon Sawan with advanced data collection, storage, and visualization capabilities.
|
|
|
|
[](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions) [](https://python.org) [](https://fastapi.tiangolo.com) [](https://docker.com) [](LICENSE) [](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/releases)
|
|
|
|
## 🌟 Features
|
|
|
|
### 📊 **Real-time Data Collection**
|
|
- **16 Monitoring Stations** across Thailand
|
|
- **15-minute Collection Frequency** with intelligent scheduling
|
|
- **Automatic Gap Filling** for missing historical data
|
|
- **Data Validation** and error recovery mechanisms
|
|
- **Rate Limiting** to prevent API abuse
|
|
|
|
### 🌐 **Web API Interface (NEW!)**
|
|
- **FastAPI-powered REST API** with interactive documentation
|
|
- **Station Management** - Add, update, and remove monitoring stations
|
|
- **Real-time health monitoring** and system status
|
|
- **Manual data collection triggers** via web interface
|
|
- **Comprehensive metrics** and performance monitoring
|
|
- **CORS support** for web applications
|
|
|
|
### 🗄️ **Multi-Database Support**
|
|
- **VictoriaMetrics** (Recommended) - High-performance time-series
|
|
- **InfluxDB** - Purpose-built time-series database
|
|
- **PostgreSQL + TimescaleDB** - Relational with time-series optimization
|
|
- **MySQL** - Traditional relational database
|
|
- **SQLite** - Local development and testing
|
|
|
|
### 🗺️ **Geolocation Support**
|
|
- **Grafana Geomap** integration ready
|
|
- **GPS coordinates** and geohash support
|
|
- **Interactive mapping** of water stations
|
|
|
|
### 📈 **Visualization & Monitoring**
|
|
- **Pre-built Grafana dashboards**
|
|
- **Real-time alerts** and notifications
|
|
- **Historical trend analysis**
|
|
- **Built-in metrics collection** (counters, gauges, histograms)
|
|
- **Health checks** for database, API, and system resources
|
|
|
|
### 🚀 **Production Ready**
|
|
- **Docker containerization** with multi-service support
|
|
- **Systemd service** configuration
|
|
- **HTTPS support** with SSL certificates
|
|
- **Comprehensive logging** with rotation and colored output
|
|
- **Type safety** with Pydantic models and type hints
|
|
- **Custom exception handling** for better error management
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Prerequisites
|
|
- Python 3.9 or higher
|
|
- Internet connection for data fetching
|
|
- Database server (optional - SQLite works out of the box)
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
# Clone the repository
|
|
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
|
|
cd Northern-Thailand-Ping-River-Monitor
|
|
|
|
# Quick setup with Make
|
|
make dev-setup
|
|
|
|
# Or manual setup:
|
|
python -m venv venv
|
|
source venv/bin/activate # Windows: venv\Scripts\activate
|
|
pip install -r requirements.txt
|
|
cp .env.example .env
|
|
```
|
|
|
|
### Basic Usage
|
|
|
|
```bash
|
|
# Test run with SQLite (default)
|
|
make run-test
|
|
# or: python run.py --test
|
|
|
|
# Run continuous monitoring
|
|
make run
|
|
# or: python run.py
|
|
|
|
# Start web API server (NEW!)
|
|
make run-api
|
|
# or: python run.py --web-api
|
|
|
|
# Run all tests
|
|
make test
|
|
|
|
# Demo different databases
|
|
python src/demo_databases.py
|
|
```
|
|
|
|
### 🌐 Web API Interface (NEW!)
|
|
|
|
The system now includes a comprehensive FastAPI web interface:
|
|
|
|
```bash
|
|
# Start the web API
|
|
python run.py --web-api
|
|
|
|
# Access the API at:
|
|
# - Dashboard: http://localhost:8000
|
|
# - Interactive docs: http://localhost:8000/docs
|
|
# - Health check: http://localhost:8000/health
|
|
# - Latest data: http://localhost:8000/measurements/latest
|
|
```
|
|
|
|
**Key API Endpoints:**
|
|
- `GET /` - Web dashboard
|
|
- `GET /health` - System health status
|
|
- `GET /metrics` - Application metrics
|
|
- `GET /stations` - List all monitoring stations
|
|
- `POST /stations` - Add new monitoring station
|
|
- `PUT /stations/{id}` - Update station information
|
|
- `DELETE /stations/{id}` - Remove monitoring station
|
|
- `GET /measurements/latest` - Latest measurements
|
|
- `GET /measurements/station/{code}` - Station-specific data
|
|
- `POST /scrape/trigger` - Trigger manual data collection
|
|
|
|
## 📊 Station Information
|
|
|
|
The system monitors **16 water stations** along the Ping River Basin in Northern Thailand:
|
|
|
|
| Station | Thai Name | English Name | Location |
|
|
|---------|-----------|--------------|----------|
|
|
| P.1 | สะพานนวรัฐ | Nawarat Bridge | Nakhon Sawan |
|
|
| P.5 | สะพานท่านาง | Tha Nang Bridge | - |
|
|
| P.20 | บ้านเชียงดาว | Ban Chiang Dao | Chiang Mai |
|
|
| P.21 | บ้านริมใต้ | Ban Rim Tai | - |
|
|
| P.4A | บ้านแม่แตง | Ban Mae Taeng | Chiang Mai |
|
|
| P.67 | บ้านแม่แต | Ban Tae | - |
|
|
| P.75 | บ้านช่อแล | Ban Chai Lat | - |
|
|
| P.76 | บ้านแม่อีไฮ | Banb Mae I Hai | - |
|
|
| P.77 | บ้านสบแม่สะป๊วด | Baan Sop Mae Sapuord | - |
|
|
| P.81 | บ้านโป่ง | Ban Pong | - |
|
|
| P.82 | บ้านสบวิน | Ban Sob win | - |
|
|
| P.84 | บ้านพันตน | Ban Panton | - |
|
|
| P.85 | บ้านหล่ายแก้ว | Baan Lai Kaew | - |
|
|
| P.87 | บ้านป่าซาง | Ban Pa Sang | - |
|
|
| P.92 | บ้านเมืองกึ๊ด | Ban Muang Aut | - |
|
|
| P.103 | สะพานวงแหวนรอบ 3 | Ring Bridge 3 | Bangkok |
|
|
|
|
### Data Metrics
|
|
- **Water Level**: Measured in meters (m)
|
|
- **Discharge**: Flow rate in cubic meters per second (cms)
|
|
- **Discharge Percentage**: Relative to station capacity
|
|
- **Timestamp**: Thai time (UTC+7) with Buddhist calendar support
|
|
|
|
## 🗄️ Database Configuration
|
|
|
|
### VictoriaMetrics (Recommended)
|
|
|
|
**High-performance time-series database with excellent compression and query speed.**
|
|
|
|
```bash
|
|
# Environment variables
|
|
export DB_TYPE=victoriametrics
|
|
export VM_HOST=localhost
|
|
export VM_PORT=8428
|
|
|
|
# Quick start with Docker
|
|
docker run -d \
|
|
--name victoriametrics \
|
|
-p 8428:8428 \
|
|
-v victoria-metrics-data:/victoria-metrics-data \
|
|
victoriametrics/victoria-metrics:latest \
|
|
--storageDataPath=/victoria-metrics-data \
|
|
--retentionPeriod=2y \
|
|
--httpListenAddr=:8428
|
|
```
|
|
|
|
### Complete Stack with Grafana
|
|
|
|
```bash
|
|
# Start the complete monitoring stack
|
|
docker-compose -f docker-compose.victoriametrics.yml up -d
|
|
|
|
# Access Grafana at http://localhost:3000
|
|
# Username: admin, Password: admin_password
|
|
```
|
|
|
|
### Other Database Options
|
|
|
|
<details>
|
|
<summary>InfluxDB Configuration</summary>
|
|
|
|
```bash
|
|
export DB_TYPE=influxdb
|
|
export INFLUX_HOST=localhost
|
|
export INFLUX_PORT=8086
|
|
export INFLUX_DATABASE=water_monitoring
|
|
export INFLUX_USERNAME=water_user
|
|
export INFLUX_PASSWORD=your_password
|
|
```
|
|
</details>
|
|
|
|
<details>
|
|
<summary>PostgreSQL Configuration</summary>
|
|
|
|
```bash
|
|
export DB_TYPE=postgresql
|
|
export POSTGRES_CONNECTION_STRING=postgresql://user:password@localhost:5432/water_monitoring
|
|
```
|
|
</details>
|
|
|
|
<details>
|
|
<summary>MySQL Configuration</summary>
|
|
|
|
```bash
|
|
export DB_TYPE=mysql
|
|
export MYSQL_CONNECTION_STRING=mysql://user:password@localhost:3306/water_monitoring
|
|
```
|
|
</details>
|
|
|
|
## 📈 Grafana Dashboards
|
|
|
|
### Pre-built Dashboard Features
|
|
- **Real-time water levels** across all stations
|
|
- **Historical trends** and patterns
|
|
- **Discharge monitoring** with percentage indicators
|
|
- **Station status** and health monitoring
|
|
- **Geomap visualization** of station locations
|
|
- **Alert thresholds** for critical water levels
|
|
|
|
### Sample Queries
|
|
|
|
**VictoriaMetrics/Prometheus:**
|
|
```promql
|
|
# Current water levels
|
|
water_level
|
|
|
|
# High discharge alerts
|
|
water_discharge_percent > 80
|
|
|
|
# Station-specific data
|
|
water_level{station_code="P.1"}
|
|
```
|
|
|
|
**SQL Databases:**
|
|
```sql
|
|
-- Latest readings from all stations
|
|
SELECT s.station_code, s.english_name, m.water_level, m.discharge
|
|
FROM stations s
|
|
JOIN water_measurements m ON s.id = m.station_id
|
|
WHERE m.timestamp = (SELECT MAX(timestamp) FROM water_measurements WHERE station_id = s.id);
|
|
```
|
|
|
|
## 🚀 Production Deployment
|
|
|
|
### Docker Deployment
|
|
|
|
```bash
|
|
# Build the image
|
|
docker build -t thailand-water-monitor .
|
|
|
|
# Run with environment variables
|
|
docker run -d \
|
|
--name water-monitor \
|
|
-e DB_TYPE=victoriametrics \
|
|
-e VM_HOST=victoriametrics \
|
|
thailand-water-monitor
|
|
```
|
|
|
|
### Systemd Service (Linux)
|
|
|
|
```bash
|
|
# Copy service file
|
|
sudo cp scripts/water-monitor.service /etc/systemd/system/
|
|
|
|
# Enable and start
|
|
sudo systemctl enable water-monitor.service
|
|
sudo systemctl start water-monitor.service
|
|
```
|
|
|
|
### Migration for Existing Systems
|
|
|
|
If you have an existing installation, use the migration script to add geolocation support:
|
|
|
|
```bash
|
|
# Stop the service
|
|
sudo systemctl stop water-monitor
|
|
|
|
# Run migration
|
|
python scripts/migrate_geolocation.py
|
|
|
|
# Restart the service
|
|
sudo systemctl start water-monitor
|
|
```
|
|
|
|
## 🔧 Command Line Tools
|
|
|
|
### Main Application
|
|
```bash
|
|
python src/water_scraper_v3.py # Run continuous monitoring
|
|
python src/water_scraper_v3.py --test # Single test cycle
|
|
python src/water_scraper_v3.py --help # Show help
|
|
```
|
|
|
|
### Data Management
|
|
```bash
|
|
python src/water_scraper_v3.py --check-gaps 7 # Check for missing data (7 days)
|
|
python src/water_scraper_v3.py --fill-gaps 7 # Fill missing data gaps
|
|
python src/water_scraper_v3.py --update-data 2 # Update existing data (2 days)
|
|
```
|
|
|
|
### Database Testing
|
|
```bash
|
|
python src/demo_databases.py # SQLite demo
|
|
python src/demo_databases.py victoriametrics # VictoriaMetrics demo
|
|
python src/demo_databases.py all # Test all databases
|
|
```
|
|
|
|
## 📚 Documentation
|
|
|
|
### Core Documentation
|
|
- **[Installation Guide](docs/DATABASE_DEPLOYMENT_GUIDE.md)** - Complete setup instructions
|
|
- **[Scheduler Guide](docs/ENHANCED_SCHEDULER_GUIDE.md)** - 15-minute scheduling system
|
|
- **[Geolocation Guide](docs/GEOLOCATION_GUIDE.md)** - Grafana geomap integration
|
|
- **[Gap Filling Guide](docs/GAP_FILLING_GUIDE.md)** - Data integrity management
|
|
|
|
### Deployment Guides
|
|
- **[VictoriaMetrics Setup](docs/VICTORIAMETRICS_SETUP.md)** - High-performance deployment
|
|
- **[HTTPS Configuration](docs/HTTPS_CONFIGURATION.md)** - Secure deployment
|
|
- **[Debian Troubleshooting](docs/DEBIAN_TROUBLESHOOTING.md)** - Linux deployment issues
|
|
|
|
### References
|
|
- **[Notable Documents](docs/references/NOTABLE_DOCUMENTS.md)** - Official Thai government resources
|
|
- **[Migration Guide](docs/MIGRATION_QUICKSTART.md)** - Updating existing systems
|
|
|
|
## 🔍 Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
**Database Connection Errors:**
|
|
```bash
|
|
# Check database status
|
|
python src/demo_databases.py
|
|
|
|
# Test specific database
|
|
python src/demo_databases.py victoriametrics
|
|
```
|
|
|
|
**Missing Data:**
|
|
```bash
|
|
# Check for gaps
|
|
python src/water_scraper_v3.py --check-gaps 7
|
|
|
|
# Fill missing data
|
|
python src/water_scraper_v3.py --fill-gaps 7
|
|
```
|
|
|
|
**Service Issues:**
|
|
```bash
|
|
# Check service status
|
|
sudo systemctl status water-monitor
|
|
|
|
# View logs
|
|
sudo journalctl -u water-monitor -f
|
|
```
|
|
|
|
### Health Checks
|
|
|
|
```bash
|
|
# VictoriaMetrics health
|
|
curl http://localhost:8428/health
|
|
|
|
# Check latest data
|
|
curl "http://localhost:8428/api/v1/query?query=water_level"
|
|
|
|
# Application logs
|
|
tail -f water_monitor.log
|
|
```
|
|
|
|
## 🌐 API Integration
|
|
|
|
### VictoriaMetrics API Examples
|
|
|
|
```bash
|
|
# Query current water levels
|
|
curl "http://localhost:8428/api/v1/query?query=water_level"
|
|
|
|
# Query discharge rates for last hour
|
|
curl "http://localhost:8428/api/v1/query_range?query=water_discharge&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=300"
|
|
|
|
# Query specific station
|
|
curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
|
|
|
|
# High discharge alerts
|
|
curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
|
|
```
|
|
|
|
## 📊 Performance
|
|
|
|
### System Requirements
|
|
- **CPU**: 1-2 cores (minimal load)
|
|
- **RAM**: 512MB - 2GB (depending on database)
|
|
- **Storage**: 1GB+ (for historical data)
|
|
- **Network**: Stable internet connection
|
|
|
|
### Performance Metrics
|
|
- **Data Collection**: ~300 data points every 15 minutes
|
|
- **Database Write Speed**: 1000+ points/second (VictoriaMetrics)
|
|
- **Query Response**: <100ms for recent data
|
|
- **Storage Efficiency**: 70x compression vs. raw data
|
|
|
|
## 🤝 Contributing
|
|
|
|
Contributions are welcome! Please:
|
|
|
|
1. Fork the repository
|
|
2. Create a feature branch
|
|
3. Make your changes
|
|
4. Add tests if applicable
|
|
5. Submit a pull request
|
|
|
|
### Development Setup
|
|
|
|
```bash
|
|
# Clone your fork
|
|
git clone https://github.com/your-username/thailand-water-monitor.git
|
|
cd thailand-water-monitor
|
|
|
|
# Install development dependencies
|
|
pip install -r requirements.txt
|
|
pip install pytest black flake8
|
|
|
|
# Run tests
|
|
pytest
|
|
|
|
# Format code
|
|
black src/
|
|
```
|
|
|
|
## 📄 License
|
|
|
|
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
|
|
## 🙏 Acknowledgments
|
|
|
|
- **Royal Irrigation Department (RID)** of Thailand for providing the data API
|
|
- **VictoriaMetrics** team for the excellent time-series database
|
|
- **Grafana** team for the visualization platform
|
|
- **Python community** for the amazing libraries and tools
|
|
|
|
## 📞 Support
|
|
|
|
- **Issues**: [GitHub Issues](https://github.com/your-username/thailand-water-monitor/issues)
|
|
- **Discussions**: [GitHub Discussions](https://github.com/your-username/thailand-water-monitor/discussions)
|
|
- **Documentation**: [Project Wiki](https://github.com/your-username/thailand-water-monitor/wiki)
|
|
|
|
---
|
|
|
|
## 📁 Project Structure
|
|
|
|
```
|
|
Northern-Thailand-Ping-River-Monitor/
|
|
├── src/ # Main application code
|
|
├── tests/ # Test suite
|
|
├── docs/ # Documentation
|
|
├── grafana/ # Grafana dashboards
|
|
├── scripts/ # Utility scripts
|
|
├── docker-compose.yml # Docker deployment
|
|
├── Makefile # Development tasks
|
|
└── requirements.txt # Dependencies
|
|
```
|
|
|
|
See [docs/PROJECT_STRUCTURE.md](docs/PROJECT_STRUCTURE.md) for detailed architecture information.
|
|
|
|
## 🔄 CI/CD & Automation
|
|
|
|
The project includes comprehensive Gitea Actions workflows:
|
|
|
|
- **🧪 CI/CD Pipeline** - Automated testing, building, and deployment
|
|
- **🔒 Security Scanning** - Daily vulnerability and dependency checks
|
|
- **📚 Documentation** - Automated API docs and validation
|
|
- **🚀 Release Management** - Automated releases with multi-arch Docker builds
|
|
|
|
See [docs/GITEA_WORKFLOWS.md](docs/GITEA_WORKFLOWS.md) for detailed workflow documentation.
|
|
|
|
## 🔗 Repository
|
|
|
|
- **Main Repository**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor
|
|
- **Issues**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/issues
|
|
- **Actions**: https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor/actions
|
|
- **Documentation**: [docs/](docs/)
|
|
|
|
**Made with ❤️ for water resource monitoring in Northern Thailand's Ping River Basin**
|