B4L/Northern-Thailand-Ping-River-Monitor
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

chore: remove the Python service; Python is the training stack only

Browse Source 
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/DATABASE_DEPLOYMENT_GUIDE.md
-447
View File
@@ -1,447 +0,0 @@
# Database Deployment Guide for Thailand Water Monitor
This guide covers deployment options for storing water monitoring data in production environments.
## 🏆 Recommendation Summary
| Database | Best For | Performance | Complexity | Cost |
|----------|----------|-------------|------------|------|
| **InfluxDB** | Time-series data, dashboards | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
| **VictoriaMetrics** | High-performance metrics | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| **PostgreSQL** | Complex queries, reliability | ⭐⭐⭐⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
| **MySQL** | Familiar, existing infrastructure | ⭐⭐⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
## 1. InfluxDB Deployment (Recommended for Time-Series)
### Why InfluxDB?
- **Purpose-built** for time-series data
- **Excellent compression** (10:1 typical ratio)
- **Built-in retention policies** and downsampling
- **Great Grafana integration** for dashboards
- **High write throughput** (100k+ points/second)
### Docker Deployment
```yaml
# docker-compose.yml
version: '3.8'
services:
influxdb:
image: influxdb:1.8
container_name: water_influxdb
ports:
- "8086:8086"
volumes:
- influxdb_data:/var/lib/influxdb
- ./influxdb.conf:/etc/influxdb/influxdb.conf:ro
environment:
- INFLUXDB_DB=water_monitoring
- INFLUXDB_ADMIN_USER=admin
- INFLUXDB_ADMIN_PASSWORD=your_secure_password
- INFLUXDB_USER=water_user
- INFLUXDB_USER_PASSWORD=water_password
restart: unless-stopped
grafana:
image: grafana/grafana:latest
container_name: water_grafana
ports:
- "3000:3000"
volumes:
- grafana_data:/var/lib/grafana
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin_password
restart: unless-stopped
volumes:
influxdb_data:
grafana_data:
```
### Environment Variables
```bash
# .env file
DB_TYPE=influxdb
INFLUX_HOST=localhost
INFLUX_PORT=8086
INFLUX_DATABASE=water_monitoring
INFLUX_USERNAME=water_user
INFLUX_PASSWORD=water_password
```
### InfluxDB Configuration
```toml
# influxdb.conf
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
# Optimize for time-series data
cache-max-memory-size = "1g"
cache-snapshot-memory-size = "25m"
cache-snapshot-write-cold-duration = "10m"
# Retention and compression
compact-full-write-cold-duration = "4h"
max-series-per-database = 1000000
max-values-per-tag = 100000
[coordinator]
write-timeout = "10s"
max-concurrent-queries = 0
query-timeout = "0s"
[retention]
enabled = true
check-interval = "30m"
[http]
enabled = true
bind-address = ":8086"
auth-enabled = true
max-body-size = "25000000"
max-concurrent-requests = 0
max-enqueued-requests = 0
```
### Production Setup Commands
```bash
# Start services
docker-compose up -d
# Create retention policies
docker exec -it water_influxdb influx -username admin -password your_secure_password -execute "
CREATE RETENTION POLICY \"raw_data\" ON \"water_monitoring\" DURATION 90d REPLICATION 1 DEFAULT;
CREATE RETENTION POLICY \"downsampled\" ON \"water_monitoring\" DURATION 730d REPLICATION 1;
"
# Create continuous queries for downsampling
docker exec -it water_influxdb influx -username admin -password your_secure_password -execute "
CREATE CONTINUOUS QUERY \"downsample_hourly\" ON \"water_monitoring\"
BEGIN
SELECT mean(water_level) AS water_level, mean(discharge) AS discharge, mean(discharge_percent) AS discharge_percent
INTO \"downsampled\".\"water_data_hourly\"
FROM \"water_data\"
GROUP BY time(1h), station_code, station_name_en, station_name_th
END
"
```
## 2. VictoriaMetrics Deployment (High Performance)
### Why VictoriaMetrics?
- **Extremely fast** and resource-efficient
- **Better compression** than InfluxDB
- **Prometheus-compatible** API
- **Lower memory usage**
- **Built-in clustering**
### Docker Deployment
```yaml
# docker-compose.yml
version: '3.8'
services:
victoriametrics:
image: victoriametrics/victoria-metrics:latest
container_name: water_victoriametrics
ports:
- "8428:8428"
volumes:
- vm_data:/victoria-metrics-data
command:
- '--storageDataPath=/victoria-metrics-data'
- '--retentionPeriod=2y'
- '--httpListenAddr=:8428'
- '--maxConcurrentInserts=16'
restart: unless-stopped
volumes:
vm_data:
```
### Environment Variables
```bash
# .env file
DB_TYPE=victoriametrics
VM_HOST=localhost
VM_PORT=8428
```
## 3. PostgreSQL Deployment (Relational + Time-Series)
### Why PostgreSQL?
- **Mature and reliable**
- **Excellent for complex queries**
- **TimescaleDB extension** for time-series optimization
- **Strong consistency guarantees**
- **Rich ecosystem**
### Docker Deployment with TimescaleDB
```yaml
# docker-compose.yml
version: '3.8'
services:
postgres:
image: timescale/timescaledb:latest-pg14
container_name: water_postgres
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
environment:
- POSTGRES_DB=water_monitoring
- POSTGRES_USER=water_user
- POSTGRES_PASSWORD=secure_password
restart: unless-stopped
volumes:
postgres_data:
```
### Database Initialization
```sql
-- init.sql
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
-- Create hypertable for time-series optimization
CREATE TABLE water_measurements (
id BIGSERIAL PRIMARY KEY,
timestamp TIMESTAMPTZ NOT NULL,
station_id INT NOT NULL,
water_level NUMERIC(10,3),
discharge NUMERIC(10,2),
discharge_percent NUMERIC(5,2),
status VARCHAR(20) DEFAULT 'active',
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Convert to hypertable (TimescaleDB)
SELECT create_hypertable('water_measurements', 'timestamp', chunk_time_interval => INTERVAL '1 day');
-- Create indexes
CREATE INDEX idx_water_measurements_station_time ON water_measurements (station_id, timestamp DESC);
CREATE INDEX idx_water_measurements_timestamp ON water_measurements (timestamp DESC);
-- Create retention policy (keep raw data for 2 years)
SELECT add_retention_policy('water_measurements', INTERVAL '2 years');
-- Create continuous aggregates for performance
CREATE MATERIALIZED VIEW water_measurements_hourly
WITH (timescaledb.continuous) AS
SELECT
time_bucket('1 hour', timestamp) AS bucket,
station_id,
AVG(water_level) as avg_water_level,
MAX(water_level) as max_water_level,
MIN(water_level) as min_water_level,
AVG(discharge) as avg_discharge,
MAX(discharge) as max_discharge,
MIN(discharge) as min_discharge,
AVG(discharge_percent) as avg_discharge_percent
FROM water_measurements
GROUP BY bucket, station_id;
-- Refresh policy for continuous aggregates
SELECT add_continuous_aggregate_policy('water_measurements_hourly',
start_offset => INTERVAL '1 day',
end_offset => INTERVAL '1 hour',
schedule_interval => INTERVAL '1 hour');
```
### Environment Variables
```bash
# .env file
DB_TYPE=postgresql
POSTGRES_CONNECTION_STRING=postgresql://water_user:secure_password@localhost:5432/water_monitoring
```
## 4. MySQL Deployment (Traditional Relational)
### Docker Deployment
```yaml
# docker-compose.yml
version: '3.8'
services:
mysql:
image: mysql:8.0
container_name: water_mysql
ports:
- "3306:3306"
volumes:
- mysql_data:/var/lib/mysql
- ./mysql.cnf:/etc/mysql/conf.d/mysql.cnf
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
environment:
- MYSQL_ROOT_PASSWORD=root_password
- MYSQL_DATABASE=water_monitoring
- MYSQL_USER=water_user
- MYSQL_PASSWORD=water_password
restart: unless-stopped
volumes:
mysql_data:
```
### MySQL Configuration
```ini
# mysql.cnf
[mysqld]
# Optimize for time-series data
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
# Partitioning support
partition = ON
# Query cache
query_cache_type = 1
query_cache_size = 128M
# Connection settings
max_connections = 200
connect_timeout = 10
wait_timeout = 600
```
### Environment Variables
```bash
# .env file
DB_TYPE=mysql
MYSQL_CONNECTION_STRING=mysql://water_user:water_password@localhost:3306/water_monitoring
```
## 5. Installation and Dependencies
### Required Python Packages
```bash
# Base requirements
pip install requests schedule
# Database-specific packages
pip install influxdb # For InfluxDB
pip install sqlalchemy pymysql # For MySQL
pip install sqlalchemy psycopg2-binary # For PostgreSQL
# VictoriaMetrics uses HTTP API (no extra packages needed)
```
### Updated requirements.txt
```txt
requests>=2.28.0
schedule>=1.2.0
pandas>=1.5.0
# Database adapters (install as needed)
influxdb>=5.3.1
sqlalchemy>=1.4.0
pymysql>=1.0.2
psycopg2-binary>=2.9.0
```
## 6. Production Deployment Examples
### Using InfluxDB (Recommended)
```bash
# Set environment variables
export DB_TYPE=influxdb
export INFLUX_HOST=your-influx-server.com
export INFLUX_PORT=8086
export INFLUX_DATABASE=water_monitoring
export INFLUX_USERNAME=water_user
export INFLUX_PASSWORD=your_secure_password
# Run the scraper
python water_scraper_v3.py
```
### Using PostgreSQL with TimescaleDB
```bash
# Set environment variables
export DB_TYPE=postgresql
export POSTGRES_CONNECTION_STRING=postgresql://water_user:password@your-postgres-server.com:5432/water_monitoring
# Run the scraper
python water_scraper_v3.py
```
### Using VictoriaMetrics
```bash
# Set environment variables
export DB_TYPE=victoriametrics
export VM_HOST=your-vm-server.com
export VM_PORT=8428
# Run the scraper
python water_scraper_v3.py
```
## 7. Monitoring and Alerting
### Grafana Dashboard Setup
1. **Add Data Source**: Configure your database as a Grafana data source
2. **Import Dashboard**: Use pre-built water monitoring dashboards
3. **Set Alerts**: Configure alerts for abnormal water levels or discharge rates
### Example Grafana Queries
#### InfluxDB Queries
```sql
-- Current water levels
SELECT last("water_level") FROM "water_data" GROUP BY "station_code"
-- Discharge trends (last 24h)
SELECT mean("discharge") FROM "water_data" WHERE time >= now() - 24h GROUP BY time(1h), "station_code"
```
#### PostgreSQL/TimescaleDB Queries
```sql
-- Current water levels
SELECT DISTINCT ON (station_id)
station_id, water_level, discharge, timestamp
FROM water_measurements
ORDER BY station_id, timestamp DESC;
-- Hourly averages (last 24h)
SELECT
time_bucket('1 hour', timestamp) as hour,
station_id,
AVG(water_level) as avg_level,
AVG(discharge) as avg_discharge
FROM water_measurements
WHERE timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY hour, station_id
ORDER BY hour DESC;
```
## 8. Performance Optimization Tips
### For All Databases
- **Batch inserts**: Insert multiple measurements at once
- **Connection pooling**: Reuse database connections
- **Indexing**: Ensure proper indexes on timestamp and station_id
- **Retention policies**: Automatically delete old data
### InfluxDB Specific
- Use **tags** for metadata (station codes, names)
- Use **fields** for numeric values (water levels, discharge)
- Configure **retention policies** and **continuous queries**
- Enable **compression** for long-term storage
### PostgreSQL/TimescaleDB Specific
- Use **hypertables** for automatic partitioning
- Create **continuous aggregates** for common queries
- Configure **compression** for older chunks
- Use **parallel queries** for large datasets
### VictoriaMetrics Specific
- Use **labels** efficiently (similar to Prometheus)
- Configure **retention periods** appropriately
- Use **downsampling** for long-term storage
- Enable **deduplication** if needed
This deployment guide provides production-ready configurations for all supported database backends. Choose the one that best fits your infrastructure and requirements.
docs/DATA_SOURCES.md
+15 -14
View File
@@ -23,7 +23,7 @@ The only source persisted to the database and used by the ML pipeline.
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
| Client | `src/water_scraper_v3.py` |
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports) |
Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
@@ -117,9 +117,9 @@ GET .../rain_24h_graph?station_type=tele_rainfall&station_id=418&start_date=...&
### 2.2 `twa-api-public.thaiwater.net` — auth-gated (x-api-key / session) 🔒
The layer our existing `src/thaiwater.py` client uses
(`GET /v2/waterlevel` with `x-api-key: $THAIWATER_API_KEY`; wired to
`GET /sensors/thaiwater` in the web API, display-only, never persisted).
The layer the website's `GET /sensors/thaiwater` uses (`GET /v2/waterlevel`
with an `x-api-key` header from `THAIWATER_API_KEY`;
`rust/collector/src/web/hii.rs`, display-only, never persisted).
Without a key: HTTP 401/500. No public key-registration page was found —
obtain a sanctioned key from HII (<https://hii.or.th>).
@@ -242,21 +242,22 @@ Oct 2024 flood). Mae Kuang Udom Thara is the second upstream reservoir.
| Source | Status | Action |
|---|---|---|
| RID hourly gauges | ✅ Ingested (hourly → PostgreSQL) | — |
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`src/thaiwater.py`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly via `src/hii_collector.py` → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| ThaiWater `/v2/waterlevel` | 🟡 Display-only (`/sensors/thaiwater`, needs `THAIWATER_API_KEY`, never persisted) | Optionally persist |
| HII `rain_24h` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_rain_stations` + `hii_rainfall` (Ping-filtered; ~300 stations) | — |
| HII `waterlevel_load` | ✅ Ingested hourly by ping-collector (`rust/collector/src/hii.rs`) → `hii_wl_stations` + `hii_waterlevel` (125 Ping stations, m MSL; `rid_code` column maps mirrors like `ridhydro_P.1` → `P.1`, `offset_msl` converts MSL → gauge datum) | — |
| HII `waterlevel_graph` | ✅ Backfill via `scripts/backfill_hii_waterlevel.py` → `hii_waterlevel` (hourly MSL + discharge, archive ≥2019; full-year windows per request; upserts never overwrite live-snapshot columns) | Run once on the box: `python scripts/backfill_hii_waterlevel.py` (defaults: 2019-01-01 → today, RID-mirror + key stations; `--stations P.1,P.67`, `--all` for every Ping station) |
| Mae Ngat reservoir (app.rid.go.th) | ✅ Ingested (2026-08-13) | Daily storage/inflow/outflow for all large dams → `rid_reservoir_daily`; candidate model features for next retrain |
| Satellite QPE (GSMaP/IMERG) | ❌ | Backfill training rainfall (GEE) |
| Open-Meteo forecasts | ❌ | Add forecast features (live + 2021 archive for training) |
| Open-Meteo rain | ✅ Hourly by ping-collector → `openmeteo_rain` (5 catchment points, past 5 days + 48 h forecast); the 2021+ archive trains the hgb-v3 rain features | — |
| HII API key (dams, forecasts) | ❌ | Contact HII for sanctioned access |
**Collector configuration** (`src/hii_collector.py`): runs automatically every
scraping cycle (hourly cadence, even while the RID scraper is in 1-minute retry
mode) from both `--web-api` and continuous-monitoring modes; one-shot via
`python -m src.main --collect-hii`. Env vars: `ENABLE_HII_COLLECTION`
(default `true`), `HII_BASIN_CODE` (default `6` = Ping). Requires a SQL
`DB_TYPE` (sqlite/postgresql/mysql); tables are created automatically.
**Collector configuration** (ping-collector): HII, the dams and Open-Meteo are
fetched once per clock hour, also while RID is being retried; one-shot via
`ping-collector once`. Env vars: `ENABLE_HII_COLLECTION`,
`ENABLE_RESERVOIR_COLLECTION`, `ENABLE_RAIN_COLLECTION` (default `true`),
`HII_BASIN_CODE` (default `6` = Ping). Tables are created automatically
(PostgreSQL). The archive backfills stay in Python (`scripts/backfill_*.py`,
using `src/hii_collector.py` and `src/rid_reservoir.py`).
**Caveats**: `api-v3` is an undocumented backend — no SLA, no ToS, may change
without notice. Poll hourly at most, cache aggressively, and pursue official
docs/DEBIAN_TROUBLESHOOTING.md
-329
View File
@@ -1,329 +0,0 @@
# Debian/Linux Troubleshooting Guide
This guide addresses common issues when running the Thailand Water Monitor on Debian and other Linux distributions.
## Fixed Issues
### SQLAlchemy Connection Error (RESOLVED)
**Error Message:**
```
2025-07-24 19:48:31,920 - ERROR - Failed to connect to SQLITE: 'Connection' object has no attribute 'commit'
2025-07-24 19:48:32,740 - ERROR - Error saving to SQLITE: 'Connection' object has no attribute 'commit'
```
**Root Cause:**
This error occurred due to incompatibility between the database adapter code and newer versions of SQLAlchemy. The code was calling `conn.commit()` on a connection object that doesn't have a `commit()` method in newer SQLAlchemy versions.
**Solution Applied:**
Changed from `engine.connect()` to `engine.begin()` context manager, which automatically handles transactions:
```python
# OLD (problematic) code:
with self.engine.connect() as conn:
conn.execute(text(sql))
conn.commit() # This fails in newer SQLAlchemy
# NEW (fixed) code:
with self.engine.begin() as conn:
conn.execute(text(sql))
# Transaction automatically committed when context exits
```
**Status:** ✅ **FIXED** - The issue has been resolved in the current version.
## Installation on Debian/Ubuntu
### System Requirements
```bash
# Update package list
sudo apt update
# Install Python and pip
sudo apt install python3 python3-pip python3-venv
# Install system dependencies for database drivers
sudo apt install build-essential python3-dev
# For MySQL support (optional)
sudo apt install default-libmysqlclient-dev
# For PostgreSQL support (optional)
sudo apt install libpq-dev
```
### Python Environment Setup
```bash
# Create virtual environment
python3 -m venv water_monitor_env
# Activate virtual environment
source water_monitor_env/bin/activate
# Install requirements
pip install -r requirements.txt
```
### Running the Monitor
```bash
# Test run
python water_scraper_v3.py --test
# Run with specific database
export DB_TYPE=sqlite
python water_scraper_v3.py
# Run demo
python demo_databases.py
```
## Common Linux Issues
### 1. Permission Errors
**Error:**
```
PermissionError: [Errno 13] Permission denied: 'water_levels.db'
```
**Solution:**
```bash
# Check current directory permissions
ls -la
# Create data directory with proper permissions
mkdir -p data
chmod 755 data
# Set database path to data directory
export WATER_DB_PATH=data/water_levels.db
```
### 2. Missing System Dependencies
**Error:**
```
ImportError: No module named '_sqlite3'
```
**Solution:**
```bash
# Install SQLite development headers
sudo apt install libsqlite3-dev
# Reinstall Python if needed
sudo apt install python3-sqlite
```
### 3. Network/Firewall Issues
**Error:**
```
requests.exceptions.ConnectionError: HTTPSConnectionPool
```
**Solution:**
```bash
# Test network connectivity
curl -I https://hyd-app-db.rid.go.th/hydro1h.html
# Check firewall rules
sudo ufw status
# Allow outbound HTTPS if needed
sudo ufw allow out 443
```
### 4. Systemd Service Setup
Create service file `/etc/systemd/system/water-monitor.service`:
```ini
[Unit]
Description=Thailand Water Level Monitor
After=network.target
[Service]
Type=simple
User=water-monitor
Group=water-monitor
WorkingDirectory=/opt/water_level_monitor
Environment=PATH=/opt/water_level_monitor/venv/bin
Environment=DB_TYPE=sqlite
Environment=WATER_DB_PATH=/opt/water_level_monitor/data/water_levels.db
ExecStart=/opt/water_level_monitor/venv/bin/python water_scraper_v3.py
Restart=always
RestartSec=60
# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/water_level_monitor/data
ReadWritePaths=/opt/water_level_monitor/logs
[Install]
WantedBy=multi-user.target
```
Enable and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable water-monitor.service
sudo systemctl start water-monitor.service
sudo systemctl status water-monitor.service
```
### 5. Log Rotation
Create `/etc/logrotate.d/water-monitor`:
```
/opt/water_level_monitor/water_monitor.log {
daily
missingok
rotate 30
compress
delaycompress
notifempty
create 644 water-monitor water-monitor
postrotate
systemctl reload water-monitor.service
endscript
}
```
## Database-Specific Issues
### SQLite
**Issue:** Database locked
```bash
# Check for processes using the database
sudo lsof /path/to/water_levels.db
# Kill processes if needed
sudo pkill -f water_scraper_v3.py
```
### VictoriaMetrics with HTTPS
**Configuration:**
```bash
export DB_TYPE=victoriametrics
export VM_HOST=https://your-vm-server.com
export VM_PORT=443
```
**Test connection:**
```bash
curl -k https://your-vm-server.com/health
```
## Performance Optimization
### 1. System Tuning
```bash
# Increase file descriptor limits
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
# Optimize network settings
echo "net.core.rmem_max = 16777216" >> /etc/sysctl.conf
echo "net.core.wmem_max = 16777216" >> /etc/sysctl.conf
sysctl -p
```
### 2. Database Optimization
```bash
# For SQLite
export SQLITE_CACHE_SIZE=10000
export SQLITE_SYNCHRONOUS=NORMAL
# Monitor database size
du -h data/water_levels.db
```
## Monitoring and Maintenance
### Health Check Script
Create `health_check.sh`:
```bash
#!/bin/bash
LOG_FILE="/opt/water_level_monitor/water_monitor.log"
SERVICE_NAME="water-monitor"
# Check if service is running
if ! systemctl is-active --quiet $SERVICE_NAME; then
echo "ERROR: $SERVICE_NAME is not running"
systemctl restart $SERVICE_NAME
exit 1
fi
# Check recent log entries
RECENT_ERRORS=$(tail -n 100 $LOG_FILE | grep -c "ERROR")
if [ $RECENT_ERRORS -gt 5 ]; then
echo "WARNING: $RECENT_ERRORS errors found in recent logs"
exit 1
fi
echo "OK: Service is healthy"
exit 0
```
### Cron Job for Health Checks
```bash
# Add to crontab
*/5 * * * * /opt/water_level_monitor/health_check.sh >> /var/log/water-monitor-health.log 2>&1
```
## Getting Help
### Debug Information
```bash
# System information
uname -a
python3 --version
pip list | grep -E "(sqlalchemy|requests|influxdb)"
# Service logs
journalctl -u water-monitor.service -f
# Application logs
tail -f water_monitor.log
# Database information
sqlite3 water_levels.db ".schema"
sqlite3 water_levels.db "SELECT COUNT(*) FROM water_measurements;"
```
### Common Commands
```bash
# Restart service
sudo systemctl restart water-monitor.service
# View logs
sudo journalctl -u water-monitor.service --since "1 hour ago"
# Test configuration
python config.py
# Test database connection
python demo_databases.py
# Manual data fetch
python water_scraper_v3.py --test
```
This troubleshooting guide should help resolve most common issues encountered when running the Thailand Water Monitor on Debian and other Linux distributions.
docs/FLOOD_FORECASTING.md
+46 -48
View File
@@ -62,8 +62,8 @@ P.5 before 2022-01-01; the rest are handled by the per-row coverage gates.
**How `src/ml/data.py` loads it.** `resolve_db_url()` picks a connection string in
priority order: an explicit `--db-url` argument, then the `FLOOD_ML_DB_URL`
environment variable, then `Config.get_database_config()` when `DB_TYPE` is
`postgresql`, else `None`. `load_measurements()` then tries three tiers:
environment variable, then the `POSTGRES_*` settings (`src/config.py`, the same
`.env` the service reads), else `None`. `load_measurements()` then tries three tiers:
1. **PostgreSQL** (`_fetch_from_db`) — the primary path. NULL discharge stays
NULL, which matters because the models must learn from the real missingness
@@ -614,32 +614,28 @@ fix, the served features match what this section evaluates.
## 6. Deployment
### API
### Serving
With `FORECAST_IN_COLLECTOR=true` the forecast is computed by `ping-collector`
(`rust/forecast`, a parity-tested port of this serving path) from
The forecast is computed by `ping-collector` (`rust/forecast`, a parity-tested
port of `src/ml/predict.py`, which stays as the reference implementation) from
`models/flood_<station>.json.gz`, which `train_all()` exports next to each
joblib bundle; `/forecast` serves its `forecast_current` row and computes
in-process only when that is older than `FORECAST_MAX_AGE_SECONDS` (2 h). See
`rust/README.md`. The rest of this section describes the in-process path.
joblib bundle. It runs after every collection cycle that brought newer
readings and at least hourly, stores the rows in `forecast_history` (section 9's
skill tracking) and `forecast_current`, and `GET /forecast` serves the stored
row: one JSON row per station × horizon with the fields listed in section 1.
A stored forecast older than `FORECAST_MAX_AGE_SECONDS` (2 h, i.e. the service
has stopped forecasting) is still served, marked `X-Data-Stale: true`. See
`rust/README.md`.
`GET /forecast` (`src/web_api.py`) returns one JSON row per station × horizon with
the fields listed in section 1. Results are cached in-process for
`FORECAST_TTL = 900` seconds (15 minutes), which matches the data cadence — the
underlying readings do not update faster than hourly. Inference runs in a thread
via `asyncio.to_thread` so it never blocks the event loop.
A station without a model bundle, or whose bundle needs a feature the Rust side
cannot build, gets the heuristic, as in Python. So **before the first training
run `/forecast` returns 200 with an all-heuristic payload**, provided there is
recent gauge data; you get an empty `200 []` only when there is no recent data
at all. Judge deployment state by the `source` field, not the status code.
Failure modes: **503** if the `src.ml` package cannot be imported (missing
scikit-learn, say), **502** on any other exception.
One behaviour worth knowing, because the code comments suggest otherwise: the
endpoint's `FileNotFoundError` ("No trained flood models found") and `RuntimeError`
handlers are unreachable — nothing in `src/ml/` raises either, and
`predict._forecast_station` checks `bundle_path.exists()` and falls back to the
heuristic instead. So **before the first training run `/forecast` returns 200 with
an all-heuristic payload**, not a 503, provided there is recent gauge data; you
get an empty `200 []` only when there is no recent data at all. Judge deployment
state by the `source` field, not the status code.
The Python serving path (`predict.get_latest_forecasts`) is not called by the
service any more; `scripts/forecast_parity.py` runs both on the live database
and `scripts/install.sh` does that on every install.
### Dashboard
@@ -661,13 +657,14 @@ The server already has the PostgreSQL connection configured, so no host override
is needed:
```bash
cd /path/to/Northern-Thailand-Ping-River-Monitor
python scripts/train_flood_model.py --stations all
cd /opt/thailand-water-monitor
uv run python scripts/train_flood_model.py --stations all
```
`resolve_db_url()` picks up `Config.get_database_config()` automatically when
`DB_TYPE=postgresql`. The run writes fifteen `models/flood_{station}.joblib`
bundles plus `models/metrics.json`.
`resolve_db_url()` picks up the `POSTGRES_*` settings from `.env`. The run writes
fifteen `models/flood_{station}.joblib` bundles, their `.json.gz` exports for the
service, and `models/metrics.json`. The monthly timer does the same through
`scripts/retrain.sh` with a staged promote (section 7).
### Artifacts and dependencies
@@ -733,7 +730,10 @@ gap between fetch and first model could not be reconstructed from the surviving
artifacts. Do not attribute it to per-station training cost. Either way the
conclusion holds: **retraining is minutes, not tens of minutes.**
**Inference** (15 bundles, 16 stations × 3 horizons = 48 rows):
**Inference in Python** (15 bundles, 16 stations × 3 horizons = 48 rows; the
retired in-process serving path, measured before the Rust forecaster took
over: there the whole run takes milliseconds, 0.25 ms per station against
158 ms, and the 15 exported models need about 70 MB resident):
| Measurement | Value |
|---|---|
@@ -753,9 +753,9 @@ one-time lazy `sklearn.ensemble` import — the remaining fourteen average 159 m
Of the ~640 ms warm compute, model prediction is ~525 ms, the hourly grid 47 ms,
and feature building 71 ms across all sixteen stations.
**Live-endpoint measurements** (a second, independent benchmark run against a real
uvicorn instance of the app, same day, same workstation, RSS summed over the
process tree):
**Live-endpoint measurements of the retired Python API** (a second, independent
benchmark run against a real uvicorn instance of the app, same day, same
workstation, RSS summed over the process tree):
| Measurement | Value |
|---|---|
@@ -775,14 +775,11 @@ Training shows the opposite (2.3× slower single-threaded), so set the variable
per process, never globally.
**Server sizing, in plain terms:** this is a small workload and almost any server
runs it. **RAM is the binding constraint, not CPU.** Budget about **1 GB for the
API process** so the ~335 MB steady state has headroom on top of the rest of the
app; training peaks at only ~210–315 MB and can share the same box. No GPU
anywhere. Pin thread counts per process — `OMP_NUM_THREADS=1` in the serving
unit, `OMP_NUM_THREADS=4` for retraining so it cannot monopolise every core while
the API is serving. And since a cold call costs seconds against a 2.4 ms cache
hit, consider warming `/forecast` once at startup rather than letting a user
absorb it.
runs it. The service (`ping-collector`, Rust) stays under 100 MB resident with
every model loaded (its unit caps it at 512 MB); the monthly retrain peaks at
~210–315 MB and shares the same box. No GPU anywhere. The retrain unit sets
`OMP_NUM_THREADS=4` so it cannot monopolise every core while the site is
serving.
## 7. Retraining policy
@@ -821,15 +818,15 @@ fires `water-monitor-retrain.service` on the 1st of every month at 03:30 server
time (`Persistent=true`, so a missed run catches up at boot). The unit runs
`scripts/retrain.sh` as the service user with `OMP_NUM_THREADS=4`, `Nice=15`:
1. trains all stations into `models/.staging/` (the API keeps serving the old
bundles throughout);
1. trains all stations into `models/.staging/` (the service keeps serving the
old bundles throughout);
2. refuses to promote unless `metrics.json` reports a `hgb-v3+` version and at
least 14 trained stations (exit 3, staging discarded, old models untouched);
3. renames the new bundles into `models/`, moving the previous generation to
`models/.previous/` for rollback.
No API restart: `predict.py` reloads bundles by mtime on the next hourly
precompute. `systemctl list-timers water-monitor-retrain.timer` shows the next
No restart: ping-collector reloads changed `.json.gz` exports by mtime on its
next forecast run. `systemctl list-timers water-monitor-retrain.timer` shows the next
run; `sudo systemctl start water-monitor-retrain.service` runs it now (after a
flood, say); `journalctl -u water-monitor-retrain` has the log. The installer
(`scripts/install.sh`) enables the timer.
@@ -912,9 +909,10 @@ non-model-backed, and all of them are visible in the payload:
horizon's head is in the bundle's `skipped_heads`, almost always because the
station had fewer than 30 positive examples for that threshold.
- A whole station flipping to heuristic after a code change — the feature-name
check in `_model_forecast` caught a mismatch between the live feature builder
and the stored `feature_names`. The fix is to retrain; the log line names the
missing columns.
check caught a mismatch between the feature builder and the stored
`feature_names` (in Python `_model_forecast`; in Rust a bundle needing a
feature `rust/forecast` cannot build). The fix is to retrain or to port the
feature; the log line names the missing columns.
A live example from the 2026-08-10 cache: of 48 forecast rows, 42 came from
models and 6 were heuristic — three for P.4A, which has no bundle by design, and
docs/GAP_FILLING_GUIDE.md
-275
View File
@@ -1,275 +0,0 @@
# Gap Filling and Data Integrity Guide
This guide explains the enhanced gap-filling functionality that addresses data gaps and missing timestamps in the Thailand Water Monitor.
## ✅ **Issues Resolved**
### **1. Data Gaps Problem**
- **Before**: Tool only fetched current day data, leaving gaps in historical records
- **After**: Automatically detects and fills missing timestamps for the last 7 days
### **2. Missing Midnight Timestamps**
- **Before**: Jump from 23:00 to 01:00 (missing 00:00 midnight data)
- **After**: Specifically checks for and fills midnight hour gaps
### **3. Changed Values**
- **Before**: No mechanism to update existing data if values changed on the server
- **After**: Compares existing data with fresh API data and updates changed values
## 🔧 **New Features**
### **Command Line Interface**
```bash
# Check for missing data gaps
python water_scraper_v3.py --check-gaps [days]
# Fill missing data gaps
python water_scraper_v3.py --fill-gaps [days]
# Update existing data with latest values
python water_scraper_v3.py --update-data [days]
# Run single test cycle
python water_scraper_v3.py --test
# Show help
python water_scraper_v3.py --help
```
### **Automatic Gap Detection**
The system now automatically:
- Generates expected hourly timestamps for the specified time range
- Compares with existing database records
- Identifies missing timestamps
- Groups missing data by date for efficient API calls
### **Intelligent Gap Filling**
- **Historical Data Fetching**: Retrieves data for specific dates to fill gaps
- **Selective Insertion**: Only inserts data for actually missing timestamps
- **API Rate Limiting**: Includes delays between API calls to be respectful
- **Error Handling**: Continues processing even if some dates fail
### **Data Update Mechanism**
- **Change Detection**: Compares water levels, discharge rates, and percentages
- **Precision Checking**: Uses appropriate thresholds (0.001m for water level, 0.1 cms for discharge)
- **Selective Updates**: Only updates records where values have actually changed
## 📊 **Test Results**
### **Before Enhancement**
```
Found 22 missing timestamps in the last 2 days:
2025-07-23: Missing hours [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
2025-07-24: Missing hours [0, 20, 21, 22, 23]
2025-07-25: Missing hours [0, 9]
```
### **After Gap Filling**
```
Gap filling completed. Filled 96 missing data points
Remaining gaps:
2025-07-24: Missing hours [10]
2025-07-25: Missing hours [0, 10]
```
**Improvement**: Reduced from 22 missing timestamps to 3 (86% improvement)
## 🚀 **Enhanced Scraping Cycle**
The regular scraping cycle now includes three phases:
### **Phase 1: Current Data Collection**
```python
# Fetch and save current data
water_data = self.fetch_water_data()
success = self.save_to_database(water_data)
```
### **Phase 2: Gap Filling (Last 7 Days)**
```python
# Check for and fill missing data
filled_count = self.fill_data_gaps(days_back=7)
```
### **Phase 3: Data Updates (Last 2 Days)**
```python
# Update existing data with latest values
updated_count = self.update_existing_data(days_back=2)
```
## 🔧 **Technical Improvements**
### **Database Connection Handling**
- **SQLite Optimization**: Added timeout and thread safety parameters
- **Retry Logic**: Exponential backoff for database lock errors
- **Transaction Management**: Proper use of `engine.begin()` for automatic commits
### **Error Recovery**
```python
# Retry logic with exponential backoff
for attempt in range(max_retries):
try:
success = self.db_adapter.save_measurements(water_data)
if success:
return True
except Exception as e:
if "database is locked" in str(e).lower():
time.sleep(2 ** attempt) # 1s, 2s, 4s delays
continue
```
### **Memory Efficiency**
- **Selective Data Processing**: Only processes data for missing timestamps
- **Batch Processing**: Groups operations by date to minimize API calls
- **Resource Management**: Proper cleanup and connection handling
## 📋 **Usage Examples**
### **Daily Maintenance**
```bash
# Check for gaps in the last week
python water_scraper_v3.py --check-gaps 7
# Fill any found gaps
python water_scraper_v3.py --fill-gaps 7
# Update recent data for accuracy
python water_scraper_v3.py --update-data 2
```
### **Historical Data Recovery**
```bash
# Check for gaps in the last month
python water_scraper_v3.py --check-gaps 30
# Fill gaps for the last month (be patient, this takes time)
python water_scraper_v3.py --fill-gaps 30
```
### **Production Monitoring**
```bash
# Quick test to ensure system is working
python water_scraper_v3.py --test
# Check for recent gaps
python water_scraper_v3.py --check-gaps 1
```
## 🔍 **Monitoring and Alerts**
### **Gap Detection Output**
```
Found 22 missing timestamps:
2025-07-23: Missing hours [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
2025-07-24: Missing hours [0, 20, 21, 22, 23]
2025-07-25: Missing hours [0, 9]
```
### **Gap Filling Progress**
```
Fetching data for 2025-07-24 to fill 5 missing timestamps
Successfully fetched 368 data points from API for 2025-07-24
Filled 80 data points for 2025-07-24
Gap filling completed. Filled 96 missing data points
```
### **Update Detection**
```
Checking for updates on 2025-07-24
Update needed for P.1 at 2025-07-24 15:00:00
Updated 5 measurements for 2025-07-24
Data update completed. Updated 5 measurements
```
## ⚙️ **Configuration Options**
### **Environment Variables**
```bash
# Database configuration
export DB_TYPE=sqlite
export WATER_DB_PATH=water_monitoring.db
# Gap filling settings (can be added to config.py)
export GAP_FILL_DAYS=7 # Days to check for gaps
export UPDATE_DAYS=2 # Days to check for updates
export API_DELAY=1 # Seconds between API calls
export MAX_RETRIES=3 # Database retry attempts
```
### **Customizable Parameters**
- **Gap Check Period**: Default 7 days, configurable via command line
- **Update Period**: Default 2 days, configurable via command line
- **API Rate Limiting**: 1-second delay between calls (configurable)
- **Retry Logic**: 3 attempts with exponential backoff (configurable)
## 🛠️ **Troubleshooting**
### **Common Issues**
#### **Database Locked Errors**
```
ERROR - Error saving to SQLITE: database is locked
```
**Solution**: The retry logic now handles this automatically with exponential backoff.
#### **API Rate Limiting**
```
WARNING - Too many requests to API
```
**Solution**: Increase delay between API calls or reduce the number of days processed at once.
#### **Missing Data Still Present**
```
Found X missing timestamps after gap filling
```
**Possible Causes**:
- Data not available on the Thai government server for those timestamps
- Network issues during API calls
- API returned empty data for those specific times
### **Debug Commands**
```bash
# Enable debug logging
export LOG_LEVEL=DEBUG
python water_scraper_v3.py --check-gaps 1
# Test specific date range
python water_scraper_v3.py --fill-gaps 1
# Check database directly
sqlite3 water_monitoring.db "SELECT COUNT(*) FROM water_measurements;"
sqlite3 water_monitoring.db "SELECT timestamp, COUNT(*) FROM water_measurements GROUP BY timestamp ORDER BY timestamp DESC LIMIT 10;"
```
## 📈 **Performance Metrics**
### **Gap Filling Efficiency**
- **API Calls**: Grouped by date to minimize requests
- **Processing Speed**: ~100-400 data points per API call
- **Success Rate**: 86% gap reduction in test case
- **Resource Usage**: Minimal memory footprint with selective processing
### **Database Performance**
- **SQLite Optimization**: Connection pooling and timeout handling
- **Transaction Efficiency**: Batch inserts with proper transaction management
- **Retry Success**: Automatic recovery from temporary lock conditions
## 🎯 **Best Practices**
### **Regular Maintenance**
1. **Daily**: Run `--check-gaps 1` to monitor recent data quality
2. **Weekly**: Run `--fill-gaps 7` to catch any missed data
3. **Monthly**: Run `--update-data 7` to ensure data accuracy
### **Production Deployment**
1. **Automated Scheduling**: Use cron or systemd timers for regular gap checks
2. **Monitoring**: Set up alerts for excessive missing data
3. **Backup**: Regular database backups before major gap-filling operations
### **Data Quality Assurance**
1. **Validation**: Check for reasonable value ranges after gap filling
2. **Comparison**: Compare filled data with nearby timestamps for consistency
3. **Documentation**: Log all gap-filling activities for audit trails
This enhanced gap-filling system ensures comprehensive and accurate water level monitoring with minimal data loss and automatic recovery capabilities.
docs/GITEA_WORKFLOWS.md
-300
View File
@@ -1,300 +0,0 @@
# 🔄 Gitea Actions Workflows - Northern Thailand Ping River Monitor
## 📋 Overview
This document describes the Gitea Actions workflows configured for the Northern Thailand Ping River Monitor project. These workflows provide comprehensive CI/CD, security scanning, and documentation generation.
## 🚀 Available Workflows
### 1. **CI/CD Pipeline** (`.gitea/workflows/ci.yml`)
**Triggers:**
- Push to `main` or `develop` branches
- Pull requests to `main`
- Daily scheduled runs at 2 AM UTC
**Jobs:**
- **Test Suite**: Multi-version Python testing (3.9-3.12)
- **Code Quality**: Linting, formatting, and type checking
- **Build**: Docker image creation and testing
- **Integration Test**: Testing with VictoriaMetrics service
- **Deploy Staging**: Automatic deployment to staging (develop branch)
- **Deploy Production**: Manual deployment to production (main branch)
- **Performance Test**: Load testing after production deployment
**Key Features:**
- ✅ Multi-Python version testing
- ✅ Docker multi-architecture builds (amd64, arm64)
- ✅ Service integration testing
- ✅ Automatic staging deployment
- ✅ Manual production approval
- ✅ Performance validation
### 2. **Security & Dependency Updates** (`.gitea/workflows/security.yml`)
**Triggers:**
- Daily scheduled runs at 3 AM UTC
- Manual dispatch
- Changes to requirements files or Dockerfile
**Jobs:**
- **Dependency Scan**: Safety, Bandit, Semgrep security scans
- **Docker Security**: Trivy vulnerability scanning
- **License Check**: License compliance verification
- **Dependency Updates**: Automated update detection
- **Code Quality**: Complexity and maintainability analysis
**Key Features:**
- 🔒 Daily security scans
- 📦 Dependency vulnerability detection
- 📄 License compliance checking
- 🔄 Automated update notifications
- 📊 Code quality metrics
### 3. **Release Workflow** (`.gitea/workflows/release.yml`)
**Triggers:**
- Git tags matching `v*.*.*` pattern
- Manual dispatch with version input
**Jobs:**
- **Create Release**: Automated release creation with changelog
- **Test Release**: Comprehensive testing across Python versions
- **Build Release**: Multi-architecture Docker images with proper tags
- **Security Scan**: Trivy security scanning of release images
- **Deploy Release**: Production deployment with health checks
- **Validate Release**: Post-deployment validation and testing
**Key Features:**
- 🏷️ Automated release creation
- 📝 Changelog generation
- 🐳 Multi-architecture Docker builds
- 🔒 Security scanning
- ✅ Comprehensive validation
### 4. **Documentation** (`.gitea/workflows/docs.yml`)
**Triggers:**
- Changes to documentation files
- Changes to Python source files
- Manual dispatch
**Jobs:**
- **Validate Docs**: Link checking and structure validation
- **Generate API Docs**: OpenAPI specification generation
- **Build Sphinx Docs**: Comprehensive API documentation
- **Documentation Summary**: Build status and artifact summary
**Key Features:**
- 📚 Automated API documentation
- 🔗 Link validation
- 📖 Sphinx documentation generation
- ✅ Documentation completeness checking
## 🔧 Workflow Configuration
### **Required Secrets**
Configure these secrets in your Gitea repository settings:
```bash
GITEA_TOKEN # Gitea access token for container registry
GITHUB_TOKEN # GitHub personal access token for better rate limits
SLACK_WEBHOOK_URL # Optional: Slack notifications
STAGING_WEBHOOK_URL # Optional: Staging deployment webhook
PRODUCTION_WEBHOOK_URL # Optional: Production deployment webhook
```
**Setting up GitHub Token:**
1. Go to GitHub → Settings → Developer settings → Personal access tokens
2. Generate a new token with `public_repo` scope (for public repos) or `repo` scope (for private repos)
3. Copy the token
4. In your Gitea repository → Settings → Secrets → Add `GITHUB_TOKEN` with the copied value
### **Environment Variables**
Key environment variables used across workflows:
```yaml
PYTHON_VERSION: '3.11' # Default Python version
REGISTRY: git.b4l.co.th # Container registry
IMAGE_NAME: grabowski/northern-thailand-ping-river-monitor
```
## 📊 Workflow Status
### **CI/CD Pipeline Status**
- **Test Coverage**: Multi-version Python testing
- **Code Quality**: Automated linting and formatting
- **Security**: Integrated security scanning
- **Deployment**: Automated staging, manual production
### **Security Monitoring**
- **Daily Scans**: Automated vulnerability detection
- **Dependency Updates**: Proactive update notifications
- **License Compliance**: Automated license checking
- **Code Quality**: Continuous quality monitoring
### **Release Management**
- **Automated Releases**: Tag-based release creation
- **Multi-Architecture**: Support for amd64 and arm64
- **Security Validation**: Pre-deployment security checks
- **Health Monitoring**: Post-deployment validation
## 🚀 Usage Examples
### **Triggering Workflows**
**Manual CI/CD Run:**
```bash
# Push to trigger CI/CD
git push origin main
# Create pull request to trigger testing
git checkout -b feature/new-feature
git push origin feature/new-feature
# Create PR in Gitea UI
```
**Manual Security Scan:**
```bash
# Trigger via Gitea Actions UI
# Go to Actions → Security & Dependency Updates → Run workflow
```
**Creating a Release:**
```bash
# Create and push a tag
git tag v3.1.1
git push origin v3.1.1
# Or use manual dispatch in Gitea Actions UI
```
### **Monitoring Workflow Results**
**Check Workflow Status:**
1. Navigate to your repository in Gitea
2. Click on "Actions" tab
3. View workflow runs and their status
**Download Artifacts:**
1. Click on a completed workflow run
2. Scroll to "Artifacts" section
3. Download reports and logs
**View Security Reports:**
1. Go to Security workflow runs
2. Download security-reports artifacts
3. Review JSON reports for vulnerabilities
## 🔍 Troubleshooting
### **Common Issues**
**Workflow Fails on Dependencies:**
```bash
# Check requirements.txt for version conflicts
pip-compile requirements.in
```
**Docker Build Fails:**
```bash
# Test Docker build locally
make docker-build
docker run --rm ping-river-monitor python run.py --test
```
**Security Scan Failures:**
```bash
# Run security scans locally
safety check -r requirements.txt
bandit -r src/
```
**Test Failures:**
```bash
# Run tests locally
make test
python tests/test_integration.py
```
### **Debugging Workflows**
**Enable Debug Logging:**
Add to workflow file:
```yaml
env:
ACTIONS_STEP_DEBUG: true
ACTIONS_RUNNER_DEBUG: true
```
**Check Workflow Logs:**
1. Go to failed workflow run
2. Click on failed job
3. Expand failed step to see detailed logs
**Validate Workflow Syntax:**
```bash
# Validate YAML syntax
make validate-workflows
```
## 📈 Performance Optimization
### **Caching Strategy**
- **Pip Cache**: Cached across workflow runs
- **Docker Layer Cache**: GitHub Actions cache for faster builds
- **Dependency Cache**: Cached based on requirements.txt hash
### **Parallel Execution**
- **Matrix Builds**: Multiple Python versions tested in parallel
- **Independent Jobs**: Security scans run independently of tests
- **Conditional Execution**: Jobs skip when not needed
### **Resource Management**
- **Timeout Settings**: Prevent hanging workflows
- **Resource Limits**: Appropriate runner sizing
- **Artifact Cleanup**: Automatic cleanup of old artifacts
## 🔒 Security Best Practices
### **Secret Management**
- Use Gitea repository secrets for sensitive data
- Never commit secrets to repository
- Rotate secrets regularly
- Use least-privilege access tokens
### **Container Security**
- Multi-stage Docker builds for smaller images
- Non-root user in containers
- Regular base image updates
- Vulnerability scanning before deployment
### **Code Security**
- Automated security scanning in CI/CD
- Dependency vulnerability monitoring
- License compliance checking
- Code quality enforcement
## 📚 Additional Resources
### **Gitea Actions Documentation**
- [Gitea Actions Overview](https://docs.gitea.io/en-us/usage/actions/)
- [Workflow Syntax](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
- [Available Actions](https://github.com/marketplace?type=actions)
### **Project-Specific Resources**
- [Contributing Guide](../CONTRIBUTING.md)
### **Monitoring and Alerts**
- Workflow status badges in README
- Email notifications for failures
- Slack/Discord integration for team updates
- Grafana dashboards for deployment metrics
---
**Workflow Version**: v3.1.3
**Last Updated**: 2025-08-12
**Maintained By**: Ping River Monitor Team
docs/MATRIX_QUICK_START.md
+2 -6
View File
@@ -32,13 +32,9 @@ curl -X POST https://matrix.org/_matrix/client/v3/login \
## Step 4: Configure .env File
Add these to your `.env` file:
```bash
# Matrix Alerting Configuration
MATRIX_HOMESERVER=https://matrix.org
MATRIX_ACCESS_TOKEN=syt_your_access_token_here
MATRIX_ROOM_ID=!your_room_id:matrix.org
# Grafana Integration (optional)
GRAFANA_URL=http://localhost:3000
```
## Step 5: Restart and check
@@ -71,8 +67,8 @@ so a Matrix outage is retried on Matrix only and does not re-send ntfy.
The old `run.py --alert-check` cron job (every 15 minutes, repeating an alert
for as long as a level stayed high) is replaced by this;
`scripts/install_collector.sh` comments it out of the crontab once
ping-collector sends Matrix, keeping a backup next to the app.
`scripts/install.sh` comments it out of the crontab if it is still there,
keeping a backup next to the app.
## Troubleshooting
- **HTTP 401/403** in `journalctl -u ping-collector`: the access token is invalid
docs/NOTIFICATIONS.md
+33 -25
View File
@@ -36,12 +36,11 @@ the public dashboard.
## How it runs
Since phase 2 of the Rust rewrite the notifier runs in `ping-collector`
(`rust/collector/src/notify.rs`) when `FORECAST_IN_COLLECTOR=true`, after each
forecast run. It is a port of `src/notify.py` with the same topics, texts and
state keys (checked word for word against Python on 751 generated messages),
so switching between the two sends nothing twice. One difference: the Rust
notifier also uses ThaiWater (HII) telemetry. For the stations whose HII twin
The notifier runs in `ping-collector` (`rust/collector/src/notify.rs`) when
`FORECAST_IN_COLLECTOR=true`, after each forecast run. It is a port of the
Python notifier it replaced in 2026-10, with the same topics, texts and state
keys (checked word for word on 751 generated messages), so the switch sent
nothing twice. One difference: it also uses ThaiWater (HII) telemetry. For the stations whose HII twin
is the same sensor (P.1, P.103, P.20, P.4A, P.67, P.75, P.82, P.84, P.92, and
P.81 bias-corrected), a reading newer than RID's drives the level alerts; the
message then says "ThaiWater telemetry; the RID report is not in yet". RID's
@@ -52,12 +51,12 @@ once after `RID_STALE_ALERT_HOURS` (18, 0 = off). With `MATRIX_*` set, the
same transitions also go to a Matrix room, each once (per station, outlook,
feed; not the per-topic summaries), with their own `matrix:` state keys.
With `FORECAST_IN_COLLECTOR=false`, `src/notify.py` is called once per collection cycle inside the API process
(leader only), right after the forecast precompute, so it sees exactly the
readings and forecasts the dashboard shows. Per-key last-sent state is stored
in the `notification_state` table of the monitor's own database, so a restart
or redeploy never re-sends and never misses a crossing that happened while
the service was down (the next cycle compares against the persisted state).
It runs in the same process that serves the dashboard, right after the
forecast, so it sees exactly the readings and forecasts the dashboard shows.
Per-key last-sent state is stored in the `notification_state` table of the
monitor's own database, so a restart or redeploy never re-sends and never
misses a crossing that happened while the service was down (the next cycle
compares against the persisted state).
If ntfy is unreachable the transition is **not** recorded, so it is retried
on the next cycle rather than silently lost. Any other failure in the notify
@@ -86,8 +85,8 @@ token, and appends `NTFY_SERVER` (public URL for subscribers),
and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
```bash
systemctl restart water-monitor
journalctl -u water-monitor -n 20 | grep ntfy # "ntfy notifications: https://... topics ping-*"
systemctl restart ping-collector
journalctl -u ping-collector -n 20 | grep ntfy # "ntfy notifications: publish to ..., topics ping-*"
curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1' # anonymous read works
```
@@ -120,21 +119,30 @@ holds back an alert. Test the pipeline before the domain is live with
| `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
| `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |
Tunables in `src/notify.py`: `CLEAR_MARGIN_M` (0.10), `OUTLOOK_ON` / `OUTLOOK_OFF`
(0.50 / 0.25), stale feed threshold (3 h, argument to `evaluate`).
| `RID_STALE_ALERT_HOURS` | `18` | report RID lagging this long while ThaiWater is current (0 = never) |
| `MATRIX_HOMESERVER` / `MATRIX_ACCESS_TOKEN` / `MATRIX_ROOM_ID` | *(empty = off)* | the team room (docs/MATRIX_QUICK_START.md) |
Tunables in `src/ml/export.py` (compiled into ping-collector through
`rust/forecast/flood_config.json`: regenerate with
`uv run python -m src.ml.export --config` and rebuild): `CLEAR_MARGIN_M`
(0.10), `OUTLOOK_ON` / `OUTLOOK_OFF` (0.50 / 0.25), `FEED_STALE_AFTER_H` (3),
the capacity guard and the station names in the messages.
## Testing
`tests/test_notify.py` covers the state machine: quiet river sends nothing;
crossing once, then silence while above, then all-clear; hysteresis on the way
down; escalation to danger and back; basin digest grouping; outlook on/off;
heuristic forecasts ignored; stale feed and recovery; state survives a restart
through sqlite; a failed publish is retried next cycle.
The tests in `rust/collector/src/notify.rs` cover the state machine: quiet
river sends nothing; crossing once, then silence while above, then all-clear;
hysteresis on the way down; escalation to danger and back; outlook on/off with
heuristic forecasts ignored; HII readings replacing a lagging RID reading;
stale feed only when both sources are; RID lagging reported once; a failed
publish retried next cycle; the Matrix rendering; and the 652-step replay of
the Python notifier's messages. `notification_state` round-trips against
PostgreSQL in `db_tests.rs`.
To exercise the real path against a real ntfy locally: run `ntfy serve` (any
platform, same binary), set `NTFY_SERVER`/`NTFY_TOKEN`, seed readings, and
poll the topic JSON. `scripts/e2e_notify.py` does exactly that if you want a
template.
`ping-collector notify --dry-run` shows the readings judged and what would be
sent right now, changing nothing. To exercise the real path against a real
ntfy locally: run `ntfy serve` (any platform, same binary), point
`NTFY_SERVER`/`NTFY_TOKEN` at it and poll the topic JSON.
## Why ntfy and not …
docs/RUST_REWRITE.md
+25 -10
View File
@@ -1,9 +1,17 @@
# A Rust rewrite: assessment
# The Rust rewrite
Status: **in progress.** Phase 1 (collector) is in production since 2026-10-17,
phase 2 (forecast + ntfy) and phase 3 (website/API + Matrix) since
2026-10-18; see
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
Status: **done** (2026-10-18). The service is `ping-collector`
([rust/README.md](../rust/README.md)); Python keeps model training,
evaluation and the archive backfills. Phases 1-3 replaced the Python
collector, the forecast + notifier and the website/API one at a time, each
checked against the Python side on production data before the switch;
phase 4 removed the Python service code. The migration tooling
(`scripts/collector_parity.py`, `scripts/api_parity.py`,
`scripts/install_collector.sh` with its cutover/rollback modes, the
`spikes/rust-forecast` spike) is in git history before the cleanup commit.
The assessment below was written before phase 1 and is kept as the record of
why and how.
## Short version
@@ -87,7 +95,7 @@ This slows model iteration, which is the most active part of the repo.
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | in production since 2026-10-17 23:09 (Python: `COLLECTION_ENABLED=false`) |
| 2 | Forecast crate + ntfy | in production since 2026-10-18 15:08 (`FORECAST_IN_COLLECTOR=true`); live check on production data: 48/48 rows identical, 96/96 probabilities bit-identical |
| 3 | Website + read API in `ping-collector`, Matrix like ntfy | in production since 2026-10-18 16:40 (`WEB_LISTEN`, `water-monitor.service` disabled); on production data 53 of 53 compared routes identical to FastAPI (`scripts/api_parity.py`) |
| 4 | Cleanup | |
| 4 | Cleanup: Python service code removed | done 2026-10-18 |
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
1. **Collector daemon** (`rust/collector`, ~1.5k LOC incl. tests): RID with the
@@ -96,7 +104,7 @@ This slows model iteration, which is the most active part of the repo.
API runs with `COLLECTION_ENABLED=false` and re-forecasts when the newest
RID reading in the database advances. ntfy moved to phase 2: the outlook
alert needs the forecasts, which are still Python. Cutover and rollback
are one command each (`scripts/install_collector.sh`).
were one command each (`scripts/install_collector.sh`, since removed).
2. **Forecast crate + ntfy** (`rust/forecast`, run by `ping-collector` with
`FORECAST_IN_COLLECTOR=true`): all stations, HII/rain/dam inputs, heuristic
fallback, probabilities and stages, `forecast_history`, `forecast_current`
@@ -130,9 +138,16 @@ This slows model iteration, which is the most active part of the repo.
their own `matrix:` state keys seeded from ntfy's, replacing the
`run.py --alert-check` cron job that repeated alerts every 15 minutes.
Python keeps training, evaluation and the monthly retrain timer.
4. **Cleanup:** drop FastAPI/uvicorn, the Python collector, notifier, CLI and
the multi-backend adapters (SQLite, MySQL, InfluxDB, VictoriaMetrics) and
demo databases; the Python environment shrinks to the training stack.
4. **Cleanup** (done): FastAPI/uvicorn, the Python collector, notifier, Matrix
alerting, CLI (`run.py`), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics), demo databases, Docker/Grafana files and the
migration scripts are gone, about 17k lines. The Python environment is the
training stack (pandas, scikit-learn, SQLAlchemy/psycopg2): `src/ml`, the
backfill clients/stores (`src/hii_collector.py`, `src/rid_reservoir.py`)
and `src/ml/predict.py` as the reference implementation the Rust forecaster
is tested against. PostgreSQL only; the Python database tests run against
a throwaway schema (`TEST_DATABASE_URL`, as the Rust ones). One installer,
`scripts/install.sh`.
Decided before phase 3 (2026-10-18): Matrix stays and behaves like ntfy;
`/docs` stays; SQLite dev mode goes; the monthly retrain stays on the same
docs/STATION_MANAGEMENT_GUIDE.md
-241
View File
@@ -1,241 +0,0 @@
# 🏔️ Station Management Guide - Northern Thailand Ping River Monitor
## 🎯 **Overview**
The Northern Thailand Ping River Monitor now includes comprehensive station management capabilities, allowing you to dynamically add, update, and remove monitoring stations through the web API.
## 🌊 **Current Coverage**
The system currently monitors **16 water stations** along the Ping River Basin:
### **Upper Ping River (Chiang Mai Province)**
- **P.20** - Ban Chiang Dao (บ้านเชียงดาว)
- **P.75** - Ban Chai Lat (บ้านช่อแล)
- **P.92** - Ban Muang Aut (บ้านเมืองกึ๊ด)
- **P.4A** - Ban Mae Taeng (บ้านแม่แตง)
- **P.67** - Ban Tae (บ้านแม่แต)
- **P.21** - Ban Rim Tai (บ้านริมใต้)
- **P.103** - Ring Bridge 3 (สะพานวงแหวนรอบ 3)
### **Middle Ping River**
- **P.1** - Nawarat Bridge (สะพานนวรัฐ) - *Main reference station*
- **P.82** - Ban Sob win (บ้านสบวิน)
- **P.84** - Ban Panton (บ้านพันตน)
- **P.81** - Ban Pong (บ้านโป่ง)
- **P.5** - Tha Nang Bridge (สะพานท่านาง)
### **Lower Ping River**
- **P.77** - Baan Sop Mae Sapuord (บ้านสบแม่สะป๊วด)
- **P.87** - Ban Pa Sang (บ้านป่าซาง)
- **P.76** - Banb Mae I Hai (บ้านแม่อีไฮ)
- **P.85** - Baan Lai Kaew (บ้านหล่ายแก้ว)
## 🔧 **Station Management API**
### **List All Stations**
```bash
GET /stations
```
**Response:**
```json
[
{
"station_id": 1,
"station_code": "P.20",
"thai_name": "บ้านเชียงดาว",
"english_name": "Ban Chiang Dao",
"latitude": 19.36731448032191,
"longitude": 98.9688487015384,
"geohash": null,
"status": "active"
}
]
```
### **Get Specific Station**
```bash
GET /stations/{station_id}
```
### **Add New Station**
```bash
POST /stations
Content-Type: application/json
{
"station_code": "P.NEW",
"thai_name": "สถานีใหม่",
"english_name": "New Station",
"latitude": 18.7875,
"longitude": 99.0045,
"geohash": "w5q6uuhvfcfp25",
"status": "active"
}
```
### **Update Station Information**
```bash
PUT /stations/{station_id}
Content-Type: application/json
{
"thai_name": "ชื่อใหม่",
"english_name": "Updated Name",
"latitude": 18.8000,
"longitude": 99.0100
}
```
### **Delete Station**
```bash
DELETE /stations/{station_id}
```
## 🧪 **Testing Station Management**
Use the provided test script to verify station management functionality:
```bash
# Test all station management endpoints
python test_station_management.py
```
This will:
1. List existing stations
2. Create a test station
3. Retrieve station details
4. Update station information
5. Verify changes
6. Delete the test station
7. Confirm deletion
## 📊 **Station Data Model**
### **Required Fields**
- `station_code`: Unique identifier (e.g., "P.1", "P.20")
- `thai_name`: Thai language name
- `english_name`: English language name
### **Optional Fields**
- `latitude`: GPS latitude coordinate (-90 to 90)
- `longitude`: GPS longitude coordinate (-180 to 180)
- `geohash`: Geohash string for location
- `status`: Station status ("active", "inactive", "maintenance", "error")
### **Validation Rules**
- Station codes must be unique
- Latitude must be between -90 and 90
- Longitude must be between -180 and 180
- Names cannot be empty
- Status must be valid enum value
## 🌐 **Web Interface**
Access the station management interface through the web dashboard:
1. **Start the API server:**
```bash
python run.py --web-api
```
2. **Open your browser:**
- Dashboard: http://localhost:8000
- API Documentation: http://localhost:8000/docs
3. **Use the interactive API docs** to test station management endpoints
## 🔄 **Integration with Data Collection**
- **Dynamic Station Discovery**: New stations are automatically included in data collection
- **Real-time Updates**: Station information changes are reflected immediately
- **Data Continuity**: Historical data is preserved when updating station details
- **Error Handling**: Invalid stations are skipped during data collection
## 📍 **Geographic Coverage**
The Ping River Basin monitoring network covers:
- **Total Distance**: ~400 km from Chiang Dao to Nakhon Sawan
- **Elevation Range**: 300m to 1,200m above sea level
- **Catchment Area**: ~25,000 km²
- **Major Cities**: Chiang Mai, Lamphun, Tak, Nakhon Sawan
## 🚀 **Usage Examples**
### **Add a New Upstream Station**
```bash
curl -X POST "http://localhost:8000/stations" \
-H "Content-Type: application/json" \
-d '{
"station_code": "P.UPSTREAM",
"thai_name": "สถานีต้นน้ำ",
"english_name": "Upstream Station",
"latitude": 19.5000,
"longitude": 98.9000,
"status": "active"
}'
```
### **Update Station Coordinates**
```bash
curl -X PUT "http://localhost:8000/stations/1" \
-H "Content-Type: application/json" \
-d '{
"latitude": 19.3700,
"longitude": 98.9700
}'
```
### **Mark Station for Maintenance**
```bash
curl -X PUT "http://localhost:8000/stations/5" \
-H "Content-Type: application/json" \
-d '{
"status": "maintenance"
}'
```
## 🔒 **Best Practices**
### **Station Naming**
- Use consistent code format (P.XX)
- Include both Thai and English names
- Use descriptive location names
### **Coordinate Accuracy**
- Use high-precision GPS coordinates (6+ decimal places)
- Verify coordinates match actual station location
- Include geohash for efficient spatial queries
### **Status Management**
- Set status to "maintenance" during repairs
- Use "inactive" for temporarily offline stations
- Use "error" for stations with data quality issues
### **Data Integrity**
- Test new stations before adding to production
- Backup station configuration before major changes
- Monitor data quality after station updates
## 🎯 **Future Enhancements**
Planned improvements for station management:
1. **Bulk Operations** - Import/export multiple stations
2. **Station Groups** - Organize stations by river section
3. **Automated Validation** - GPS coordinate verification
4. **Historical Tracking** - Track station configuration changes
5. **Alert Integration** - Notifications for station status changes
6. **Map Interface** - Visual station management on interactive map
## 📞 **Support**
For station management issues:
1. Check the API documentation at `/docs`
2. Run the test script: `python test_station_management.py`
3. Review logs for error details
4. Verify station data format and validation rules
The station management system provides flexible control over your monitoring network while maintaining data integrity and system reliability.
docs/VICTORIAMETRICS_SETUP.md
-443
View File
@@ -1,443 +0,0 @@
# VictoriaMetrics Setup Guide for Thailand Water Monitor
This guide provides comprehensive instructions for setting up VictoriaMetrics as the time-series database backend for the Thailand Water Monitor.
## Why VictoriaMetrics?
VictoriaMetrics is an excellent choice for water monitoring data because:
- **High Performance**: Up to 10x faster than InfluxDB
- **Low Resource Usage**: Uses 10x less RAM than Prometheus
- **Better Compression**: 70x better compression than Prometheus
- **Prometheus Compatible**: Drop-in replacement for Prometheus
- **Easy to Deploy**: Single binary, no dependencies
- **Cost Effective**: Open source with commercial support available
## Quick Start
### 1. Environment Variables
Set these environment variables to configure VictoriaMetrics:
```bash
# Windows (PowerShell)
$env:DB_TYPE="victoriametrics"
$env:VM_HOST="localhost"
$env:VM_PORT="8428"
# Linux/Mac
export DB_TYPE=victoriametrics
export VM_HOST=localhost
export VM_PORT=8428
```
### 2. Start VictoriaMetrics with Docker
```bash
# Simple setup
docker run -d \
--name victoriametrics \
-p 8428:8428 \
-v victoria-metrics-data:/victoria-metrics-data \
victoriametrics/victoria-metrics:latest \
--storageDataPath=/victoria-metrics-data \
--retentionPeriod=2y \
--httpListenAddr=:8428
# Verify it's running
curl http://localhost:8428/health
```
### 3. Run the Water Monitor
```bash
python water_scraper_v3.py
```
### 4. Access Grafana Dashboard
```bash
# Start with Docker Compose (includes Grafana)
docker-compose -f docker-compose.victoriametrics.yml up -d
# Access Grafana at http://localhost:3000
# Username: admin
# Password: admin_password
```
## Production Setup
### Docker Compose Configuration
Use the provided `docker-compose.victoriametrics.yml` file:
```bash
# Start the complete stack
docker-compose -f docker-compose.victoriametrics.yml up -d
# Check status
docker-compose -f docker-compose.victoriametrics.yml ps
# View logs
docker-compose -f docker-compose.victoriametrics.yml logs -f
```
### Manual VictoriaMetrics Configuration
#### High-Performance Configuration
```bash
docker run -d \
--name victoriametrics \
-p 8428:8428 \
-v victoria-metrics-data:/victoria-metrics-data \
victoriametrics/victoria-metrics:latest \
--storageDataPath=/victoria-metrics-data \
--retentionPeriod=2y \
--httpListenAddr=:8428 \
--maxConcurrentInserts=32 \
--search.maxQueryDuration=60s \
--search.maxConcurrentRequests=16 \
--dedup.minScrapeInterval=30s \
--memory.allowedPercent=80 \
--loggerLevel=INFO \
--loggerFormat=json \
--search.maxSeries=1000000 \
--search.maxPointsPerTimeseries=100000
```
#### Configuration Parameters Explained
| Parameter | Description | Recommended Value |
|-----------|-------------|-------------------|
| `--storageDataPath` | Data storage directory | `/victoria-metrics-data` |
| `--retentionPeriod` | How long to keep data | `2y` (2 years) |
| `--httpListenAddr` | HTTP listen address | `:8428` |
| `--maxConcurrentInserts` | Max concurrent inserts | `32` |
| `--search.maxQueryDuration` | Max query duration | `60s` |
| `--search.maxConcurrentRequests` | Max concurrent queries | `16` |
| `--dedup.minScrapeInterval` | Deduplication interval | `30s` |
| `--memory.allowedPercent` | Max memory usage | `80` |
| `--loggerLevel` | Log level | `INFO` |
| `--search.maxSeries` | Max time series | `1000000` |
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: victoriametrics
spec:
replicas: 1
selector:
matchLabels:
app: victoriametrics
template:
metadata:
labels:
app: victoriametrics
spec:
containers:
- name: victoriametrics
image: victoriametrics/victoria-metrics:latest
ports:
- containerPort: 8428
args:
- --storageDataPath=/victoria-metrics-data
- --retentionPeriod=2y
- --httpListenAddr=:8428
- --maxConcurrentInserts=32
volumeMounts:
- name: storage
mountPath: /victoria-metrics-data
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2000m"
volumes:
- name: storage
persistentVolumeClaim:
claimName: victoriametrics-pvc
---
apiVersion: v1
kind: Service
metadata:
name: victoriametrics
spec:
selector:
app: victoriametrics
ports:
- port: 8428
targetPort: 8428
type: ClusterIP
```
## Data Queries
### HTTP API Queries
VictoriaMetrics provides a Prometheus-compatible HTTP API:
```bash
# Current water levels for all stations
curl "http://localhost:8428/api/v1/query?query=water_level"
# Water levels for specific station
curl "http://localhost:8428/api/v1/query?query=water_level{station_code=\"P.1\"}"
# Average discharge over last hour
curl "http://localhost:8428/api/v1/query?query=avg_over_time(water_discharge[1h])"
# High discharge alerts (>80%)
curl "http://localhost:8428/api/v1/query?query=water_discharge_percent>80"
# Time range query (last 6 hours)
START=$(date -d '6 hours ago' +%s)
END=$(date +%s)
curl "http://localhost:8428/api/v1/query_range?query=water_level&start=${START}&end=${END}&step=300"
```
### PromQL Examples
```promql
# Current water levels
water_level
# Water level trends (last 24h)
water_level[24h]
# Discharge rates by station
water_discharge{station_code="P.1"}
# Average discharge across all stations
avg(water_discharge)
# Stations with high discharge (>80%)
water_discharge_percent > 80
# Rate of change in water level
rate(water_level[5m])
# Maximum water level in last hour
max_over_time(water_level[1h])
# Stations with increasing water levels
increase(water_level[1h]) > 0
```
## Grafana Integration
### Data Source Configuration
1. **Add VictoriaMetrics as Prometheus Data Source**:
- URL: `http://localhost:8428` (or `http://victoriametrics:8428` in Docker)
- Access: Server (default)
- HTTP Method: POST
2. **Import Dashboard**:
- Use the provided `water-monitoring-dashboard.json`
- Or create custom dashboards with the queries above
### Dashboard Panels
The included dashboard provides:
- **Time Series**: Water levels and discharge over time
- **Table**: Current status of all stations
- **Pie Chart**: Discharge percentage distribution
- **Gauge**: Average discharge percentage
- **Variables**: Filter by station
## Monitoring and Maintenance
### Health Checks
```bash
# Check VictoriaMetrics health
curl http://localhost:8428/health
# Check metrics endpoint
curl http://localhost:8428/metrics
# Check configuration
curl http://localhost:8428/api/v1/status/config
```
### Performance Monitoring
```bash
# Query performance stats
curl http://localhost:8428/api/v1/status/tsdb
# Memory usage
curl http://localhost:8428/api/v1/status/runtime
# Active queries
curl http://localhost:8428/api/v1/status/active_queries
```
### Backup and Restore
```bash
# Create backup
docker exec victoriametrics /usr/bin/vmbackup \
-storageDataPath=/victoria-metrics-data \
-dst=fs:///backup/$(date +%Y%m%d)
# Restore from backup
docker exec victoriametrics /usr/bin/vmrestore \
-src=fs:///backup/20250724 \
-storageDataPath=/victoria-metrics-data
```
### Log Analysis
```bash
# View logs
docker logs victoriametrics
# Follow logs
docker logs -f victoriametrics
# Search for errors
docker logs victoriametrics 2>&1 | grep ERROR
```
## Troubleshooting
### Common Issues
1. **Connection Refused**:
```bash
# Check if VictoriaMetrics is running
docker ps | grep victoriametrics
# Check port binding
netstat -tlnp | grep 8428
```
2. **High Memory Usage**:
```bash
# Reduce memory limit
docker run ... --memory.allowedPercent=60 ...
```
3. **Slow Queries**:
```bash
# Increase query timeout
docker run ... --search.maxQueryDuration=120s ...
```
4. **Data Not Appearing**:
```bash
# Check if data is being written
curl "http://localhost:8428/api/v1/query?query=up"
# Check water monitor logs
tail -f water_monitor.log
```
### Performance Tuning
1. **For High Write Load**:
```bash
--maxConcurrentInserts=64
--insert.maxQueueDuration=60s
```
2. **For High Query Load**:
```bash
--search.maxConcurrentRequests=32
--search.maxQueryDuration=120s
```
3. **For Large Datasets**:
```bash
--search.maxSeries=10000000
--search.maxPointsPerTimeseries=1000000
```
## Security
### Authentication
VictoriaMetrics doesn't have built-in authentication. Use a reverse proxy:
```nginx
server {
listen 80;
server_name victoriametrics.example.com;
auth_basic "VictoriaMetrics";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass http://localhost:8428;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
### TLS/SSL
```bash
# Use nginx or traefik for TLS termination
# Or use VictoriaMetrics with TLS:
docker run ... \
-v /path/to/cert.pem:/cert.pem \
-v /path/to/key.pem:/key.pem \
victoriametrics/victoria-metrics:latest \
--tls \
--tlsCertFile=/cert.pem \
--tlsKeyFile=/key.pem
```
## Scaling
### Cluster Setup
For high availability and horizontal scaling:
```bash
# Start multiple VictoriaMetrics instances
docker run -d --name vm1 -p 8428:8428 victoriametrics/victoria-metrics:latest
docker run -d --name vm2 -p 8429:8428 victoriametrics/victoria-metrics:latest
# Use load balancer to distribute queries
# Use vminsert/vmselect/vmstorage for true clustering
```
### Resource Requirements
| Data Points/Hour | RAM | CPU | Storage/Day |
|------------------|-----|-----|-------------|
| 1,000 | 100MB | 0.1 CPU | 10MB |
| 10,000 | 500MB | 0.5 CPU | 100MB |
| 100,000 | 2GB | 1 CPU | 1GB |
| 1,000,000 | 8GB | 2 CPU | 10GB |
## Migration
### From InfluxDB
```bash
# Export from InfluxDB
influx -database water_monitoring -execute "SELECT * FROM water_data" -format csv > data.csv
# Import to VictoriaMetrics (convert to Prometheus format first)
# Use vmctl tool for migration
```
### From Prometheus
```bash
# Use vmctl for direct migration
vmctl prometheus --prom-snapshot=/path/to/prometheus/data --vm-addr=http://localhost:8428
```
This comprehensive setup guide should help you configure VictoriaMetrics for optimal performance with the Thailand Water Monitor system.