Add comprehensive Matrix alerting system with Grafana integration
- Implement custom Python alerting system (src/alerting.py) with water level monitoring, data freshness checks, and Matrix notifications - Add complete Grafana Matrix alerting setup guide (docs/GRAFANA_MATRIX_SETUP.md) with webhook configuration, alert rules, and notification policies - Create Matrix quick start guide (docs/MATRIX_QUICK_START.md) for rapid deployment - Integrate alerting commands into main application (--alert-check, --alert-test) - Add Matrix configuration to environment variables (.env.example) - Update Makefile with alerting targets (alert-check, alert-test) - Enhance status command to show Matrix notification status - Support station-specific water level thresholds and escalation rules - Provide dual alerting approach: native Grafana alerts and custom Python system 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
12
.env.example
12
.env.example
@@ -82,6 +82,18 @@ SMTP_PORT=587
|
|||||||
SMTP_USERNAME=
|
SMTP_USERNAME=
|
||||||
SMTP_PASSWORD=
|
SMTP_PASSWORD=
|
||||||
|
|
||||||
|
# Matrix Alerting Configuration
|
||||||
|
MATRIX_HOMESERVER=https://matrix.org
|
||||||
|
MATRIX_ACCESS_TOKEN=
|
||||||
|
MATRIX_ROOM_ID=
|
||||||
|
|
||||||
|
# Grafana Integration
|
||||||
|
GRAFANA_URL=http://localhost:3000
|
||||||
|
|
||||||
|
# Alert Configuration
|
||||||
|
ALERT_MAX_AGE_HOURS=2
|
||||||
|
ALERT_CHECK_INTERVAL_MINUTES=15
|
||||||
|
|
||||||
# Development Settings
|
# Development Settings
|
||||||
DEBUG=false
|
DEBUG=false
|
||||||
DEVELOPMENT_MODE=false
|
DEVELOPMENT_MODE=false
|
12
Makefile
12
Makefile
@@ -21,6 +21,11 @@ help:
|
|||||||
@echo " run Run the monitor in continuous mode"
|
@echo " run Run the monitor in continuous mode"
|
||||||
@echo " run-api Run the web API server"
|
@echo " run-api Run the web API server"
|
||||||
@echo " run-test Run a single test cycle"
|
@echo " run-test Run a single test cycle"
|
||||||
|
@echo " run-status Show system status"
|
||||||
|
@echo ""
|
||||||
|
@echo "Alerting:"
|
||||||
|
@echo " alert-check Check water levels and send alerts"
|
||||||
|
@echo " alert-test Send test Matrix message"
|
||||||
@echo ""
|
@echo ""
|
||||||
@echo "Distribution:"
|
@echo "Distribution:"
|
||||||
@echo " build-exe Build standalone executable"
|
@echo " build-exe Build standalone executable"
|
||||||
@@ -92,6 +97,13 @@ run-test:
|
|||||||
run-status:
|
run-status:
|
||||||
uv run python run.py --status
|
uv run python run.py --status
|
||||||
|
|
||||||
|
# Alerting
|
||||||
|
alert-check:
|
||||||
|
uv run python run.py --alert-check
|
||||||
|
|
||||||
|
alert-test:
|
||||||
|
uv run python run.py --alert-test
|
||||||
|
|
||||||
# Docker
|
# Docker
|
||||||
docker-build:
|
docker-build:
|
||||||
docker build -t ping-river-monitor .
|
docker build -t ping-river-monitor .
|
||||||
|
168
docs/GRAFANA_MATRIX_ALERTING.md
Normal file
168
docs/GRAFANA_MATRIX_ALERTING.md
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
# Grafana Matrix Alerting Setup
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Grafana instance with your PostgreSQL data source
|
||||||
|
- Matrix account and access token
|
||||||
|
- Matrix room for alerts
|
||||||
|
|
||||||
|
## Step 1: Configure Matrix Contact Point
|
||||||
|
|
||||||
|
1. **In Grafana, go to Alerting → Contact Points**
|
||||||
|
2. **Add new contact point:**
|
||||||
|
```
|
||||||
|
Name: matrix-water-alerts
|
||||||
|
Integration: Webhook
|
||||||
|
URL: https://matrix.org/_matrix/client/v3/rooms/!ROOM_ID:matrix.org/send/m.room.message
|
||||||
|
HTTP Method: POST
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Add Headers:**
|
||||||
|
```
|
||||||
|
Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
|
||||||
|
Content-Type: application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Message Template:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"msgtype": "m.text",
|
||||||
|
"body": "🌊 WATER ALERT: {{ .CommonLabels.alertname }}\n\nStation: {{ .CommonLabels.station_code }}\nLevel: {{ .CommonAnnotations.water_level }}m\nStatus: {{ .CommonLabels.severity }}\n\nTime: {{ .CommonAnnotations.time }}"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 2: Create Alert Rules
|
||||||
|
|
||||||
|
### High Water Level Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: high-water-level
|
||||||
|
Query: water_level > 6.0
|
||||||
|
Condition: IS ABOVE 6.0 FOR 5m
|
||||||
|
Labels:
|
||||||
|
- severity: critical
|
||||||
|
- station_code: {{ .station_code }}
|
||||||
|
Annotations:
|
||||||
|
- water_level: {{ .water_level }}
|
||||||
|
- summary: "Critical water level at {{ .station_code }}"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Low Water Level Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: low-water-level
|
||||||
|
Query: water_level < 1.0
|
||||||
|
Condition: IS BELOW 1.0 FOR 10m
|
||||||
|
Labels:
|
||||||
|
- severity: warning
|
||||||
|
- station_code: {{ .station_code }}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Gap Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: data-gap
|
||||||
|
Query: increase(measurements_total[1h]) == 0
|
||||||
|
Condition: IS EQUAL TO 0 FOR 30m
|
||||||
|
Labels:
|
||||||
|
- severity: warning
|
||||||
|
- issue: data-gap
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 3: Matrix Setup
|
||||||
|
|
||||||
|
### Get Matrix Access Token
|
||||||
|
```bash
|
||||||
|
curl -X POST https://matrix.org/_matrix/client/v3/login \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"type": "m.login.password",
|
||||||
|
"user": "your_username",
|
||||||
|
"password": "your_password"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create Alert Room
|
||||||
|
```bash
|
||||||
|
curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
|
||||||
|
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"name": "Water Level Alerts - Northern Thailand",
|
||||||
|
"topic": "Automated alerts for Ping River water monitoring",
|
||||||
|
"preset": "trusted_private_chat"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Alert Queries
|
||||||
|
|
||||||
|
### Critical Water Levels
|
||||||
|
```promql
|
||||||
|
# High water alert
|
||||||
|
water_level{station_code=~"P.1|P.4A|P.20"} > 6.0
|
||||||
|
|
||||||
|
# Dangerous discharge
|
||||||
|
discharge{station_code=~".*"} > 500
|
||||||
|
|
||||||
|
# Rapid level change
|
||||||
|
increase(water_level[15m]) > 0.5
|
||||||
|
```
|
||||||
|
|
||||||
|
### System Health
|
||||||
|
```promql
|
||||||
|
# No data received
|
||||||
|
up{job="water-monitor"} == 0
|
||||||
|
|
||||||
|
# Old data
|
||||||
|
(time() - timestamp) > 7200
|
||||||
|
```
|
||||||
|
|
||||||
|
## Alert Notification Format
|
||||||
|
|
||||||
|
Your Matrix messages will look like:
|
||||||
|
```
|
||||||
|
🌊 WATER ALERT: High Water Level
|
||||||
|
|
||||||
|
Station: P.1 (Chiang Mai)
|
||||||
|
Level: 6.2m (CRITICAL)
|
||||||
|
Discharge: 450 cms
|
||||||
|
Status: DANGER
|
||||||
|
|
||||||
|
Time: 2025-09-26 14:30:00
|
||||||
|
Trend: Rising (+0.3m in 30min)
|
||||||
|
|
||||||
|
📍 Location: 18.7883°N, 98.9853°E
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced Features
|
||||||
|
|
||||||
|
### Escalation Rules
|
||||||
|
```yaml
|
||||||
|
# Send to different rooms based on severity
|
||||||
|
- if: severity == "critical"
|
||||||
|
receiver: matrix-emergency
|
||||||
|
- if: severity == "warning"
|
||||||
|
receiver: matrix-alerts
|
||||||
|
- if: time_of_day() outside "08:00-20:00"
|
||||||
|
receiver: matrix-night-duty
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rate Limiting
|
||||||
|
```yaml
|
||||||
|
group_wait: 5m
|
||||||
|
group_interval: 10m
|
||||||
|
repeat_interval: 30m
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Alerts
|
||||||
|
|
||||||
|
1. **Test Contact Point** - Use Grafana's test button
|
||||||
|
2. **Simulate Alert** - Manually trigger with test data
|
||||||
|
3. **Verify Matrix** - Check message formatting and delivery
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
- **403 Forbidden**: Check Matrix access token
|
||||||
|
- **Room not found**: Verify room ID format
|
||||||
|
- **No alerts**: Check query syntax and thresholds
|
||||||
|
- **Spam**: Configure proper grouping and intervals
|
351
docs/GRAFANA_MATRIX_SETUP.md
Normal file
351
docs/GRAFANA_MATRIX_SETUP.md
Normal file
@@ -0,0 +1,351 @@
|
|||||||
|
# Complete Grafana Matrix Alerting Setup Guide
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Grafana instance running (v8.0+)
|
||||||
|
- PostgreSQL data source configured in Grafana
|
||||||
|
- Matrix account
|
||||||
|
- Matrix room for alerts
|
||||||
|
|
||||||
|
## Step 1: Get Matrix Access Token
|
||||||
|
|
||||||
|
### Method 1: Using curl
|
||||||
|
```bash
|
||||||
|
curl -X POST https://matrix.org/_matrix/client/v3/login \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"type": "m.login.password",
|
||||||
|
"user": "your_username",
|
||||||
|
"password": "your_password"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Method 2: Using Element Web Client
|
||||||
|
1. Open Element in browser: https://app.element.io
|
||||||
|
2. Login to your account
|
||||||
|
3. Go to Settings → Help & About → Advanced
|
||||||
|
4. Copy your Access Token
|
||||||
|
|
||||||
|
### Method 3: Using Matrix Admin Panel
|
||||||
|
- If you have admin access to your homeserver, generate token via admin API
|
||||||
|
|
||||||
|
## Step 2: Create Alert Room
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
|
||||||
|
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"name": "Water Level Alerts - Northern Thailand",
|
||||||
|
"topic": "Automated alerts for Ping River water monitoring",
|
||||||
|
"preset": "private_chat"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Save the `room_id` from the response (format: !roomid:homeserver.com)
|
||||||
|
|
||||||
|
## Step 3: Configure Grafana Contact Point
|
||||||
|
|
||||||
|
### Navigate to Alerting
|
||||||
|
1. In Grafana, go to **Alerting → Contact Points**
|
||||||
|
2. Click **Add contact point**
|
||||||
|
|
||||||
|
### Contact Point Settings
|
||||||
|
```
|
||||||
|
Name: matrix-water-alerts
|
||||||
|
Integration: Webhook
|
||||||
|
URL: https://matrix.org/_matrix/client/v3/rooms/!YOUR_ROOM_ID:matrix.org/send/m.room.message/{{ .GroupLabels.alertname }}_{{ .GroupLabels.severity }}_{{ now.Unix }}
|
||||||
|
HTTP Method: POST
|
||||||
|
```
|
||||||
|
|
||||||
|
### Headers
|
||||||
|
```
|
||||||
|
Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
|
||||||
|
Content-Type: application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Message Template (JSON Body)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"msgtype": "m.text",
|
||||||
|
"body": "🌊 **PING RIVER WATER ALERT**\n\n**Alert:** {{ .GroupLabels.alertname }}\n**Severity:** {{ .GroupLabels.severity | toUpper }}\n**Station:** {{ .GroupLabels.station_code }} ({{ .GroupLabels.station_name }})\n\n{{ range .Alerts }}**Status:** {{ .Status | toUpper }}\n**Water Level:** {{ .Annotations.water_level }}m\n**Threshold:** {{ .Annotations.threshold }}m\n**Time:** {{ .StartsAt.Format \"2006-01-02 15:04:05\" }}\n{{ if .Annotations.discharge }}**Discharge:** {{ .Annotations.discharge }} cms\n{{ end }}{{ if .Annotations.message }}**Details:** {{ .Annotations.message }}\n{{ end }}{{ end }}\n📈 **Dashboard:** {{ .ExternalURL }}\n📍 **Location:** Northern Thailand Ping River"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 4: Create Alert Rules
|
||||||
|
|
||||||
|
### High Water Level Alert
|
||||||
|
```yaml
|
||||||
|
# Rule Configuration
|
||||||
|
Rule Name: high-water-level
|
||||||
|
Evaluation Group: water-level-alerts
|
||||||
|
Folder: Water Monitoring
|
||||||
|
|
||||||
|
# Query A
|
||||||
|
SELECT
|
||||||
|
station_code,
|
||||||
|
station_name_th as station_name,
|
||||||
|
water_level,
|
||||||
|
discharge,
|
||||||
|
timestamp
|
||||||
|
FROM water_measurements
|
||||||
|
WHERE
|
||||||
|
timestamp > now() - interval '5 minutes'
|
||||||
|
AND water_level > 6.0
|
||||||
|
|
||||||
|
# Condition
|
||||||
|
IS ABOVE 6.0 FOR 5 minutes
|
||||||
|
|
||||||
|
# Labels
|
||||||
|
severity: critical
|
||||||
|
alertname: High Water Level
|
||||||
|
station_code: {{ $labels.station_code }}
|
||||||
|
station_name: {{ $labels.station_name }}
|
||||||
|
|
||||||
|
# Annotations
|
||||||
|
water_level: {{ $values.water_level }}
|
||||||
|
threshold: 6.0
|
||||||
|
discharge: {{ $values.discharge }}
|
||||||
|
summary: Critical water level detected at {{ $labels.station_code }}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Emergency Water Level Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: emergency-water-level
|
||||||
|
Query: water_level > 8.0
|
||||||
|
Condition: IS ABOVE 8.0 FOR 2 minutes
|
||||||
|
Labels:
|
||||||
|
severity: emergency
|
||||||
|
alertname: Emergency Water Level
|
||||||
|
Annotations:
|
||||||
|
threshold: 8.0
|
||||||
|
message: IMMEDIATE ACTION REQUIRED - Flood risk imminent
|
||||||
|
```
|
||||||
|
|
||||||
|
### Low Water Level Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: low-water-level
|
||||||
|
Query: water_level < 1.0
|
||||||
|
Condition: IS BELOW 1.0 FOR 15 minutes
|
||||||
|
Labels:
|
||||||
|
severity: warning
|
||||||
|
alertname: Low Water Level
|
||||||
|
Annotations:
|
||||||
|
threshold: 1.0
|
||||||
|
message: Drought conditions detected
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Gap Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: data-gap
|
||||||
|
Query:
|
||||||
|
SELECT
|
||||||
|
station_code,
|
||||||
|
MAX(timestamp) as last_seen
|
||||||
|
FROM water_measurements
|
||||||
|
GROUP BY station_code
|
||||||
|
HAVING MAX(timestamp) < now() - interval '2 hours'
|
||||||
|
|
||||||
|
Condition: HAS NO DATA FOR 30 minutes
|
||||||
|
Labels:
|
||||||
|
severity: warning
|
||||||
|
alertname: Data Gap
|
||||||
|
issue: missing-data
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rapid Level Change Alert
|
||||||
|
```yaml
|
||||||
|
Rule Name: rapid-level-change
|
||||||
|
Query:
|
||||||
|
SELECT
|
||||||
|
station_code,
|
||||||
|
water_level,
|
||||||
|
LAG(water_level, 1) OVER (PARTITION BY station_code ORDER BY timestamp) as prev_level
|
||||||
|
FROM water_measurements
|
||||||
|
WHERE timestamp > now() - interval '15 minutes'
|
||||||
|
HAVING ABS(water_level - prev_level) > 0.5
|
||||||
|
|
||||||
|
Condition: CHANGE > 0.5m FOR 1 minute
|
||||||
|
Labels:
|
||||||
|
severity: warning
|
||||||
|
alertname: Rapid Water Level Change
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 5: Configure Notification Policy
|
||||||
|
|
||||||
|
### Create Notification Policy
|
||||||
|
```yaml
|
||||||
|
# Policy Tree
|
||||||
|
- receiver: matrix-water-alerts
|
||||||
|
match:
|
||||||
|
severity: emergency|critical
|
||||||
|
group_wait: 10s
|
||||||
|
group_interval: 5m
|
||||||
|
repeat_interval: 30m
|
||||||
|
|
||||||
|
- receiver: matrix-water-alerts
|
||||||
|
match:
|
||||||
|
severity: warning
|
||||||
|
group_wait: 30s
|
||||||
|
group_interval: 10m
|
||||||
|
repeat_interval: 2h
|
||||||
|
```
|
||||||
|
|
||||||
|
### Grouping Rules
|
||||||
|
```yaml
|
||||||
|
group_by: [alertname, station_code]
|
||||||
|
group_wait: 10s
|
||||||
|
group_interval: 5m
|
||||||
|
repeat_interval: 1h
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 6: Station-Specific Thresholds
|
||||||
|
|
||||||
|
Create separate rules for each station with appropriate thresholds:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- P.1 (Chiang Mai) - Urban area, higher thresholds
|
||||||
|
SELECT * FROM water_measurements
|
||||||
|
WHERE station_code = 'P.1' AND water_level > 6.5
|
||||||
|
|
||||||
|
-- P.4A (Mae Ping) - Agricultural area
|
||||||
|
SELECT * FROM water_measurements
|
||||||
|
WHERE station_code = 'P.4A' AND water_level > 5.0
|
||||||
|
|
||||||
|
-- P.20 (Downstream) - Lower threshold
|
||||||
|
SELECT * FROM water_measurements
|
||||||
|
WHERE station_code = 'P.20' AND water_level > 4.0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 7: Advanced Features
|
||||||
|
|
||||||
|
### Time-Based Routing
|
||||||
|
```yaml
|
||||||
|
# Different receivers for day/night
|
||||||
|
time_intervals:
|
||||||
|
- name: working_hours
|
||||||
|
time_intervals:
|
||||||
|
- times:
|
||||||
|
- start_time: '08:00'
|
||||||
|
end_time: '20:00'
|
||||||
|
weekdays: ['monday:friday']
|
||||||
|
|
||||||
|
routes:
|
||||||
|
- receiver: matrix-alerts-day
|
||||||
|
match:
|
||||||
|
severity: warning
|
||||||
|
active_time_intervals: [working_hours]
|
||||||
|
|
||||||
|
- receiver: matrix-alerts-night
|
||||||
|
match:
|
||||||
|
severity: warning
|
||||||
|
active_time_intervals: ['!working_hours']
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-Channel Alerts
|
||||||
|
```yaml
|
||||||
|
# Send critical alerts to multiple rooms
|
||||||
|
- receiver: matrix-emergency
|
||||||
|
webhook_configs:
|
||||||
|
- url: https://matrix.org/_matrix/client/v3/rooms/!emergency:matrix.org/send/m.room.message
|
||||||
|
http_config:
|
||||||
|
authorization:
|
||||||
|
credentials: "Bearer EMERGENCY_TOKEN"
|
||||||
|
- url: https://matrix.org/_matrix/client/v3/rooms/!general:matrix.org/send/m.room.message
|
||||||
|
http_config:
|
||||||
|
authorization:
|
||||||
|
credentials: "Bearer GENERAL_TOKEN"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 8: Testing
|
||||||
|
|
||||||
|
### Test Contact Point
|
||||||
|
1. Go to Contact Points in Grafana
|
||||||
|
2. Select your Matrix contact point
|
||||||
|
3. Click "Test" button
|
||||||
|
4. Check Matrix room for test message
|
||||||
|
|
||||||
|
### Test Alert Rules
|
||||||
|
1. Temporarily lower thresholds
|
||||||
|
2. Wait for condition to trigger
|
||||||
|
3. Verify alert appears in Grafana
|
||||||
|
4. Verify Matrix message received
|
||||||
|
5. Reset thresholds
|
||||||
|
|
||||||
|
### Manual Alert Trigger
|
||||||
|
```bash
|
||||||
|
# Simulate high water level in database
|
||||||
|
INSERT INTO water_measurements (station_code, water_level, timestamp)
|
||||||
|
VALUES ('P.1', 7.5, NOW());
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
#### 403 Forbidden
|
||||||
|
- **Cause**: Invalid Matrix access token
|
||||||
|
- **Fix**: Regenerate token or check permissions
|
||||||
|
|
||||||
|
#### Room Not Found
|
||||||
|
- **Cause**: Incorrect room ID format
|
||||||
|
- **Fix**: Ensure room ID starts with ! and includes homeserver
|
||||||
|
|
||||||
|
#### No Alerts Firing
|
||||||
|
- **Cause**: Query returns no results
|
||||||
|
- **Fix**: Test queries in Grafana Explore, check data availability
|
||||||
|
|
||||||
|
#### Alert Spam
|
||||||
|
- **Cause**: No grouping configured
|
||||||
|
- **Fix**: Configure proper group_by and intervals
|
||||||
|
|
||||||
|
#### Messages Not Formatted
|
||||||
|
- **Cause**: Template syntax errors
|
||||||
|
- **Fix**: Validate JSON template, check Grafana template docs
|
||||||
|
|
||||||
|
### Debug Steps
|
||||||
|
1. Check Grafana alert rule status
|
||||||
|
2. Verify contact point test succeeds
|
||||||
|
3. Check Grafana logs: `/var/log/grafana/grafana.log`
|
||||||
|
4. Test Matrix API directly with curl
|
||||||
|
5. Verify database connectivity and query results
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
Add to your `.env`:
|
||||||
|
```bash
|
||||||
|
MATRIX_HOMESERVER=https://matrix.org
|
||||||
|
MATRIX_ACCESS_TOKEN=your_access_token_here
|
||||||
|
MATRIX_ROOM_ID=!your_room_id:matrix.org
|
||||||
|
GRAFANA_URL=http://your-grafana-host:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Alert Message
|
||||||
|
Your Matrix messages will appear as:
|
||||||
|
```
|
||||||
|
🌊 **PING RIVER WATER ALERT**
|
||||||
|
|
||||||
|
**Alert:** High Water Level
|
||||||
|
**Severity:** CRITICAL
|
||||||
|
**Station:** P.1 (สถานีเชียงใหม่)
|
||||||
|
|
||||||
|
**Status:** FIRING
|
||||||
|
**Water Level:** 6.75m
|
||||||
|
**Threshold:** 6.0m
|
||||||
|
**Time:** 2025-09-26 14:30:00
|
||||||
|
**Discharge:** 450.2 cms
|
||||||
|
|
||||||
|
📈 **Dashboard:** http://grafana:3000
|
||||||
|
📍 **Location:** Northern Thailand Ping River
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
- Store Matrix tokens securely (environment variables)
|
||||||
|
- Use room-specific tokens when possible
|
||||||
|
- Enable rate limiting to prevent spam
|
||||||
|
- Consider using dedicated alerting user account
|
||||||
|
- Regularly rotate access tokens
|
||||||
|
|
||||||
|
This setup provides comprehensive water level monitoring with immediate Matrix notifications when thresholds are exceeded.
|
85
docs/MATRIX_QUICK_START.md
Normal file
85
docs/MATRIX_QUICK_START.md
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
# Quick Matrix Alerting Setup
|
||||||
|
|
||||||
|
## Step 1: Get Matrix Account
|
||||||
|
1. Go to https://app.element.io or install Element app
|
||||||
|
2. Create account or login with existing Matrix account
|
||||||
|
|
||||||
|
## Step 2: Get Access Token
|
||||||
|
|
||||||
|
### Method 1: Element Web (Recommended)
|
||||||
|
1. Open Element in browser: https://app.element.io
|
||||||
|
2. Login to your account
|
||||||
|
3. Click Settings (gear icon) → Help & About → Advanced
|
||||||
|
4. Copy your "Access Token" (starts with `syt_...` or similar)
|
||||||
|
|
||||||
|
### Method 2: Command Line
|
||||||
|
```bash
|
||||||
|
curl -X POST https://matrix.org/_matrix/client/v3/login \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"type": "m.login.password",
|
||||||
|
"user": "your_username",
|
||||||
|
"password": "your_password"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 3: Create Alert Room
|
||||||
|
1. In Element, click "+" to create new room
|
||||||
|
2. Name: "Water Level Alerts"
|
||||||
|
3. Set to Private
|
||||||
|
4. Copy the room ID from room settings (format: `!roomid:matrix.org`)
|
||||||
|
|
||||||
|
## Step 4: Configure .env File
|
||||||
|
Add these to your `.env` file:
|
||||||
|
```bash
|
||||||
|
# Matrix Alerting Configuration
|
||||||
|
MATRIX_HOMESERVER=https://matrix.org
|
||||||
|
MATRIX_ACCESS_TOKEN=syt_your_access_token_here
|
||||||
|
MATRIX_ROOM_ID=!your_room_id:matrix.org
|
||||||
|
|
||||||
|
# Grafana Integration (optional)
|
||||||
|
GRAFANA_URL=http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 5: Test Configuration
|
||||||
|
```bash
|
||||||
|
# Test Matrix connection
|
||||||
|
uv run python run.py --alert-test
|
||||||
|
|
||||||
|
# Check system status (shows Matrix config)
|
||||||
|
uv run python run.py --status
|
||||||
|
|
||||||
|
# Run alert check
|
||||||
|
uv run python run.py --alert-check
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Alert Message
|
||||||
|
When thresholds are exceeded, you'll receive messages like:
|
||||||
|
```
|
||||||
|
🌊 **WATER LEVEL ALERT**
|
||||||
|
|
||||||
|
**Station:** P.1 (สถานีเชียงใหม่)
|
||||||
|
**Alert Type:** Critical Water Level
|
||||||
|
**Severity:** CRITICAL
|
||||||
|
|
||||||
|
**Current Level:** 6.75m
|
||||||
|
**Threshold:** 6.0m
|
||||||
|
**Difference:** +0.75m
|
||||||
|
**Discharge:** 450.2 cms
|
||||||
|
|
||||||
|
**Time:** 2025-09-26 14:30:00
|
||||||
|
|
||||||
|
📈 View dashboard: http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cron Job Setup (Optional)
|
||||||
|
Add to crontab for automatic alerting:
|
||||||
|
```bash
|
||||||
|
# Check water levels every 15 minutes
|
||||||
|
*/15 * * * * cd /path/to/monitor && uv run python run.py --alert-check >> alerts.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
- **403 Error**: Check Matrix access token is valid
|
||||||
|
- **Room Not Found**: Verify room ID includes `!` prefix and `:homeserver.com` suffix
|
||||||
|
- **No Alerts**: Check database has recent data with `uv run python run.py --status`
|
334
src/alerting.py
Normal file
334
src/alerting.py
Normal file
@@ -0,0 +1,334 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Water Level Alerting System with Matrix Integration
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
import requests
|
||||||
|
import datetime
|
||||||
|
from typing import List, Dict, Optional
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from enum import Enum
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .config import Config
|
||||||
|
from .database_adapters import create_database_adapter
|
||||||
|
from .logging_config import get_logger
|
||||||
|
except ImportError:
|
||||||
|
from config import Config
|
||||||
|
from database_adapters import create_database_adapter
|
||||||
|
import logging
|
||||||
|
def get_logger(name):
|
||||||
|
return logging.getLogger(name)
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
class AlertLevel(Enum):
|
||||||
|
INFO = "info"
|
||||||
|
WARNING = "warning"
|
||||||
|
CRITICAL = "critical"
|
||||||
|
EMERGENCY = "emergency"
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class WaterAlert:
|
||||||
|
station_code: str
|
||||||
|
station_name: str
|
||||||
|
alert_type: str
|
||||||
|
level: AlertLevel
|
||||||
|
water_level: float
|
||||||
|
threshold: float
|
||||||
|
discharge: Optional[float] = None
|
||||||
|
timestamp: Optional[datetime.datetime] = None
|
||||||
|
message: Optional[str] = None
|
||||||
|
|
||||||
|
class MatrixNotifier:
|
||||||
|
def __init__(self, homeserver: str, access_token: str, room_id: str):
|
||||||
|
self.homeserver = homeserver.rstrip('/')
|
||||||
|
self.access_token = access_token
|
||||||
|
self.room_id = room_id
|
||||||
|
self.session = requests.Session()
|
||||||
|
|
||||||
|
def send_message(self, message: str, msgtype: str = "m.text") -> bool:
|
||||||
|
"""Send message to Matrix room"""
|
||||||
|
try:
|
||||||
|
url = f"{self.homeserver}/_matrix/client/v3/rooms/{self.room_id}/send/m.room.message"
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {self.access_token}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
data = {
|
||||||
|
"msgtype": msgtype,
|
||||||
|
"body": message
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add transaction ID to prevent duplicates
|
||||||
|
txn_id = datetime.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
|
||||||
|
url += f"/{txn_id}"
|
||||||
|
|
||||||
|
response = self.session.post(url, headers=headers, json=data, timeout=10)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
logger.info(f"Matrix message sent successfully: {response.json().get('event_id')}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to send Matrix message: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def send_alert(self, alert: WaterAlert) -> bool:
|
||||||
|
"""Send formatted water alert to Matrix"""
|
||||||
|
emoji_map = {
|
||||||
|
AlertLevel.INFO: "ℹ️",
|
||||||
|
AlertLevel.WARNING: "⚠️",
|
||||||
|
AlertLevel.CRITICAL: "🚨",
|
||||||
|
AlertLevel.EMERGENCY: "🆘"
|
||||||
|
}
|
||||||
|
|
||||||
|
emoji = emoji_map.get(alert.level, "📊")
|
||||||
|
|
||||||
|
message = f"""{emoji} **WATER LEVEL ALERT**
|
||||||
|
|
||||||
|
**Station:** {alert.station_code} ({alert.station_name})
|
||||||
|
**Alert Type:** {alert.alert_type}
|
||||||
|
**Severity:** {alert.level.value.upper()}
|
||||||
|
|
||||||
|
**Current Level:** {alert.water_level:.2f}m
|
||||||
|
**Threshold:** {alert.threshold:.2f}m
|
||||||
|
**Difference:** {(alert.water_level - alert.threshold):+.2f}m
|
||||||
|
"""
|
||||||
|
|
||||||
|
if alert.discharge:
|
||||||
|
message += f"**Discharge:** {alert.discharge:.1f} cms\n"
|
||||||
|
|
||||||
|
if alert.timestamp:
|
||||||
|
message += f"**Time:** {alert.timestamp.strftime('%Y-%m-%d %H:%M:%S')}\n"
|
||||||
|
|
||||||
|
if alert.message:
|
||||||
|
message += f"\n**Details:** {alert.message}\n"
|
||||||
|
|
||||||
|
message += f"\n📈 View dashboard: {os.getenv('GRAFANA_URL', 'http://localhost:3000')}"
|
||||||
|
|
||||||
|
return self.send_message(message)
|
||||||
|
|
||||||
|
class WaterLevelAlertSystem:
|
||||||
|
def __init__(self):
|
||||||
|
self.db_adapter = None
|
||||||
|
self.matrix_notifier = None
|
||||||
|
self.thresholds = self._load_thresholds()
|
||||||
|
|
||||||
|
# Matrix configuration from environment
|
||||||
|
matrix_homeserver = os.getenv('MATRIX_HOMESERVER', 'https://matrix.org')
|
||||||
|
matrix_token = os.getenv('MATRIX_ACCESS_TOKEN')
|
||||||
|
matrix_room = os.getenv('MATRIX_ROOM_ID')
|
||||||
|
|
||||||
|
if matrix_token and matrix_room:
|
||||||
|
self.matrix_notifier = MatrixNotifier(matrix_homeserver, matrix_token, matrix_room)
|
||||||
|
logger.info("Matrix notifications enabled")
|
||||||
|
else:
|
||||||
|
logger.warning("Matrix configuration missing - notifications disabled")
|
||||||
|
|
||||||
|
def _load_thresholds(self) -> Dict[str, Dict[str, float]]:
|
||||||
|
"""Load alert thresholds from config or database"""
|
||||||
|
# Default thresholds for Northern Thailand stations
|
||||||
|
return {
|
||||||
|
"P.1": {"warning": 5.0, "critical": 6.5, "emergency": 8.0},
|
||||||
|
"P.4A": {"warning": 4.5, "critical": 6.0, "emergency": 7.5},
|
||||||
|
"P.20": {"warning": 3.0, "critical": 4.5, "emergency": 6.0},
|
||||||
|
"P.21": {"warning": 4.0, "critical": 5.5, "emergency": 7.0},
|
||||||
|
"P.67": {"warning": 6.0, "critical": 8.0, "emergency": 10.0},
|
||||||
|
"P.75": {"warning": 5.5, "critical": 7.5, "emergency": 9.5},
|
||||||
|
"P.103": {"warning": 7.0, "critical": 9.0, "emergency": 11.0},
|
||||||
|
# Default for unknown stations
|
||||||
|
"default": {"warning": 4.0, "critical": 6.0, "emergency": 8.0}
|
||||||
|
}
|
||||||
|
|
||||||
|
def connect_database(self):
|
||||||
|
"""Initialize database connection"""
|
||||||
|
try:
|
||||||
|
db_config = Config.get_database_config()
|
||||||
|
self.db_adapter = create_database_adapter(
|
||||||
|
db_config['type'],
|
||||||
|
connection_string=db_config['connection_string']
|
||||||
|
)
|
||||||
|
|
||||||
|
if self.db_adapter.connect():
|
||||||
|
logger.info("Database connection established for alerting")
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
logger.error("Failed to connect to database")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Database connection error: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def check_water_levels(self) -> List[WaterAlert]:
|
||||||
|
"""Check current water levels against thresholds"""
|
||||||
|
alerts = []
|
||||||
|
|
||||||
|
if not self.db_adapter:
|
||||||
|
logger.error("Database not connected")
|
||||||
|
return alerts
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Get latest measurements
|
||||||
|
measurements = self.db_adapter.get_latest_measurements(limit=50)
|
||||||
|
|
||||||
|
for measurement in measurements:
|
||||||
|
station_code = measurement.get('station_code', 'UNKNOWN')
|
||||||
|
water_level = measurement.get('water_level')
|
||||||
|
|
||||||
|
if not water_level:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Get thresholds for this station
|
||||||
|
station_thresholds = self.thresholds.get(station_code, self.thresholds['default'])
|
||||||
|
|
||||||
|
# Check each threshold level
|
||||||
|
alert_level = None
|
||||||
|
threshold_value = None
|
||||||
|
alert_type = None
|
||||||
|
|
||||||
|
if water_level >= station_thresholds['emergency']:
|
||||||
|
alert_level = AlertLevel.EMERGENCY
|
||||||
|
threshold_value = station_thresholds['emergency']
|
||||||
|
alert_type = "Emergency Water Level"
|
||||||
|
elif water_level >= station_thresholds['critical']:
|
||||||
|
alert_level = AlertLevel.CRITICAL
|
||||||
|
threshold_value = station_thresholds['critical']
|
||||||
|
alert_type = "Critical Water Level"
|
||||||
|
elif water_level >= station_thresholds['warning']:
|
||||||
|
alert_level = AlertLevel.WARNING
|
||||||
|
threshold_value = station_thresholds['warning']
|
||||||
|
alert_type = "High Water Level"
|
||||||
|
|
||||||
|
if alert_level:
|
||||||
|
alert = WaterAlert(
|
||||||
|
station_code=station_code,
|
||||||
|
station_name=measurement.get('station_name_th', f'Station {station_code}'),
|
||||||
|
alert_type=alert_type,
|
||||||
|
level=alert_level,
|
||||||
|
water_level=water_level,
|
||||||
|
threshold=threshold_value,
|
||||||
|
discharge=measurement.get('discharge'),
|
||||||
|
timestamp=measurement.get('timestamp')
|
||||||
|
)
|
||||||
|
alerts.append(alert)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error checking water levels: {e}")
|
||||||
|
|
||||||
|
return alerts
|
||||||
|
|
||||||
|
def check_data_freshness(self, max_age_hours: int = 2) -> List[WaterAlert]:
|
||||||
|
"""Check if data is fresh enough"""
|
||||||
|
alerts = []
|
||||||
|
|
||||||
|
if not self.db_adapter:
|
||||||
|
return alerts
|
||||||
|
|
||||||
|
try:
|
||||||
|
measurements = self.db_adapter.get_latest_measurements(limit=20)
|
||||||
|
cutoff_time = datetime.datetime.now() - datetime.timedelta(hours=max_age_hours)
|
||||||
|
|
||||||
|
for measurement in measurements:
|
||||||
|
timestamp = measurement.get('timestamp')
|
||||||
|
if timestamp and timestamp < cutoff_time:
|
||||||
|
station_code = measurement.get('station_code', 'UNKNOWN')
|
||||||
|
|
||||||
|
age_hours = (datetime.datetime.now() - timestamp).total_seconds() / 3600
|
||||||
|
|
||||||
|
alert = WaterAlert(
|
||||||
|
station_code=station_code,
|
||||||
|
station_name=measurement.get('station_name_th', f'Station {station_code}'),
|
||||||
|
alert_type="Stale Data",
|
||||||
|
level=AlertLevel.WARNING,
|
||||||
|
water_level=measurement.get('water_level', 0),
|
||||||
|
threshold=max_age_hours,
|
||||||
|
timestamp=timestamp,
|
||||||
|
message=f"No fresh data for {age_hours:.1f} hours"
|
||||||
|
)
|
||||||
|
alerts.append(alert)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error checking data freshness: {e}")
|
||||||
|
|
||||||
|
return alerts
|
||||||
|
|
||||||
|
def send_alerts(self, alerts: List[WaterAlert]) -> int:
|
||||||
|
"""Send alerts via configured channels"""
|
||||||
|
sent_count = 0
|
||||||
|
|
||||||
|
if not alerts:
|
||||||
|
return sent_count
|
||||||
|
|
||||||
|
if self.matrix_notifier:
|
||||||
|
for alert in alerts:
|
||||||
|
if self.matrix_notifier.send_alert(alert):
|
||||||
|
sent_count += 1
|
||||||
|
|
||||||
|
# Could add other notification channels here:
|
||||||
|
# - Email
|
||||||
|
# - Discord
|
||||||
|
# - Telegram
|
||||||
|
# - SMS
|
||||||
|
|
||||||
|
return sent_count
|
||||||
|
|
||||||
|
def run_alert_check(self) -> Dict[str, int]:
|
||||||
|
"""Run complete alert check cycle"""
|
||||||
|
if not self.connect_database():
|
||||||
|
return {"error": 1}
|
||||||
|
|
||||||
|
# Check water levels
|
||||||
|
water_alerts = self.check_water_levels()
|
||||||
|
|
||||||
|
# Check data freshness
|
||||||
|
data_alerts = self.check_data_freshness()
|
||||||
|
|
||||||
|
# Combine alerts
|
||||||
|
all_alerts = water_alerts + data_alerts
|
||||||
|
|
||||||
|
# Send alerts
|
||||||
|
sent_count = self.send_alerts(all_alerts)
|
||||||
|
|
||||||
|
logger.info(f"Alert check complete: {len(all_alerts)} alerts, {sent_count} sent")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"water_alerts": len(water_alerts),
|
||||||
|
"data_alerts": len(data_alerts),
|
||||||
|
"total_alerts": len(all_alerts),
|
||||||
|
"sent": sent_count
|
||||||
|
}
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Standalone alerting check"""
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description="Water Level Alert System")
|
||||||
|
parser.add_argument("--check", action="store_true", help="Run alert check")
|
||||||
|
parser.add_argument("--test", action="store_true", help="Send test message")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
alerting = WaterLevelAlertSystem()
|
||||||
|
|
||||||
|
if args.test:
|
||||||
|
if alerting.matrix_notifier:
|
||||||
|
test_message = "🧪 **Test Alert**\n\nThis is a test message from the Water Level Alert System.\n\nIf you received this, Matrix notifications are working correctly!"
|
||||||
|
success = alerting.matrix_notifier.send_message(test_message)
|
||||||
|
print(f"Test message sent: {success}")
|
||||||
|
else:
|
||||||
|
print("Matrix notifier not configured")
|
||||||
|
|
||||||
|
elif args.check:
|
||||||
|
results = alerting.run_alert_check()
|
||||||
|
print(f"Alert check results: {results}")
|
||||||
|
|
||||||
|
else:
|
||||||
|
print("Use --check or --test")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
91
src/main.py
91
src/main.py
@@ -180,6 +180,65 @@ def run_web_api():
|
|||||||
logger.error(f"Web API failed: {e}")
|
logger.error(f"Web API failed: {e}")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def run_alert_check():
|
||||||
|
"""Run water level alert check"""
|
||||||
|
logger.info("Running water level alert check...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .alerting import WaterLevelAlertSystem
|
||||||
|
|
||||||
|
# Initialize alerting system
|
||||||
|
alerting = WaterLevelAlertSystem()
|
||||||
|
|
||||||
|
# Run alert check
|
||||||
|
results = alerting.run_alert_check()
|
||||||
|
|
||||||
|
if 'error' in results:
|
||||||
|
logger.error("❌ Alert check failed due to database connection")
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info(f"✅ Alert check completed:")
|
||||||
|
logger.info(f" • Water level alerts: {results['water_alerts']}")
|
||||||
|
logger.info(f" • Data freshness alerts: {results['data_alerts']}")
|
||||||
|
logger.info(f" • Total alerts generated: {results['total_alerts']}")
|
||||||
|
logger.info(f" • Alerts sent: {results['sent']}")
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Alert check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def run_alert_test():
|
||||||
|
"""Send test alert message"""
|
||||||
|
logger.info("Sending test alert message...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .alerting import WaterLevelAlertSystem
|
||||||
|
|
||||||
|
# Initialize alerting system
|
||||||
|
alerting = WaterLevelAlertSystem()
|
||||||
|
|
||||||
|
if not alerting.matrix_notifier:
|
||||||
|
logger.error("❌ Matrix notifier not configured")
|
||||||
|
logger.info("Please set MATRIX_ACCESS_TOKEN and MATRIX_ROOM_ID in your .env file")
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Send test message
|
||||||
|
test_message = "🧪 **Test Alert**\n\nThis is a test message from the Northern Thailand Ping River Monitor.\n\nIf you received this, Matrix notifications are working correctly!"
|
||||||
|
success = alerting.matrix_notifier.send_message(test_message)
|
||||||
|
|
||||||
|
if success:
|
||||||
|
logger.info("✅ Test alert message sent successfully")
|
||||||
|
else:
|
||||||
|
logger.error("❌ Test alert message failed to send")
|
||||||
|
|
||||||
|
return success
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Test alert failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
def show_status():
|
def show_status():
|
||||||
"""Show current system status"""
|
"""Show current system status"""
|
||||||
logger.info("=== Northern Thailand Ping River Monitor Status ===")
|
logger.info("=== Northern Thailand Ping River Monitor Status ===")
|
||||||
@@ -210,6 +269,20 @@ def show_status():
|
|||||||
else:
|
else:
|
||||||
logger.error("❌ Database connection failed")
|
logger.error("❌ Database connection failed")
|
||||||
|
|
||||||
|
# Test alerting system
|
||||||
|
logger.info("\n=== Alerting System Status ===")
|
||||||
|
try:
|
||||||
|
from .alerting import WaterLevelAlertSystem
|
||||||
|
alerting = WaterLevelAlertSystem()
|
||||||
|
|
||||||
|
if alerting.matrix_notifier:
|
||||||
|
logger.info("✅ Matrix notifications configured")
|
||||||
|
else:
|
||||||
|
logger.warning("⚠️ Matrix notifications not configured")
|
||||||
|
logger.info("Set MATRIX_ACCESS_TOKEN and MATRIX_ROOM_ID in .env file")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Alerting system error: {e}")
|
||||||
|
|
||||||
# Show metrics if available
|
# Show metrics if available
|
||||||
metrics_collector = get_metrics_collector()
|
metrics_collector = get_metrics_collector()
|
||||||
metrics = metrics_collector.get_all_metrics()
|
metrics = metrics_collector.get_all_metrics()
|
||||||
@@ -239,6 +312,8 @@ Examples:
|
|||||||
%(prog)s --fill-gaps 7 # Fill missing data for last 7 days
|
%(prog)s --fill-gaps 7 # Fill missing data for last 7 days
|
||||||
%(prog)s --update-data 2 # Update existing data for last 2 days
|
%(prog)s --update-data 2 # Update existing data for last 2 days
|
||||||
%(prog)s --status # Show system status
|
%(prog)s --status # Show system status
|
||||||
|
%(prog)s --alert-check # Check water levels and send alerts
|
||||||
|
%(prog)s --alert-test # Send test Matrix message
|
||||||
"""
|
"""
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -274,6 +349,18 @@ Examples:
|
|||||||
help="Show current system status"
|
help="Show current system status"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--alert-check",
|
||||||
|
action="store_true",
|
||||||
|
help="Run water level alert check"
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--alert-test",
|
||||||
|
action="store_true",
|
||||||
|
help="Send test alert message to Matrix"
|
||||||
|
)
|
||||||
|
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--log-level",
|
"--log-level",
|
||||||
choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
|
choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
|
||||||
@@ -314,6 +401,10 @@ Examples:
|
|||||||
success = run_data_update(args.update_data)
|
success = run_data_update(args.update_data)
|
||||||
elif args.status:
|
elif args.status:
|
||||||
success = show_status()
|
success = show_status()
|
||||||
|
elif args.alert_check:
|
||||||
|
success = run_alert_check()
|
||||||
|
elif args.alert_test:
|
||||||
|
success = run_alert_test()
|
||||||
else:
|
else:
|
||||||
success = run_continuous_monitoring()
|
success = run_continuous_monitoring()
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user