Northern-Thailand-Ping-River-Monitor/docs/GRAFANA_MATRIX_ALERTING.md

Grafana Matrix Alerting Setup

Overview

Configure Grafana to send water level alerts directly to Matrix channels when thresholds are exceeded.

Prerequisites

• Grafana instance with your PostgreSQL data source
• Matrix account and access token
• Matrix room for alerts

Step 1: Configure Matrix Contact Point

1. In Grafana, go to Alerting → Contact Points

2. Add new contact point:

   Name: matrix-water-alerts
   Integration: Webhook
   URL: https://matrix.org/_matrix/client/v3/rooms/!ROOM_ID:matrix.org/send/m.room.message
   HTTP Method: POST
   

3. Add Headers:

   Authorization: Bearer YOUR_MATRIX_ACCESS_TOKEN
   Content-Type: application/json
   

4. Message Template:

   {
     "msgtype": "m.text",
     "body": "🌊 WATER ALERT: {{ .CommonLabels.alertname }}\n\nStation: {{ .CommonLabels.station_code }}\nLevel: {{ .CommonAnnotations.water_level }}m\nStatus: {{ .CommonLabels.severity }}\n\nTime: {{ .CommonAnnotations.time }}"
   }
   

Step 2: Create Alert Rules

High Water Level Alert

Rule Name: high-water-level
Query: water_level > 6.0
Condition: IS ABOVE 6.0 FOR 5m
Labels:
  - severity: critical
  - station_code: {{ .station_code }}
Annotations:
  - water_level: {{ .water_level }}
  - summary: "Critical water level at {{ .station_code }}"

Low Water Level Alert

Rule Name: low-water-level
Query: water_level < 1.0
Condition: IS BELOW 1.0 FOR 10m
Labels:
  - severity: warning
  - station_code: {{ .station_code }}

Data Gap Alert

Rule Name: data-gap
Query: increase(measurements_total[1h]) == 0
Condition: IS EQUAL TO 0 FOR 30m
Labels:
  - severity: warning
  - issue: data-gap

Step 3: Matrix Setup

Get Matrix Access Token

curl -X POST https://matrix.org/_matrix/client/v3/login \
  -H "Content-Type: application/json" \
  -d '{
    "type": "m.login.password",
    "user": "your_username",
    "password": "your_password"
  }'

Create Alert Room

curl -X POST "https://matrix.org/_matrix/client/v3/createRoom" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Water Level Alerts - Northern Thailand",
    "topic": "Automated alerts for Ping River water monitoring",
    "preset": "trusted_private_chat"
  }'

Example Alert Queries

Critical Water Levels

# High water alert
water_level{station_code=~"P.1|P.4A|P.20"} > 6.0

# Dangerous discharge
discharge{station_code=~".*"} > 500

# Rapid level change
increase(water_level[15m]) > 0.5

System Health

# No data received
up{job="water-monitor"} == 0

# Old data
(time() - timestamp) > 7200

Alert Notification Format

Your Matrix messages will look like:

🌊 WATER ALERT: High Water Level

Station: P.1 (Chiang Mai)
Level: 6.2m (CRITICAL)
Discharge: 450 cms
Status: DANGER

Time: 2025-09-26 14:30:00
Trend: Rising (+0.3m in 30min)

📍 Location: 18.7883°N, 98.9853°E

Advanced Features

Escalation Rules

# Send to different rooms based on severity
- if: severity == "critical"
  receiver: matrix-emergency
- if: severity == "warning"
  receiver: matrix-alerts
- if: time_of_day() outside "08:00-20:00"
  receiver: matrix-night-duty

Rate Limiting

group_wait: 5m
group_interval: 10m
repeat_interval: 30m

Testing Alerts

1. Test Contact Point - Use Grafana's test button
2. Simulate Alert - Manually trigger with test data
3. Verify Matrix - Check message formatting and delivery

Troubleshooting

Common Issues

• 403 Forbidden: Check Matrix access token
• Room not found: Verify room ID format
• No alerts: Check query syntax and thresholds
• Spam: Configure proper grouping and intervals