B4L/Northern-Thailand-Ping-River-Monitor
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

chore: remove the Python service; Python is the training stack only

Browse Source 
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/conftest.py
+31 -2
View File
@@ -1,12 +1,41 @@
"""Shared pytest configuration.
Ensures the repository root is on sys.path so tests can import the ``src``
package regardless of the working directory pytest is invoked from.
Puts the repository root on sys.path so tests can import the ``src`` package
from any working directory, and provides ``pg_url``: a throwaway PostgreSQL
schema for the database tests (stores, loaders). They need
TEST_DATABASE_URL -- the variable the Rust tests use, e.g.
postgres://postgres:ping@localhost:55432/ping -- and are skipped without it.
"""
import os
import sys
import uuid
import pytest
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if REPO_ROOT not in sys.path:
sys.path.insert(0, REPO_ROOT)
@pytest.fixture
def pg_url():
"""Connection string whose search_path is a fresh schema, dropped after."""
base = os.environ.get("TEST_DATABASE_URL", "").strip()
if not base:
pytest.skip("TEST_DATABASE_URL not set (PostgreSQL test)")
from sqlalchemy import create_engine, text
if base.startswith("postgres://"):
base = "postgresql://" + base[len("postgres://") :]
schema = f"pytest_{uuid.uuid4().hex[:12]}"
admin = create_engine(base)
with admin.begin() as conn:
conn.execute(text(f'CREATE SCHEMA "{schema}"'))
sep = "&" if "?" in base else "?"
try:
yield f"{base}{sep}options=-csearch_path%3D{schema}"
finally:
with admin.begin() as conn:
conn.execute(text(f'DROP SCHEMA "{schema}" CASCADE'))
admin.dispose()
tests/test_alerting.py
-383
View File
@@ -1,383 +0,0 @@
#!/usr/bin/env python3
"""
Comprehensive tests for the alerting system
Tests both zone-based and rate-of-change alerts
"""
import sys
import os
import datetime
import sqlite3
import time
import gc
# Add src directory to path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
from src.alerting import WaterLevelAlertSystem, AlertLevel
from src.database_adapters import create_database_adapter
def setup_test_database(test_name='default'):
"""Create a test database with sample data"""
db_path = f'test_alerts_{test_name}.db'
# Remove existing test database
if os.path.exists(db_path):
try:
os.remove(db_path)
except PermissionError:
# If locked, use a different name with timestamp
import random
db_path = f'test_alerts_{test_name}_{random.randint(1000, 9999)}.db'
# Create new database
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# Create stations table
cursor.execute("""
CREATE TABLE stations (
id INTEGER PRIMARY KEY,
station_code TEXT NOT NULL UNIQUE,
english_name TEXT,
thai_name TEXT,
latitude REAL,
longitude REAL,
basin TEXT,
province TEXT,
status TEXT DEFAULT 'active',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""")
# Create water_measurements table
cursor.execute("""
CREATE TABLE water_measurements (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp DATETIME NOT NULL,
station_id INTEGER NOT NULL,
water_level REAL NOT NULL,
discharge REAL,
discharge_percent REAL,
status TEXT DEFAULT 'active',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (station_id) REFERENCES stations (id)
)
""")
# Insert P.1 station (id=8 to match existing data)
cursor.execute("""
INSERT INTO stations (id, station_code, english_name, thai_name, basin, province)
VALUES (8, 'P.1', 'Nawarat Bridge', 'สะพานนวรัฐ', 'Ping', 'Chiang Mai')
""")
conn.commit()
conn.close()
return db_path
def test_zone_level_alerts():
"""Test that zone-based alerts trigger correctly"""
print("="*70)
print("TEST 1: Zone-Based Water Level Alerts")
print("="*70)
db_path = setup_test_database('zone_tests')
# Test cases for P.1 zone thresholds
test_cases = [
(2.5, None, "Below all zones"),
(3.7, AlertLevel.INFO, "Zone 1"),
(3.9, AlertLevel.INFO, "Zone 2"),
(4.0, AlertLevel.WARNING, "Zone 3"),
(4.2, AlertLevel.WARNING, "Zone 5"),
(4.3, AlertLevel.CRITICAL, "Zone 6"),
(4.6, AlertLevel.CRITICAL, "Zone 7"),
(4.8, AlertLevel.EMERGENCY, "Zone 8/NewEdge"),
(5.0, AlertLevel.EMERGENCY, "Above all zones"),
]
print("\nTesting P.1 (Nawarat Bridge) zone thresholds:")
print("-" * 70)
passed = 0
failed = 0
for water_level, expected_level, zone_description in test_cases:
# Insert test data
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("DELETE FROM water_measurements")
current_time = datetime.datetime.now()
cursor.execute("""
INSERT INTO water_measurements (timestamp, station_id, water_level, discharge)
VALUES (?, 8, ?, 350.0)
""", (current_time, water_level))
conn.commit()
conn.close()
# Check alerts
alerting = WaterLevelAlertSystem()
alerting.db_adapter = create_database_adapter('sqlite', connection_string=f'sqlite:///{db_path}')
alerting.db_adapter.connect()
alerts = alerting.check_water_levels()
# Verify result
if expected_level is None:
# Should not trigger any alert
if len(alerts) == 0:
print(f"[PASS] {water_level:.1f}m: {zone_description} - No alert")
passed += 1
else:
print(f"[FAIL] {water_level:.1f}m: {zone_description} - Unexpected alert")
failed += 1
else:
# Should trigger alert with specific level
if len(alerts) > 0 and alerts[0].level == expected_level:
print(f"[PASS] {water_level:.1f}m: {zone_description} - {expected_level.value.upper()} alert")
passed += 1
elif len(alerts) == 0:
print(f"[FAIL] {water_level:.1f}m: {zone_description} - No alert triggered")
failed += 1
else:
print(f"[FAIL] {water_level:.1f}m: {zone_description} - Wrong alert level: {alerts[0].level.value}")
failed += 1
print("-" * 70)
print(f"Zone Alert Tests: {passed} passed, {failed} failed")
# Cleanup - force garbage collection and wait briefly before removing file
gc.collect()
time.sleep(0.5)
try:
os.remove(db_path)
except PermissionError:
print(f"Warning: Could not remove test database {db_path}")
return failed == 0
def test_rate_of_change_alerts():
"""Test that rate-of-change alerts trigger correctly"""
print("\n" + "="*70)
print("TEST 2: Rate-of-Change Water Level Alerts")
print("="*70)
db_path = setup_test_database('rate_tests')
# Test cases: (initial_level, final_level, hours_elapsed, expected_alert_level, description)
test_cases = [
(3.0, 3.1, 3.0, None, "Slow rise (0.03m/h)"),
(3.0, 3.5, 3.0, AlertLevel.WARNING, "Moderate rise (0.17m/h)"),
(3.0, 3.8, 3.0, AlertLevel.CRITICAL, "Rapid rise (0.27m/h)"),
(3.0, 4.2, 3.0, AlertLevel.EMERGENCY, "Very rapid rise (0.40m/h)"),
(4.0, 3.5, 3.0, None, "Falling water (negative rate)"),
]
print("\nTesting P.1 rate-of-change thresholds:")
print(" Warning: 0.15 m/h (15 cm/h)")
print(" Critical: 0.25 m/h (25 cm/h)")
print(" Emergency: 0.40 m/h (40 cm/h)")
print("-" * 70)
passed = 0
failed = 0
for initial_level, final_level, hours, expected_level, description in test_cases:
# Insert test data simulating water level change over time
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("DELETE FROM water_measurements")
current_time = datetime.datetime.now()
start_time = current_time - datetime.timedelta(hours=hours)
# Insert initial measurement
cursor.execute("""
INSERT INTO water_measurements (timestamp, station_id, water_level, discharge)
VALUES (?, 8, ?, 350.0)
""", (start_time, initial_level))
# Insert final measurement
cursor.execute("""
INSERT INTO water_measurements (timestamp, station_id, water_level, discharge)
VALUES (?, 8, ?, 380.0)
""", (current_time, final_level))
conn.commit()
conn.close()
# Check rate-of-change alerts
alerting = WaterLevelAlertSystem()
alerting.db_adapter = create_database_adapter('sqlite', connection_string=f'sqlite:///{db_path}')
alerting.db_adapter.connect()
rate_alerts = alerting.check_rate_of_change(lookback_hours=int(hours) + 1)
# Calculate actual rate for display
level_change = final_level - initial_level
rate = level_change / hours if hours > 0 else 0
# Verify result
if expected_level is None:
# Should not trigger any alert
if len(rate_alerts) == 0:
print(f"[PASS] {rate:+.2f}m/h: {description} - No alert")
passed += 1
else:
print(f"[FAIL] {rate:+.2f}m/h: {description} - Unexpected alert")
print(f" Alert: {rate_alerts[0].alert_type} - {rate_alerts[0].level.value}")
failed += 1
else:
# Should trigger alert with specific level
if len(rate_alerts) > 0 and rate_alerts[0].level == expected_level:
print(f"[PASS] {rate:+.2f}m/h: {description} - {expected_level.value.upper()} alert")
print(f" Message: {rate_alerts[0].message}")
passed += 1
elif len(rate_alerts) == 0:
print(f"[FAIL] {rate:+.2f}m/h: {description} - No alert triggered")
failed += 1
else:
print(f"[FAIL] {rate:+.2f}m/h: {description} - Wrong alert level: {rate_alerts[0].level.value}")
failed += 1
print("-" * 70)
print(f"Rate-of-Change Tests: {passed} passed, {failed} failed")
# Cleanup - force garbage collection and wait briefly before removing file
gc.collect()
time.sleep(0.5)
try:
os.remove(db_path)
except PermissionError:
print(f"Warning: Could not remove test database {db_path}")
return failed == 0
def test_combined_alerts():
"""Test scenario where both zone and rate-of-change alerts trigger"""
print("\n" + "="*70)
print("TEST 3: Combined Zone + Rate-of-Change Alerts")
print("="*70)
db_path = setup_test_database('combined_tests')
print("\nScenario: Water rising rapidly from 3.5m to 4.5m over 3 hours")
print(" Expected: Both Zone 7 alert AND Critical rate-of-change alert")
print("-" * 70)
# Insert test data
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
current_time = datetime.datetime.now()
start_time = current_time - datetime.timedelta(hours=3)
# Water rising from 3.5m to 4.5m over 3 hours (0.33 m/h - Critical rate)
cursor.execute("""
INSERT INTO water_measurements (timestamp, station_id, water_level, discharge)
VALUES (?, 8, 3.5, 350.0)
""", (start_time,))
cursor.execute("""
INSERT INTO water_measurements (timestamp, station_id, water_level, discharge)
VALUES (?, 8, 4.5, 450.0)
""", (current_time,))
conn.commit()
conn.close()
# Check both types of alerts
alerting = WaterLevelAlertSystem()
alerting.db_adapter = create_database_adapter('sqlite', connection_string=f'sqlite:///{db_path}')
alerting.db_adapter.connect()
zone_alerts = alerting.check_water_levels()
rate_alerts = alerting.check_rate_of_change(lookback_hours=4)
all_alerts = zone_alerts + rate_alerts
print(f"\nTotal alerts triggered: {len(all_alerts)}")
zone_alert_found = False
rate_alert_found = False
for alert in all_alerts:
print(f"\n Alert Type: {alert.alert_type}")
print(f" Severity: {alert.level.value.upper()}")
print(f" Water Level: {alert.water_level:.2f}m")
if alert.message:
print(f" Details: {alert.message}")
if "Zone" in alert.alert_type:
zone_alert_found = True
if "Rise" in alert.alert_type or "rate" in alert.alert_type.lower():
rate_alert_found = True
print("-" * 70)
if zone_alert_found and rate_alert_found:
print("[PASS] Combined Alert Test - Both alert types triggered")
success = True
else:
print("[FAIL] Combined Alert Test")
if not zone_alert_found:
print(" Missing: Zone-based alert")
if not rate_alert_found:
print(" Missing: Rate-of-change alert")
success = False
# Cleanup - force garbage collection and wait briefly before removing file
gc.collect()
time.sleep(0.5)
try:
os.remove(db_path)
except PermissionError:
print(f"Warning: Could not remove test database {db_path}")
return success
def main():
"""Run all alert tests"""
print("\n" + "="*70)
print("WATER LEVEL ALERTING SYSTEM - COMPREHENSIVE TESTS")
print("="*70)
results = []
# Run tests
results.append(("Zone-Based Alerts", test_zone_level_alerts()))
results.append(("Rate-of-Change Alerts", test_rate_of_change_alerts()))
results.append(("Combined Alerts", test_combined_alerts()))
# Summary
print("\n" + "="*70)
print("TEST SUMMARY")
print("="*70)
all_passed = True
for test_name, passed in results:
status = "PASS" if passed else "FAIL"
print(f"{test_name}: [{status}]")
if not passed:
all_passed = False
print("="*70)
if all_passed:
print("\nAll tests PASSED!")
return 0
else:
print("\nSome tests FAILED!")
return 1
if __name__ == "__main__":
sys.exit(main())
tests/test_api.py
-61
View File
@@ -1,61 +0,0 @@
#!/usr/bin/env python3
"""
Simple API test script
"""
import requests
import time
import json
def test_api_endpoints():
"""Test the main API endpoints"""
base_url = "http://localhost:8000"
endpoints = [
"/health",
"/metrics",
"/stations",
"/measurements/latest?limit=5",
"/scraping/status",
"/config"
]
print("🧪 Testing API endpoints...")
print("Make sure the API server is running: python run.py --web-api")
print()
for endpoint in endpoints:
try:
print(f"Testing {endpoint}...")
response = requests.get(f"{base_url}{endpoint}", timeout=5)
if response.status_code == 200:
data = response.json()
print(f"✅ {endpoint} - Status: {response.status_code}")
# Show some sample data
if endpoint == "/stations":
print(f" Found {len(data)} stations")
elif endpoint == "/measurements/latest?limit=5":
print(f" Found {len(data)} measurements")
if data:
latest = data[0]
print(f" Latest: {latest['station_code']} - {latest['water_level']}m")
elif endpoint == "/health":
print(f" Overall status: {data.get('overall_status', 'unknown')}")
elif endpoint == "/scraping/status":
print(f" Is running: {data.get('is_running', False)}")
print(f" Total runs: {data.get('total_runs', 0)}")
else:
print(f"❌ {endpoint} - Status: {response.status_code}")
except requests.exceptions.ConnectionError:
print(f"❌ {endpoint} - Connection failed (is the server running?)")
except Exception as e:
print(f"❌ {endpoint} - Error: {e}")
print()
if __name__ == "__main__":
test_api_endpoints()
tests/test_collection_disabled.py
-123
View File
@@ -1,123 +0,0 @@
"""COLLECTION_ENABLED=false: the Rust ping-collector collects, this app only
re-forecasts and notifies when the database's newest RID reading advances
(and once per clock hour), and /scraping/status reports the collector's file."""
import asyncio
import datetime
import json
from src import web_api
from src.config import Config
def test_forecasts_follow_new_readings_and_the_clock_hour(monkeypatch):
clock = {"now": datetime.datetime(2026, 10, 17, 21, 30)}
class FixedClock(datetime.datetime):
@classmethod
def now(cls, tz=None):
return clock["now"]
# newest reading per poll: unchanged, unchanged, advances, next hour
latest = iter(
[
datetime.datetime(2026, 10, 17, 20),
datetime.datetime(2026, 10, 17, 20),
datetime.datetime(2026, 10, 17, 21),
datetime.datetime(2026, 10, 17, 21),
]
)
calls = {"precompute": 0, "notify": 0}
sleeps = []
async def precompute():
calls["precompute"] += 1
async def notify():
calls["notify"] += 1
async def fake_sleep(seconds):
sleeps.append(seconds)
if len(sleeps) == 3: # the last poll happens in the next clock hour
clock["now"] = datetime.datetime(2026, 10, 17, 22, 1)
if len(sleeps) == 4:
raise asyncio.CancelledError
monkeypatch.setattr(web_api, "datetime", FixedClock)
monkeypatch.setattr(web_api.asyncio, "sleep", fake_sleep)
monkeypatch.setattr(web_api, "_latest_rid_reading", lambda: next(latest))
monkeypatch.setattr(web_api, "_precompute_forecasts", precompute)
monkeypatch.setattr(web_api, "_notify_transitions", notify)
monkeypatch.setattr(Config, "FORECAST_POLL_SECONDS", 60)
asyncio.run(web_api.forecast_notify_task())
# first poll (new + hourly), third (new reading), fourth (new clock hour)
assert calls == {"precompute": 3, "notify": 3}
assert sleeps == [60, 60, 60, 60]
def test_status_comes_from_the_collector_file(tmp_path, monkeypatch):
status = tmp_path / "collector-status.json"
status.write_text(
json.dumps(
{
"is_running": False,
"total_runs": 4,
"successful_runs": 3,
"failed_runs": 1,
"last_run": "2026-10-17T22:05:01",
"next_run": "2026-10-17T23:05:00",
"feed_current": True,
"latest_reading": "2026-10-17T22:00:00",
"readings_saved": 352,
"collector": "ping-collector 0.1.0",
}
)
)
monkeypatch.setattr(Config, "COLLECTION_ENABLED", False)
monkeypatch.setattr(Config, "SCRAPING_STATUS_FILE", str(status))
monkeypatch.setitem(web_api.app_state, "leader_lock", object())
resp = asyncio.run(web_api.get_scraping_status())
assert resp.total_runs == 4
assert resp.feed_current is True
assert resp.latest_reading == datetime.datetime(2026, 10, 17, 22)
def test_manual_trigger_is_refused_when_collection_runs_elsewhere(monkeypatch):
import pytest
from fastapi import BackgroundTasks, HTTPException
monkeypatch.setattr(Config, "COLLECTION_ENABLED", False)
with pytest.raises(HTTPException) as err:
asyncio.run(web_api.trigger_scraping(BackgroundTasks()))
assert err.value.status_code == 409
def test_stale_cache_older_than_max_stale_is_recomputed(monkeypatch):
"""/health must not answer with a result from hours ago."""
import asyncio
import threading
import time
from src import web_api
cache, lock, compute_lock = {}, threading.Lock(), threading.Lock()
cache["k"] = (time.monotonic() - 3600, "an hour old")
value, stale = asyncio.run(
web_api._cached_swr(
cache, lock, compute_lock, "k", 30, lambda: "fresh", max_stale=120
)
)
assert (value, stale) == ("fresh", False)
cache["k"] = (time.monotonic() - 60, "a minute old")
value, stale = asyncio.run(
web_api._cached_swr(
cache, lock, compute_lock, "k", 30, lambda: "fresh", max_stale=120
)
)
assert (value, stale) == ("a minute old", True) # SWR within the bound
tests/test_dam_features.py
+9 -7
View File
@@ -51,7 +51,7 @@ class TestHourlyFrame:
class TestLoadDaily:
def test_empty_db_result_does_not_wipe_cache(self, tmp_path):
def test_empty_db_result_does_not_wipe_cache(self, tmp_path, pg_url):
from sqlalchemy import create_engine, text
from src.ml.dam import CACHE_FILE, load_daily
@@ -60,15 +60,16 @@ class TestLoadDaily:
cache_path = tmp_path / CACHE_FILE
_daily(3).rename_axis("date").to_csv(cache_path, compression="gzip")
# Reachable DB whose table exists but is empty
db = f"sqlite:///{tmp_path}/empty_dam.db"
with create_engine(db).begin() as conn:
engine = create_engine(pg_url)
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE rid_reservoir_daily (dam_id TEXT, date DATE, "
"storage_pct REAL, inflow_mcm REAL, outflow_mcm REAL)"
)
)
result = load_daily(db_url=db, cache_dir=tmp_path)
engine.dispose()
result = load_daily(db_url=pg_url, cache_dir=tmp_path)
assert result.empty # honest empty result...
cached = pd.read_csv(cache_path, index_col=0)
assert len(cached) == 3 # ...but the good cache survives
@@ -122,7 +123,8 @@ class TestFeatureGating:
dam = hourly_frame(_daily(days=20, start="2024-09-10"))
X = features.build_features(grid, "P.1", dam=dam)
ts = pd.Timestamp("2024-09-24 12:00")
expected = X.loc[ts, "dam_storage_pct"] - X.loc[
ts - pd.Timedelta(hours=72), "dam_storage_pct"
]
expected = (
X.loc[ts, "dam_storage_pct"]
- X.loc[ts - pd.Timedelta(hours=72), "dam_storage_pct"]
)
assert abs(X.loc[ts, "dam_storage_pct_d3"] - expected) < 1e-9
tests/test_dashboard.py
+46 -13
View File
@@ -1,23 +1,53 @@
import re
from pathlib import Path
DASHBOARD_PATH = Path(__file__).parents[1] / "src" / "static" / "dashboard.html"
ROUTER_PATH = (
Path(__file__).parents[1] / "rust" / "collector" / "src" / "web" / "mod.rs"
)
def _rust_routes() -> set:
"""Paths the website serves (ping-collector's axum router)."""
source = ROUTER_PATH.read_text(encoding="utf-8")
return set(re.findall(r'\.route\(\s*"([^"]+)"', source))
def test_every_api_path_the_dashboard_fetches_is_served():
html = DASHBOARD_PATH.read_text(encoding="utf-8")
routes = _rust_routes()
patterns = [
re.compile("^" + re.sub(r"\\\{[^}]+\\\}", "[^/]+", re.escape(r)) + "$")
for r in routes
]
fetched = set(re.findall(r"fetch\(\s*[`'\"](/[^`'\"?$]*)", html))
assert fetched, "no fetch() calls found"
missing = [
path
for path in fetched
if not path.startswith("/static/")
and not any(p.match(path.rstrip("/") or "/") for p in patterns)
and not any(r.startswith(path) for r in routes)
]
assert not missing, f"dashboard fetches paths the server does not route: {missing}"
def test_dashboard_contains_live_map_and_flow_visualization():
html = DASHBOARD_PATH.read_text(encoding="utf-8")
assert "id=\"station-map\"" in html
assert "id=\"river-flow\"" in html
assert 'id="station-map"' in html
assert 'id="river-flow"' in html
assert "fetch('/stations')" in html or 'fetch("/stations")' in html
assert "fetch('/measurements/latest" in html or 'fetch(\"/measurements/latest' in html
assert (
"fetch('/measurements/latest" in html or 'fetch("/measurements/latest' in html
)
assert "leaflet" in html.lower()
def _body(html: str) -> str:
"""Markup only. The STRINGS table repeats every English phrase, so a
whole-file search would pass even if an element were deleted."""
return html[html.index("<body>"):html.index("<script src=")]
return html[html.index("<body>") : html.index("<script src=")]
def test_dashboard_explains_flow_legend_and_refresh():
@@ -57,13 +87,14 @@ def test_dashboard_has_seo_and_indexing_files():
robots = (static_dir / "robots.txt").read_text(encoding="utf-8")
assert "Sitemap: https://water.buildfor.life/sitemap.xml" in robots
assert "<loc>https://water.buildfor.life/</loc>" in (static_dir / "sitemap.xml").read_text(encoding="utf-8")
assert "Ping River Live Monitor" in (static_dir / "llms.txt").read_text(encoding="utf-8")
assert "<loc>https://water.buildfor.life/</loc>" in (
static_dir / "sitemap.xml"
).read_text(encoding="utf-8")
assert "Ping River Live Monitor" in (static_dir / "llms.txt").read_text(
encoding="utf-8"
)
from src import web_api
routes = {route.path for route in web_api.app.routes}
assert {"/robots.txt", "/llms.txt", "/sitemap.xml"} <= routes
assert {"/robots.txt", "/llms.txt", "/sitemap.xml"} <= _rust_routes()
def test_dashboard_shows_hii_rainfall_layer():
@@ -124,7 +155,9 @@ def test_dashboard_translations_cover_both_languages():
missing_th = tables["en"] - tables["th"]
missing_en = tables["th"] - tables["en"]
assert not missing_th, f"keys missing a Thai translation: {sorted(missing_th)}"
assert not missing_en, f"Thai-only keys with no English fallback: {sorted(missing_en)}"
assert (
not missing_en
), f"Thai-only keys with no English fallback: {sorted(missing_en)}"
def test_dashboard_i18n_markup_keys_exist():
@@ -156,7 +189,7 @@ def test_dashboard_translates_user_visible_aria_labels():
body = _body(html)
for match in re.finditer(r'<[^>]*\saria-label="[^"]+"[^>]*>', body):
tag = match.group(0)
if "data-i18n-aria" in tag or "id=\"lang-toggle\"" in tag:
if "data-i18n-aria" in tag or 'id="lang-toggle"' in tag:
continue # the toggle sets its own label per language in JS
raise AssertionError(f"aria-label without a translation key: {tag[:120]}")
tests/test_flood_forecast.py
+79 -12
View File
@@ -34,7 +34,9 @@ def make_synth(
level = base + noise
for pulse_start, width, peak_add in (pulses or {}).get(station, []):
t = np.arange(n_hours)
bump = peak_add * np.exp(-0.5 * ((t - (pulse_start + width / 2)) / (width / 4)) ** 2)
bump = peak_add * np.exp(
-0.5 * ((t - (pulse_start + width / 2)) / (width / 4)) ** 2
)
level = level + bump
discharge = 20.0 * level + rng.normal(0, 1.0, n_hours)
@@ -107,7 +109,12 @@ def test_label_coverage_gate():
idx = pd.date_range("2020-01-01", periods=n, freq="h")
levels = [1.0] * n
df = pd.DataFrame(
{"timestamp": idx, "station_code": "P.1", "water_level": levels, "discharge": [20.0] * n}
{
"timestamp": idx,
"station_code": "P.1",
"water_level": levels,
"discharge": [20.0] * n,
}
)
# Drop 70% of a future window (hours 21..26) for the row at t=20, no exceedance in it.
df_missing = df[~df["timestamp"].isin(idx[21:26])].copy()
@@ -119,7 +126,14 @@ def test_label_coverage_gate():
# Same sparse window, but WITH an observed exceedance inside it -> must be 1, not NaN.
df_with_peak = df_missing.copy()
peak_row = pd.DataFrame(
[{"timestamp": idx[22], "station_code": "P.1", "water_level": 5.0, "discharge": 100.0}]
[
{
"timestamp": idx[22],
"station_code": "P.1",
"water_level": 5.0,
"discharge": 100.0,
}
]
)
df_with_peak = pd.concat([df_with_peak, peak_row], ignore_index=True)
grid2 = features.make_hourly_grid(df_with_peak)
@@ -193,11 +207,20 @@ def test_train_smoke_and_roundtrip(tmp_path):
data_stations = ["P.1"] + upstream
target_stations = ["P.1", "P.20"]
n = 700
pulses = {station: [(start, 20, 2.0) for start in range(50, n - 50, 110)] for station in data_stations}
pulses = {
station: [(start, 20, 2.0) for start in range(50, n - 50, 110)]
for station in data_stations
}
df = make_synth(n, data_stations, seed=7, pulses=pulses)
metrics = train.train_all(
df, target_stations, models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 20}, use_rain=False, use_dam=False
df,
target_stations,
models_dir=tmp_path,
skip_eval=True,
hgb_overrides={"max_iter": 20},
use_rain=False,
use_dam=False,
)
assert metrics["stations"]["P.1"]["status"] == "trained"
assert metrics["stations"]["P.20"]["status"] == "trained"
@@ -220,7 +243,9 @@ def test_train_smoke_and_roundtrip(tmp_path):
def test_heuristic_fallback(tmp_path):
df = make_synth(50, ["P.1"], seed=3)
readings_by_station = {"P.1": df[["timestamp", "water_level", "discharge"]].to_dict("records")}
readings_by_station = {
"P.1": df[["timestamp", "water_level", "discharge"]].to_dict("records")
}
now = df["timestamp"].max()
forecasts = predict.get_forecasts(readings_by_station, models_dir=tmp_path, now=now)
@@ -235,7 +260,9 @@ def test_heuristic_fallback(tmp_path):
def _p1_synth(n: int = 300, seed: int = 11) -> pd.DataFrame:
upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
return make_synth(n, ["P.1"] + upstream, seed=seed, pulses={"P.1": [(100, 20, 2.0)]})
return make_synth(
n, ["P.1"] + upstream, seed=seed, pulses={"P.1": [(100, 20, 2.0)]}
)
def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
@@ -252,7 +279,15 @@ def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
# Case 1: the loader returns None (archive unreachable, no cache file)
monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: None)
with pytest.raises(train.RainUnavailableError, match="--no-rain"):
train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides=overrides,
use_rain=True,
use_dam=False,
)
assert not (tmp_path / "flood_P.1.joblib").exists()
assert not (tmp_path / "metrics.json").exists()
@@ -262,11 +297,27 @@ def test_train_refuses_silent_rain_downgrade(tmp_path, monkeypatch):
monkeypatch.setattr(rain_mod, "load_history", boom)
with pytest.raises(train.RainUnavailableError, match="simulated Open-Meteo outage"):
train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=True, use_dam=False)
train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides=overrides,
use_rain=True,
use_dam=False,
)
assert not (tmp_path / "flood_P.1.joblib").exists()
# Explicit opt-out still produces v2 bundles as before
metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides=overrides, use_rain=False, use_dam=False)
metrics = train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides=overrides,
use_rain=False,
use_dam=False,
)
assert metrics["model_version"].startswith("hgb-v2+")
assert (tmp_path / "flood_P.1.joblib").exists()
@@ -279,7 +330,15 @@ def test_train_with_rain_series_yields_v3(tmp_path, monkeypatch):
fake_rain = pd.DataFrame({"a": np.linspace(0, 1, len(idx)), "b": 0.5}, index=idx)
monkeypatch.setattr(rain_mod, "load_history", lambda *a, **k: fake_rain)
metrics = train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 10}, use_rain=True, use_dam=False)
metrics = train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides={"max_iter": 10},
use_rain=True,
use_dam=False,
)
assert metrics["model_version"].startswith("hgb-v3+")
bundle = joblib.load(tmp_path / "flood_P.1.joblib")
assert set(features.RAIN_FEATURES) <= set(bundle["feature_names"])
@@ -306,7 +365,15 @@ def test_feature_name_stability(tmp_path):
upstream = [code for code, _lead in features.UPSTREAM_LEADS["P.1"]]
data_stations = ["P.1"] + upstream
df = make_synth(300, data_stations, seed=11, pulses={"P.1": [(100, 20, 2.0)]})
train.train_all(df, ["P.1"], models_dir=tmp_path, skip_eval=True, hgb_overrides={"max_iter": 10}, use_rain=False, use_dam=False)
train.train_all(
df,
["P.1"],
models_dir=tmp_path,
skip_eval=True,
hgb_overrides={"max_iter": 10},
use_rain=False,
use_dam=False,
)
# Safe: loading the bundle this same test just wrote to tmp_path, not an external file.
bundle = joblib.load(tmp_path / "flood_P.1.joblib")
tests/test_forecast_history.py
-99
View File
@@ -1,99 +0,0 @@
"""Tests for the issued-forecast archive (store + API endpoint)."""
import asyncio
import datetime
import pytest
from src.forecast_history import ForecastHistoryStore
def _rows(as_of="2026-08-12T10:00:00"):
return [
{
"as_of": as_of,
"station_code": "P.1",
"horizon_hours": h,
"predicted_max_level": 2.8 + h / 100,
"p_warning": 0.02,
"p_danger": 0.001,
"current_level": 2.76,
"model_version": "hgb-v1+test",
"trained_at": "2026-08-12T10:52:00",
"source": "model",
}
for h in (6, 12, 24)
]
class TestForecastHistoryStore:
@pytest.fixture
def store(self, tmp_path):
store = ForecastHistoryStore(f"sqlite:///{tmp_path}/fh.db", "sqlite")
assert store.connect()
return store
def test_rejects_non_sql(self):
with pytest.raises(ValueError):
ForecastHistoryStore("http://x", "victoriametrics")
def test_roundtrip_and_upsert(self, store):
assert store.save_rows(_rows()) == 3
# Same as_of again -> upsert, still 3 rows
assert store.save_rows(_rows()) == 3
rows = store.fetch("P.1")
assert len(rows) == 3
assert [r["horizon_hours"] for r in rows] == [6, 12, 24]
assert rows[2]["predicted_max_level"] == pytest.approx(3.04)
assert rows[0]["model_version"] == "hgb-v1+test"
def test_fetch_filters(self, store):
store.save_rows(_rows("2026-08-12T10:00:00"))
store.save_rows(_rows("2026-08-12T11:00:00"))
only_24 = store.fetch("P.1", horizon_hours=24)
assert len(only_24) == 2
assert all(r["horizon_hours"] == 24 for r in only_24)
windowed = store.fetch(
"P.1",
start=datetime.datetime(2026, 8, 12, 10, 30),
end=datetime.datetime(2026, 8, 12, 12, 0),
)
assert len(windowed) == 3 # only the 11:00 issue
assert store.fetch("P.99") == []
def test_skips_malformed_rows(self, store):
rows = _rows() + [{"station_code": None, "as_of": None}]
assert store.save_rows(rows) == 3
class TestForecastHistoryEndpoint:
def test_endpoint_returns_rows(self, tmp_path, monkeypatch):
from src import web_api
store = ForecastHistoryStore(f"sqlite:///{tmp_path}/api-fh.db", "sqlite")
assert store.connect()
now = datetime.datetime.now().replace(minute=0, second=0, microsecond=0)
store.save_rows(_rows(now.isoformat()))
monkeypatch.setitem(web_api.app_state, "forecast_store", store)
rows = asyncio.run(
web_api.get_forecast_history(
"P.1", hours=48, start=None, end=None, horizon=24
)
)
assert len(rows) == 1
assert rows[0]["horizon_hours"] == 24
assert rows[0]["station_code"] == "P.1"
def test_endpoint_without_store(self, monkeypatch):
from src import web_api
monkeypatch.setitem(web_api.app_state, "forecast_store", None)
assert (
asyncio.run(
web_api.get_forecast_history(
"P.1", hours=168, start=None, end=None, horizon=None
)
)
== []
)
tests/test_forecast_in_collector.py
-107
View File
@@ -1,107 +0,0 @@
"""FORECAST_IN_COLLECTOR=true: ping-collector forecasts and notifies; the API
serves its stored forecast and falls back to computing when that is stale."""
import asyncio
import datetime
import json
import types
import pytest
from sqlalchemy import create_engine, text
from src import web_api
from src.config import Config
ROWS = [
{
"station_code": "P.1",
"horizon_hours": 24,
"p_warning": 0.012,
"p_danger": 0.001,
"predicted_max_level": 1.31,
"current_level": 1.23,
"as_of": "2026-10-18T13:00:00",
"model_version": "hgb-v3+abc",
"trained_at": "2026-10-01T03:42:00",
"source": "model",
"threshold_warning": 3.7,
"threshold_danger": 4.2,
}
]
@pytest.fixture
def collector_db(tmp_path, monkeypatch):
engine = create_engine(f"sqlite:///{tmp_path / 'db.sqlite'}")
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE forecast_current (id VARCHAR(16) PRIMARY KEY, "
"computed_at TIMESTAMP NOT NULL, model_versions TEXT, payload TEXT NOT NULL)"
)
)
scraper = types.SimpleNamespace(db_adapter=types.SimpleNamespace(engine=engine))
monkeypatch.setitem(web_api.app_state, "scraper", scraper)
monkeypatch.setattr(Config, "FORECAST_IN_COLLECTOR", True)
monkeypatch.setattr(web_api, "FORECAST_CACHE", {})
def store(age_minutes):
computed = datetime.datetime.now() - datetime.timedelta(minutes=age_minutes)
with engine.begin() as conn:
conn.execute(text("DELETE FROM forecast_current"))
conn.execute(
text(
"INSERT INTO forecast_current VALUES ('all', :t, 'hgb-v3+abc', :p)"
),
{"t": computed, "p": json.dumps(ROWS)},
)
return store
def test_serves_the_collectors_forecast(collector_db, monkeypatch):
collector_db(age_minutes=20)
def must_not_compute():
raise AssertionError("computed in-process although a fresh forecast is stored")
monkeypatch.setattr("src.ml.predict.get_latest_forecasts", must_not_compute)
assert asyncio.run(web_api.get_flood_forecasts()) == ROWS
# served from the short cache on the next request
assert "stored" in web_api.FORECAST_CACHE
def test_stale_stored_forecast_falls_back_to_computing(collector_db, monkeypatch):
collector_db(age_minutes=Config.FORECAST_MAX_AGE_SECONDS / 60 + 5)
computed = [dict(ROWS[0], source="heuristic")]
monkeypatch.setattr("src.ml.predict.get_latest_forecasts", lambda: computed)
assert asyncio.run(web_api.get_flood_forecasts()) == computed
def test_missing_table_falls_back_to_computing(tmp_path, monkeypatch):
engine = create_engine(f"sqlite:///{tmp_path / 'empty.sqlite'}")
scraper = types.SimpleNamespace(db_adapter=types.SimpleNamespace(engine=engine))
monkeypatch.setitem(web_api.app_state, "scraper", scraper)
monkeypatch.setattr(Config, "FORECAST_IN_COLLECTOR", True)
monkeypatch.setattr(web_api, "FORECAST_CACHE", {})
monkeypatch.setattr("src.ml.predict.get_latest_forecasts", lambda: ROWS)
assert asyncio.run(web_api.get_flood_forecasts()) == ROWS
def test_this_app_neither_forecasts_nor_notifies(monkeypatch):
published = []
class Publisher:
def publish(self, n):
published.append(n)
return True
monkeypatch.setattr(Config, "FORECAST_IN_COLLECTOR", True)
monkeypatch.setitem(web_api.app_state, "notify", (Publisher(), object()))
monkeypatch.setattr(
"src.ml.predict.get_latest_forecasts",
lambda: pytest.fail("forecast precompute ran in the API"),
)
asyncio.run(web_api._precompute_forecasts())
asyncio.run(web_api._notify_transitions())
assert published == []
tests/test_forecast_skill.py
-93
View File
@@ -1,93 +0,0 @@
"""Forecast skill verification: issued forecasts vs observed peaks (sqlite)."""
import datetime
import pytest
from sqlalchemy import create_engine, text
from src.ml import skill
@pytest.fixture
def engine(tmp_path):
eng = create_engine(f"sqlite:///{tmp_path / 'skill.db'}")
with eng.begin() as c:
c.execute(text("CREATE TABLE stations (id INTEGER PRIMARY KEY, station_code TEXT)"))
c.execute(text("INSERT INTO stations VALUES (1, 'P.1')"))
c.execute(
text(
"CREATE TABLE water_measurements (timestamp DATETIME, station_id INTEGER, water_level REAL)"
)
)
c.execute(
text(
"CREATE TABLE forecast_history (as_of TIMESTAMP, station_code TEXT, horizon_hours INTEGER, "
"predicted_max_level REAL, p_warning REAL, p_danger REAL, current_level REAL, "
"model_version TEXT, source TEXT)"
)
)
return eng
def _fill(engine, start, hours, level_fn, forecasts):
"""hours of hourly observations from `start`, plus (as_of_offset_h, version, pred) rows."""
with engine.begin() as c:
for h in range(hours):
ts = start + datetime.timedelta(hours=h)
c.execute(
text("INSERT INTO water_measurements VALUES (:t, 1, :l)"),
{"t": ts, "l": level_fn(h)},
)
for off, version, pred in forecasts:
ts = start + datetime.timedelta(hours=off)
c.execute(
text(
"INSERT INTO forecast_history VALUES (:t, 'P.1', 24, :p, 0, 0, :cur, :v, 'model')"
),
{"t": ts, "p": pred, "cur": level_fn(off), "v": version},
)
def test_skill_per_version_and_trend(engine):
start = datetime.datetime(2026, 8, 1)
# river: flat 1.5 m, with a bump to 2.4 m around hour 100
level = lambda h: 2.4 if 96 <= h <= 104 else 1.5
forecasts = []
# old version: always predicts 1.5 (persistence-like, misses the bump)
for off in range(0, 60):
forecasts.append((off, "hgb-v2+aaaaaaa", 1.5))
# new version: predicts 1.5 normally and 2.3 ahead of the bump
for off in range(60, 200):
pred = 2.3 if 72 <= off <= 104 else 1.5
forecasts.append((off, "hgb-v3+bbbbbbb", pred))
_fill(engine, start, 260, level, forecasts)
out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=300))
assert [v["model_version"] for v in out["versions"]] == ["hgb-v2+aaaaaaa", "hgb-v3+bbbbbbb"]
old, new = out["versions"]
assert old["n"] == 60 and old["enough_data"]
assert new["n"] == 140 and new["enough_data"]
# the old version issued only on flat hours: perfect there, no bump rows
assert old["mae_m"] == 0.0 and old["above_2m_n"] == 0
# the new version saw the bump: nonzero MAE but positive skill vs persistence
assert new["above_2m_n"] > 0
assert new["skill"] is not None and new["skill"] > 0
assert out["current"]["model_version"] == "hgb-v3+bbbbbbb"
assert out["trend"]["previous_version"] == "hgb-v2+aaaaaaa"
assert out["trend"]["better"] is False # honest: old had an easier period
def test_skill_requires_full_window(engine):
start = datetime.datetime(2026, 8, 1)
# forecasts issued at the very end have no observed window yet
_fill(engine, start, 30, lambda h: 1.5, [(o, "hgb-v3+ccccccc", 1.5) for o in range(0, 30)])
out = skill.compute_skill(engine, "sqlite", "P.1", 24, now=start + datetime.timedelta(hours=30))
# only as_of <= now-24h AND with >= 18 observed hours in the window count
assert out["versions"] and out["versions"][0]["n"] == 7 # as_of 0..6 h: <= now-24h with >= 18 observed hours
assert out["versions"][0]["enough_data"] is False
assert out["trend"] is None
def test_skill_empty(engine):
out = skill.compute_skill(engine, "sqlite", "P.1", 24)
assert out["versions"] == [] and out["current"] is None and out["trend"] is None
tests/test_hii_collector.py
+5 -184
View File
@@ -1,4 +1,4 @@
"""Tests for the HII/ThaiWater api-v3 collector (parsing + persistence)."""
"""Tests for the HII/ThaiWater api-v3 client/store (parsing + persistence)."""
import datetime
@@ -162,14 +162,11 @@ class TestParseWaterlevel:
class TestHiiStore:
@pytest.fixture
def store(self, tmp_path):
store = HiiStore(f"sqlite:///{tmp_path}/hii_test.db", "sqlite")
def store(self, pg_url):
store = HiiStore(pg_url)
assert store.connect()
return store
def test_rejects_non_sql_backend(self):
with pytest.raises(ValueError):
HiiStore("http://localhost:8428", "victoriametrics")
yield store
store.engine.dispose()
def test_rain_roundtrip_and_upsert(self, store):
records = parse_rain_records(_rain_payload())
@@ -275,182 +272,6 @@ class TestParseGraphRows:
assert parse_graph_rows({}) == []
class TestHiiApiEndpoints:
"""Call the endpoint coroutines directly (the venv's httpx/starlette
combination is incompatible with TestClient)."""
@pytest.fixture
def web_api(self, tmp_path, monkeypatch):
from src import web_api
from src.hii_collector import HiiCollector
collector = HiiCollector(
{"type": "sqlite", "connection_string": f"sqlite:///{tmp_path}/api.db"}
)
now = datetime.datetime.now().replace(microsecond=0)
rain = parse_rain_records(_rain_payload())
wl = parse_waterlevel_records(_waterlevel_payload())
for record in rain + wl:
record["timestamp"] = now
assert collector.store.save_rain(rain) == 1
assert collector.store.save_waterlevel(wl) == 2
monkeypatch.setitem(web_api.app_state, "hii_collector", collector)
web_api.HII_CACHE.clear() # response cache would leak across tests
web_api._REFRESH_IN_FLIGHT.clear()
return web_api
@staticmethod
def _get(web_api_module, endpoint, **kwargs):
import asyncio
from fastapi import Response
response = Response()
rows = asyncio.run(
getattr(web_api_module, endpoint)(response=response, **kwargs)
)
return rows, response
def test_rainfall_latest(self, web_api):
rows, _ = self._get(web_api, "get_hii_rainfall_latest", hours=26)
assert len(rows) == 1
assert rows[0]["oldcode"] == "CHM005"
assert rows[0]["rain_24h"] == 49.6
assert rows[0]["latitude"] == pytest.approx(19.12207)
def test_waterlevel_latest(self, web_api):
rows, _ = self._get(web_api, "get_hii_waterlevel_latest", hours=26)
assert len(rows) == 2
p1 = next(r for r in rows if r["oldcode"] == "P.1")
assert p1["rid_code"] == "P.1"
assert p1["wl_msl"] == 303.27
assert p1["offset_msl"] == 300.5
assert p1["situation_level"] == 4
def test_empty_when_collector_disabled(self, monkeypatch):
from src import web_api
monkeypatch.setitem(web_api.app_state, "hii_collector", None)
web_api.HII_CACHE.clear()
web_api._REFRESH_IN_FLIGHT.clear()
assert self._get(web_api, "get_hii_rainfall_latest", hours=26)[0] == []
assert self._get(web_api, "get_hii_waterlevel_latest", hours=26)[0] == []
def test_latest_responses_are_cached(self, web_api, monkeypatch):
calls = {"n": 0}
real = web_api._hii_rows
def counting(sql, params):
calls["n"] += 1
return real(sql, params)
monkeypatch.setattr(web_api, "_hii_rows", counting)
first, _ = self._get(web_api, "get_hii_rainfall_latest", hours=26)
second, _ = self._get(web_api, "get_hii_rainfall_latest", hours=26)
assert first == second and len(first) == 1
assert calls["n"] == 1 # second call served from the TTL cache
# different hours -> different cache key -> fresh query
self._get(web_api, "get_hii_rainfall_latest", hours=48)
assert calls["n"] == 2
def test_rainfall_catchment(self, web_api):
"""One gauge in the box (CHM005, 19.12N 98.94E) is below the
MIN_GAUGES_PER_HOUR floor, so the catchment mean is NaN -> null, the
openmeteo_rain table does not exist in this store, and the comparison
reports no overlap. Shape is what matters: the endpoint must not 500
on a fresh database."""
payload, response = self._get(web_api, "get_hii_rainfall_catchment", days=7)
assert "x-data-stale" not in response.headers
assert list(payload) == ["box", "gauge", "openmeteo", "comparison_24h_sums"]
assert payload["openmeteo"] == []
assert payload["comparison_24h_sums"] == {"overlap_hours": 0}
assert len(payload["gauge"]) == 1
assert payload["gauge"][0]["rain_mm"] is None # < MIN_GAUGES_PER_HOUR
def test_rainfall_catchment_disabled(self, monkeypatch):
from src import web_api
monkeypatch.setitem(web_api.app_state, "hii_collector", None)
web_api.HII_CACHE.clear()
web_api._REFRESH_IN_FLIGHT.clear()
payload, _ = self._get(web_api, "get_hii_rainfall_catchment", days=7)
assert payload["gauge"] == [] and payload["openmeteo"] == []
def test_stale_served_on_recompute_failure(self, web_api, monkeypatch):
# Prime the cache, expire it, break the DB: the stale copy is served
# and flagged via the X-Data-Stale header.
good, response = self._get(web_api, "get_hii_rainfall_latest", hours=26)
assert good and "x-data-stale" not in response.headers
from src.config import Config
monkeypatch.setattr(Config, "HII_CACHE_TTL_SECONDS", 0)
def broken(sql, params):
raise RuntimeError("db unreachable")
monkeypatch.setattr(web_api, "_hii_rows", broken)
rows, response = self._get(web_api, "get_hii_rainfall_latest", hours=26)
assert rows == good
assert response.headers["X-Data-Stale"] == "true"
def test_empty_results_are_not_cached(self, tmp_path, monkeypatch):
from src import web_api
from src.hii_collector import HiiCollector
collector = HiiCollector(
{"type": "sqlite", "connection_string": f"sqlite:///{tmp_path}/empty.db"}
)
monkeypatch.setitem(web_api.app_state, "hii_collector", collector)
web_api.HII_CACHE.clear()
web_api._REFRESH_IN_FLIGHT.clear()
assert self._get(web_api, "get_hii_rainfall_latest", hours=26)[0] == []
assert web_api.HII_CACHE == {} # empty response left uncached
def test_measurements_latest_cached_and_stale(self, monkeypatch):
from types import SimpleNamespace
from src import web_api
from src.config import Config
calls = {"n": 0}
def get_latest_data(limit=100):
calls["n"] += 1
return [
{
"timestamp": "2026-08-12 10:00:00",
"station_code": "P.1",
"station_name_en": "Nawarat Bridge",
"station_name_th": "สะพานนวรัฐ",
"water_level": 2.76,
"discharge": 331.0,
"discharge_percent": 77.9,
}
]
scraper = SimpleNamespace(db_adapter=True, get_latest_data=get_latest_data)
monkeypatch.setitem(web_api.app_state, "scraper", scraper)
web_api.LATEST_CACHE.clear()
web_api._REFRESH_IN_FLIGHT.clear()
rows, response = self._get(web_api, "get_latest_measurements", limit=500)
rows2, _ = self._get(web_api, "get_latest_measurements", limit=500)
assert calls["n"] == 1 # second call cached
assert rows2[0].station_code == "P.1"
# DB failure after expiry -> stale copy + header
monkeypatch.setattr(Config, "LATEST_CACHE_TTL_SECONDS", 0)
def broken(limit=100):
raise RuntimeError("db unreachable")
scraper.get_latest_data = broken
rows3, response = self._get(web_api, "get_latest_measurements", limit=500)
assert rows3[0].water_level == 2.76
assert response.headers["X-Data-Stale"] == "true"
class TestBackfillHelpers:
def test_chunk_date_range(self):
chunks = chunk_date_range(
tests/test_hii_fill.py
+8 -7
View File
@@ -28,12 +28,13 @@ def _base_frame(code="P.20", n=200, level=2.0):
@pytest.fixture
def hii_db(tmp_path):
"""SQLite DB with hii_* tables; returns (db_url, store)."""
db_url = f"sqlite:///{tmp_path}/hii_fill.db"
store = HiiStore(db_url, "sqlite")
def hii_db(pg_url):
"""A schema with the hii_* tables; returns (db_url, store)."""
store = HiiStore(pg_url)
assert store.connect()
return db_url, store
yield pg_url, store
store.engine.dispose()
def _seed_mirror(store, station_id, rid_code, hours, wl_msl, discharge=250.0):
from sqlalchemy import text
@@ -113,9 +114,9 @@ class TestFillFromHii:
filled = fill_from_hii(base, db_url, min_overlap_hours=1)
assert len(filled) == len(base) # P.76 not in HII_FILL_STATIONS
def test_missing_tables_degrade_gracefully(self, tmp_path):
def test_missing_tables_degrade_gracefully(self, pg_url):
base = _base_frame("P.20")
filled = fill_from_hii(base, f"sqlite:///{tmp_path}/empty.db")
filled = fill_from_hii(base, pg_url) # empty schema: no hii_* tables
assert filled.equals(base)
def test_no_station_overlap_returns_input(self, hii_db):
tests/test_integration.py
-210
View File
@@ -1,210 +0,0 @@
#!/usr/bin/env python3
"""
Integration test script for the enhanced water monitoring system
"""
import sys
import os
import time
from datetime import datetime
# Add project root to Python path
project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, project_root)
def test_imports():
"""Test that all modules can be imported"""
print("Testing imports...")
try:
from src.config import Config
from src.models import WaterMeasurement, StationInfo
from src.exceptions import WaterMonitorException
from src.validators import DataValidator
from src.metrics import get_metrics_collector
from src.health_check import HealthCheckManager
from src.rate_limiter import RateLimiter
from src.logging_config import setup_logging
print("✅ All imports successful")
return True
except ImportError as e:
print(f"❌ Import failed: {e}")
return False
def test_configuration():
"""Test configuration validation"""
print("Testing configuration...")
try:
from src.config import Config
# Test configuration loading
settings = Config.get_all_settings()
print(f"✅ Configuration loaded: {len(settings)} settings")
# Test database config
db_config = Config.get_database_config()
print(f"✅ Database config: {db_config['type']}")
return True
except Exception as e:
print(f"❌ Configuration test failed: {e}")
return False
def test_metrics():
"""Test metrics collection"""
print("Testing metrics...")
try:
from src.metrics import increment_counter, set_gauge, record_histogram, get_metrics_collector
# Test metrics
increment_counter("test_counter", 5)
set_gauge("test_gauge", 42.0)
record_histogram("test_histogram", 1.5)
# Get metrics
collector = get_metrics_collector()
metrics = collector.get_all_metrics()
print(f"✅ Metrics collected: {len(metrics['counters'])} counters, {len(metrics['gauges'])} gauges")
return True
except Exception as e:
print(f"❌ Metrics test failed: {e}")
return False
def test_validation():
"""Test data validation"""
print("Testing data validation...")
try:
from src.validators import DataValidator
# Test valid measurement
valid_measurement = {
'timestamp': datetime.now(),
'station_id': 1,
'water_level': 5.5,
'discharge': 100.0,
'discharge_percent': 75.0
}
result = DataValidator.validate_measurement(valid_measurement)
if result:
print("✅ Valid measurement passed validation")
else:
print("❌ Valid measurement failed validation")
return False
# Test invalid measurement
invalid_measurement = {
'timestamp': datetime.now(),
'station_id': 999, # Invalid station ID
'water_level': -999.0, # Invalid water level
'discharge': -50.0 # Invalid discharge
}
result = DataValidator.validate_measurement(invalid_measurement)
if not result:
print("✅ Invalid measurement correctly rejected")
else:
print("❌ Invalid measurement incorrectly accepted")
return False
return True
except Exception as e:
print(f"❌ Validation test failed: {e}")
return False
def test_rate_limiter():
"""Test rate limiting"""
print("Testing rate limiter...")
try:
from src.rate_limiter import RateLimiter
# Create rate limiter (2 requests per second)
limiter = RateLimiter(max_requests=2, time_window_seconds=1)
# Test allowed requests
for i in range(2):
if not limiter.is_allowed():
print(f"❌ Request {i+1} should be allowed")
return False
# Test blocked request
if limiter.is_allowed():
print("❌ Third request should be blocked")
return False
print("✅ Rate limiter working correctly")
return True
except Exception as e:
print(f"❌ Rate limiter test failed: {e}")
return False
def test_logging():
"""Test logging configuration"""
print("Testing logging...")
try:
from src.logging_config import setup_logging, get_logger
# Setup logging
logger = setup_logging(log_level="INFO", enable_console=False)
# Get a logger
test_logger = get_logger("test")
test_logger.info("Test log message")
print("✅ Logging setup successful")
return True
except Exception as e:
print(f"❌ Logging test failed: {e}")
return False
def main():
"""Run all tests"""
print("🧪 Running integration tests for Northern Thailand Ping River Monitor v3.1.3")
print("=" * 60)
tests = [
test_imports,
test_configuration,
test_metrics,
test_validation,
test_rate_limiter,
test_logging
]
passed = 0
failed = 0
for test in tests:
try:
if test():
passed += 1
else:
failed += 1
except Exception as e:
print(f"❌ Test {test.__name__} crashed: {e}")
failed += 1
print()
print("=" * 60)
print(f"Test Results: {passed} passed, {failed} failed")
if failed == 0:
print("🎉 All tests passed! The system is ready to use.")
print("\nNext steps:")
print("1. Run 'python run.py --test' to test data collection")
print("2. Run 'python run.py --web-api' to start the web interface")
print("3. Visit http://localhost:8000 for the dashboard")
return True
else:
print("❌ Some tests failed. Please check the errors above.")
return False
if __name__ == "__main__":
success = main()
sys.exit(0 if success else 1)
tests/test_matrix_formatting.py
-98
View File
@@ -1,98 +0,0 @@
"""Assert-based tests for Matrix message formatting.
Matrix clients only render formatting from an HTML ``formatted_body``; Markdown
in the plain ``body`` shows as literal characters. These tests lock in that the
notifier emits real HTML plus a clean plain-text fallback, and that untrusted
station data is HTML-escaped.
"""
import datetime
from src.alerting import AlertLevel, MatrixNotifier, WaterAlert, markdown_to_matrix_html, strip_markdown
class _FakeResponse:
def raise_for_status(self):
pass
def json(self):
return {"event_id": "$test"}
def _notifier_capturing(captured):
"""A MatrixNotifier whose HTTP PUT records the JSON payload into ``captured``."""
notifier = MatrixNotifier("https://hs.example", "token", "!room:hs.example")
def fake_put(url, headers=None, json=None, timeout=None):
captured.update(json)
return _FakeResponse()
notifier.session.put = fake_put
return notifier
def test_bold_becomes_strong():
assert markdown_to_matrix_html("**hi**") == "<strong>hi</strong>"
def test_url_is_linkified():
out = markdown_to_matrix_html("see https://x.example/z")
assert '<a href="https://x.example/z">https://x.example/z</a>' in out
def test_newlines_become_br():
assert markdown_to_matrix_html("a\nb") == "a<br/>b"
def test_html_is_escaped():
out = markdown_to_matrix_html("<script> & 'stuff'")
assert "&lt;script&gt;" in out
assert "&amp;" in out
assert "<script>" not in out
def test_strip_markdown_removes_bold_markers():
assert strip_markdown("**WATER LEVEL ALERT**") == "WATER LEVEL ALERT"
def test_send_message_sends_html_and_plain_fallback():
captured = {}
notifier = _notifier_capturing(captured)
assert notifier.send_message("**hi** http://x.example") is True
assert captured["format"] == "org.matrix.custom.html"
assert "<strong>hi</strong>" in captured["formatted_body"]
# Plain body has the markdown markers stripped.
assert captured["body"] == "hi http://x.example"
def test_send_message_plain_when_markdown_disabled():
captured = {}
notifier = _notifier_capturing(captured)
assert notifier.send_message("**raw**", markdown=False) is True
assert "formatted_body" not in captured
assert captured["body"] == "**raw**"
def test_send_alert_renders_alert_fields():
captured = {}
notifier = _notifier_capturing(captured)
alert = WaterAlert(
station_code="P.1",
station_name="สะพานนวรัฐ",
alert_type="Zone 7 - Critical",
level=AlertLevel.CRITICAL,
water_level=4.62,
threshold=4.60,
discharge=612.0,
timestamp=datetime.datetime(2026, 7, 22, 14, 30, 0),
)
assert notifier.send_alert(alert) is True
html = captured["formatted_body"]
assert "<strong>WATER LEVEL ALERT</strong>" in html
assert "สะพานนวรัฐ" in html # Thai station name preserved
assert "<strong>Current Level:</strong>" in html
# Plain fallback carries no leftover markdown markers.
assert "**" not in captured["body"]
tests/test_notify.py
-285
View File
@@ -1,285 +0,0 @@
"""ntfy notification state machine: transitions only, hysteresis, restart-safe."""
import datetime
import pytest
from src import notify
class FakePublisher(notify.NtfyPublisher):
def __init__(self):
super().__init__("http://ntfy.test", prefix="ping")
self.sent = []
def publish(self, n):
self.sent.append(n)
return True
@pytest.fixture
def pub():
return FakePublisher()
def _reading(code, level, ts="2026-09-24T12:00:00"):
return {"station_code": code, "water_level": level, "timestamp": ts}
def _fc(p, peak=None):
return [
{
"station_code": "P.1",
"horizon_hours": 24,
"p_warning": p,
"predicted_max_level": peak,
"source": "model",
}
]
NOW = datetime.datetime(2026, 9, 24, 12, 30)
def topics(pub):
return [n.topic for n in pub.sent]
def test_quiet_river_sends_nothing(pub):
state = notify.InMemoryState()
for h in range(48):
notify.evaluate(
[_reading("P.1", 1.6), _reading("P.103", 3.2)],
_fc(0.01),
state,
pub,
now=NOW,
)
assert pub.sent == []
def test_warning_crossing_once_then_silence_then_clear(pub):
state = notify.InMemoryState()
# rising through 3.70 (P.1 warning)
notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
assert pub.sent == []
notify.evaluate([_reading("P.1", 3.72)], [], state, pub, now=NOW)
assert topics(pub) == ["ping-p1-warning", "ping-warning"]
assert pub.sent[0].priority == 4 and "3.72 m" in pub.sent[0].message
# stays above: no repeats for many hours
for level in (3.80, 3.95, 4.05, 3.90, 3.75):
notify.evaluate([_reading("P.1", level)], [], state, pub, now=NOW)
assert len(pub.sent) == 2
# dips to 3.65: within hysteresis, still no message
notify.evaluate([_reading("P.1", 3.65)], [], state, pub, now=NOW)
assert len(pub.sent) == 2
# 3.55: clear
notify.evaluate([_reading("P.1", 3.55)], [], state, pub, now=NOW)
assert topics(pub)[2:] == ["ping-p1-warning", "ping-warning"]
assert "back to normal" in pub.sent[2].title
def test_danger_escalation_and_deescalation(pub):
state = notify.InMemoryState()
notify.evaluate([_reading("P.1", 3.9)], [], state, pub, now=NOW) # warning
notify.evaluate(
[_reading("P.1", 4.25)], [], state, pub, now=NOW
) # danger (>= 4.20)
assert topics(pub) == [
"ping-p1-warning",
"ping-warning",
"ping-p1-danger",
"ping-danger",
]
assert pub.sent[2].priority == 5
notify.evaluate(
[_reading("P.1", 4.15)], [], state, pub, now=NOW
) # hysteresis: still danger
assert len(pub.sent) == 4
notify.evaluate([_reading("P.1", 4.05)], [], state, pub, now=NOW) # back to warning
assert topics(pub)[4:] == ["ping-p1-danger", "ping-warning"]
assert "below danger" in pub.sent[4].title
def test_jump_straight_to_danger(pub):
state = notify.InMemoryState()
notify.evaluate(
[_reading("P.103", 7.0)], [], state, pub, now=NOW
) # P.103 danger 6.75
assert topics(pub) == ["ping-p103-danger", "ping-danger"]
def test_basin_digest_groups_stations(pub):
state = notify.InMemoryState()
notify.evaluate(
[_reading("P.1", 3.8), _reading("P.103", 6.0), _reading("P.67", 1.0)],
[],
state,
pub,
now=NOW,
)
basin = [n for n in pub.sent if n.topic == "ping-warning"]
assert len(basin) == 1 and "P.1" in basin[0].message and "P.103" in basin[0].message
def test_outlook_on_off_with_hysteresis(pub):
state = notify.InMemoryState()
r = [_reading("P.1", 2.9)]
notify.evaluate(r, _fc(0.30), state, pub, now=NOW)
assert pub.sent == []
notify.evaluate(r, _fc(0.55, 3.9), state, pub, now=NOW)
assert topics(pub) == ["ping-p1-outlook"]
assert "55%" in pub.sent[0].message and "3.90 m" in pub.sent[0].message
assert "not an official warning" in pub.sent[0].message
notify.evaluate(
r, _fc(0.40), state, pub, now=NOW
) # between OFF and ON: stays on, silent
assert len(pub.sent) == 1
notify.evaluate(r, _fc(0.20), state, pub, now=NOW)
assert len(pub.sent) == 2 and "easing" in pub.sent[1].title
def test_heuristic_forecast_ignored(pub):
state = notify.InMemoryState()
fc = [
{
"station_code": "P.1",
"horizon_hours": 24,
"p_warning": 0.9,
"source": "heuristic",
}
]
notify.evaluate([_reading("P.1", 2.0)], fc, state, pub, now=NOW)
assert pub.sent == []
def test_stale_feed_and_recovery(pub):
state = notify.InMemoryState()
notify.evaluate(
[_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=NOW
)
assert pub.sent == []
later = NOW + datetime.timedelta(hours=4)
notify.evaluate(
[_reading("P.1", 1.6, "2026-09-24T12:00:00")], [], state, pub, now=later
)
assert topics(pub) == ["ping-status"] and "stale" in pub.sent[0].title
notify.evaluate(
[_reading("P.1", 1.6, "2026-09-24T12:00:00")],
[],
state,
pub,
now=later + datetime.timedelta(hours=1),
)
assert len(pub.sent) == 1 # still stale, no repeat
notify.evaluate(
[_reading("P.1", 1.6, "2026-09-24T17:00:00")],
[],
state,
pub,
now=later + datetime.timedelta(hours=1),
)
assert len(pub.sent) == 2 and "recovered" in pub.sent[1].title
def test_capacity_guard_blocks_stale_threshold(pub):
"""P.77 2026-09: 3.02 m >= 2.85 m 'warning' at 22 % capacity -> not a flood."""
state = notify.InMemoryState()
r = {
"station_code": "P.77",
"water_level": 4.40,
"timestamp": "2026-09-24T12:00:00",
"discharge_percent": 10.3,
}
notify.evaluate([r], [], state, pub, now=NOW)
assert pub.sent == [] and state.get("level:P.77") is None
# same level with capacity agreeing -> alert
r["discharge_percent"] = 82.0
notify.evaluate([r], [], state, pub, now=NOW)
assert topics(pub) == ["ping-p77-warning", "ping-warning"]
def test_capacity_guard_exempts_p1_and_missing_pct(pub):
state = notify.InMemoryState()
notify.evaluate(
[
{
"station_code": "P.1",
"water_level": 3.75,
"timestamp": "2026-09-24T12:00:00",
"discharge_percent": 40.0,
}
],
[],
state,
pub,
now=NOW,
)
assert topics(pub) == ["ping-p1-warning", "ping-warning"]
pub.sent.clear()
notify.evaluate(
[
{
"station_code": "P.103",
"water_level": 6.0,
"timestamp": "2026-09-24T12:00:00",
}
],
[],
state,
pub,
now=NOW,
)
assert topics(pub) == ["ping-p103-warning", "ping-warning"]
def test_capacity_guard_does_not_block_clearing(pub):
"""Guard applies only to the clear->alert edge; the all-clear always goes out."""
state = notify.InMemoryState()
r = {
"station_code": "P.67",
"water_level": 2.6,
"timestamp": "2026-09-24T12:00:00",
"discharge_percent": 90.0,
}
notify.evaluate([r], [], state, pub, now=NOW)
assert len(pub.sent) == 2
r.update(water_level=2.2, discharge_percent=30.0)
notify.evaluate([r], [], state, pub, now=NOW)
assert "back to normal" in pub.sent[2].title
def test_state_survives_restart_via_sql(tmp_path, pub):
from sqlalchemy import create_engine
eng = create_engine(f"sqlite:///{tmp_path / 'n.db'}")
state = notify.NotificationState(eng, "sqlite")
notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
assert len(pub.sent) == 2
# "restart": new state object on the same DB, same reading -> nothing re-sent
state2 = notify.NotificationState(eng, "sqlite")
notify.evaluate([_reading("P.1", 3.8)], [], state2, pub, now=NOW)
assert len(pub.sent) == 2
def test_publish_failure_does_not_advance_state():
"""If ntfy is down the transition must be retried next cycle, not lost."""
class Down(notify.NtfyPublisher):
def __init__(self):
super().__init__("http://ntfy.test")
self.calls = 0
def publish(self, n):
self.calls += 1
return False
pub = Down()
state = notify.InMemoryState()
notify.evaluate([_reading("P.1", 3.8)], [], state, pub, now=NOW)
assert pub.calls == 2 and state.get("level:P.1") is None
# next cycle, ntfy back: the crossing is delivered
good = FakePublisher()
notify.evaluate([_reading("P.1", 3.8)], [], state, good, now=NOW)
assert topics(good) == ["ping-p1-warning", "ping-warning"]
assert state.get("level:P.1") == "warning"
tests/test_postgres_history.py
-49
View File
@@ -1,49 +0,0 @@
import datetime
from sqlalchemy import create_engine, text
from src.postgres_history import PostgresHistory
def test_history_returns_station_series_in_chronological_order(tmp_path):
engine = create_engine(f"sqlite:///{tmp_path / 'history.db'}")
with engine.begin() as connection:
connection.execute(text("CREATE TABLE stations (id INTEGER PRIMARY KEY, station_code TEXT)"))
connection.execute(
text(
"CREATE TABLE water_measurements ("
"timestamp DATETIME, station_id INTEGER, water_level REAL, "
"discharge REAL, discharge_percent REAL)"
)
)
connection.execute(text("INSERT INTO stations VALUES (1, 'P.1'), (2, 'P.20')"))
connection.execute(
text(
"INSERT INTO water_measurements VALUES "
"('2026-08-09 13:00:00', 1, 3.2, 110.0, 40.0),"
"('2026-08-09 14:00:00', 1, 3.4, 120.0, 42.0),"
"('2026-08-09 14:00:00', 2, 2.1, 30.0, 15.0)"
)
)
history = PostgresHistory(engine=engine).station_history(
"P.1",
start=datetime.datetime(2026, 8, 9, 12),
end=datetime.datetime(2026, 8, 9, 15),
limit=100,
)
assert [row["timestamp"].hour for row in history] == [13, 14]
assert [row["discharge"] for row in history] == [110.0, 120.0]
assert all(row["station_code"] == "P.1" for row in history)
def test_history_rejects_excessive_limit():
history = PostgresHistory.__new__(PostgresHistory)
try:
history.station_history("P.1", datetime.datetime.now(), datetime.datetime.now(), 100001)
except ValueError as error:
assert "limit" in str(error)
else:
raise AssertionError("Expected excessive history limit to be rejected")
tests/test_rid_reservoir.py
+32 -49
View File
@@ -1,4 +1,4 @@
"""Tests for the RID large-dam daily collector (parsing + persistence)."""
"""Tests for the RID large-dam client/store/backfill (parsing + persistence)."""
import datetime
@@ -6,7 +6,6 @@ import pytest
from src.rid_reservoir import (
MAE_NGAT_DAM_ID,
RidReservoirCollector,
RidReservoirStore,
backfill,
backfill_dam,
@@ -131,8 +130,9 @@ class FakeRangeClient:
start + datetime.timedelta(days=i) for i in range((end - start).days + 1)
]
return parse_dam_range_records(
_range_payload(rows=[_range_row(d.isoformat()) for d in days
if d not in self.gaps])
_range_payload(
rows=[_range_row(d.isoformat()) for d in days if d not in self.gaps]
)
)
@@ -177,7 +177,9 @@ class TestParsing:
)
def test_parse_dam_range_zero_level_is_missing_not_a_reading(self):
payload = _range_payload(rows=[_range_row("2024-09-24", DMD_ULevel_curr="0.00")])
payload = _range_payload(
rows=[_range_row("2024-09-24", DMD_ULevel_curr="0.00")]
)
assert parse_dam_range_records(payload)[0]["level_msl"] is None
def test_parse_dam_range_skips_broken_rows(self):
@@ -196,14 +198,11 @@ class TestParsing:
class TestStore:
@pytest.fixture
def store(self, tmp_path):
store = RidReservoirStore(f"sqlite:///{tmp_path}/rsv_test.db", "sqlite")
def store(self, pg_url):
store = RidReservoirStore(pg_url)
assert store.connect()
return store
def test_rejects_non_sql(self):
with pytest.raises(ValueError):
RidReservoirStore("http://localhost:8428", "victoriametrics")
yield store
store.engine.dispose()
def test_roundtrip_and_upsert(self, store):
records = parse_dam_records(_dams_payload())
@@ -269,8 +268,11 @@ class TestStore:
date = "2026-08-13"
assert store.save(parse_dam_range_records(_range_payload((date,)))) == 1
# api/dams has no level column at all; its rewrite must not clear one
dams_row = [r for r in parse_dam_records(_dams_payload(date))
if r["dam_id"] == MAE_NGAT_DAM_ID]
dams_row = [
r
for r in parse_dam_records(_dams_payload(date))
if r["dam_id"] == MAE_NGAT_DAM_ID
]
assert dams_row[0]["level_msl"] is None
store.save(dams_row)
from sqlalchemy import text
@@ -315,27 +317,15 @@ class TestStore:
assert row[2] == "เหนือ"
class TestCollectorAndBackfill:
def test_run_cycle_today_and_yesterday(self, tmp_path):
collector = RidReservoirCollector(
{"type": "sqlite", "connection_string": f"sqlite:///{tmp_path}/c.db"},
client=FakeClient(),
)
saved = collector.run_cycle()
assert saved == 4 # 2 dams x (today + yesterday)
assert collector.client.calls[0] is None # today via empty date
def test_run_cycle_survives_fetch_failure(self, tmp_path):
yesterday = datetime.date.today() - datetime.timedelta(days=1)
collector = RidReservoirCollector(
{"type": "sqlite", "connection_string": f"sqlite:///{tmp_path}/f.db"},
client=FakeClient(fail_dates={yesterday}),
)
assert collector.run_cycle() == 2 # today only
def test_backfill_range(self, tmp_path):
store = RidReservoirStore(f"sqlite:///{tmp_path}/b.db", "sqlite")
class TestBackfill:
@pytest.fixture
def store(self, pg_url):
store = RidReservoirStore(pg_url)
assert store.connect()
yield store
store.engine.dispose()
def test_backfill_range(self, store):
client = FakeClient()
saved = backfill(
store,
@@ -351,9 +341,7 @@ class TestCollectorAndBackfill:
datetime.date(2024, 9, 26),
]
def test_backfill_skips_present_days_so_reruns_repair_holes(self, tmp_path):
store = RidReservoirStore(f"sqlite:///{tmp_path}/skip.db", "sqlite")
assert store.connect()
def test_backfill_skips_present_days_so_reruns_repair_holes(self, store):
# Day 25 already stored (e.g. by the live collector)
store.save(parse_dam_records(_dams_payload("2024-09-25")))
client = FakeClient()
@@ -370,9 +358,7 @@ class TestCollectorAndBackfill:
]
assert saved == 4
def test_backfill_aborts_when_db_saves_nothing(self, tmp_path, monkeypatch):
store = RidReservoirStore(f"sqlite:///{tmp_path}/deaddb.db", "sqlite")
assert store.connect()
def test_backfill_aborts_when_db_saves_nothing(self, store, monkeypatch):
monkeypatch.setattr(store, "save", lambda records: 0) # broken DB
client = FakeClient()
backfill(
@@ -384,9 +370,7 @@ class TestCollectorAndBackfill:
)
assert len(client.calls) == 5 # aborted, not one request per day
def test_backfill_aborts_after_consecutive_failures(self, tmp_path):
store = RidReservoirStore(f"sqlite:///{tmp_path}/a.db", "sqlite")
assert store.connect()
def test_backfill_aborts_after_consecutive_failures(self, store):
start = datetime.date(2024, 1, 1)
fail_dates = {start + datetime.timedelta(days=i) for i in range(2, 30)}
client = FakeClient(fail_dates=fail_dates)
@@ -403,10 +387,11 @@ class TestCollectorAndBackfill:
class TestBackfillDam:
@pytest.fixture
def store(self, tmp_path):
store = RidReservoirStore(f"sqlite:///{tmp_path}/range.db", "sqlite")
def store(self, pg_url):
store = RidReservoirStore(pg_url)
assert store.connect()
return store
yield store
store.engine.dispose()
def test_whole_range_in_one_request(self, store):
client = FakeRangeClient()
@@ -451,9 +436,7 @@ class TestBackfillDam:
assert saved == 4 # the day the source never published stays absent
assert store.present_dates(
datetime.date(2024, 1, 1), datetime.date(2024, 1, 5)
) == {
datetime.date(2024, 1, d) for d in (1, 2, 4, 5)
}
) == {datetime.date(2024, 1, d) for d in (1, 2, 4, 5)}
def test_stored_days_are_skipped_and_whole_chunks_cost_no_request(self, store):
client = FakeRangeClient()
tests/test_scraper_parsing.py
-125
View File
@@ -1,125 +0,0 @@
"""Assert-based tests for the RID API response parsing.
The parsing in ``fetch_water_data_for_date`` is the riskiest, previously
untested code: it maps the API's 1..24 "hourlytime" onto real timestamps
(hour 24 rolls to next-day midnight) and treats ``"***"``/``None`` discharge as
missing. These tests mock the HTTP call so no network is touched and stub the
validator so we assert on the parser's output directly.
"""
import datetime
from unittest.mock import MagicMock
import pytest
import src.water_scraper_v3 as scraper_mod
from src.water_scraper_v3 import EnhancedWaterMonitorScraper as Scraper
TARGET = datetime.datetime(2026, 7, 22)
@pytest.fixture
def make_scraper(monkeypatch):
"""Return a factory that builds a bare scraper returning the given API rows."""
def _factory(rows):
scraper = Scraper.__new__(Scraper) # bypass __init__ (no DB/network)
scraper.api_url = "https://example.invalid/api"
scraper.rate_limiter = MagicMock()
scraper.request_tracker = MagicMock()
scraper.station_config_path = "/nonexistent/stations.json"
scraper.station_mapping = scraper._load_station_mapping() # bundled defaults
response = MagicMock()
response.json.return_value = {"rows": rows}
response.raise_for_status.return_value = None
scraper.session = MagicMock()
scraper.session.post.return_value = response
# Isolate parsing from validation.
monkeypatch.setattr(
scraper_mod.DataValidator,
"validate_measurements",
staticmethod(lambda m: m),
)
return scraper
return _factory
def test_parses_water_level_and_discharge(make_scraper):
rows = [
{
"hourlytime": "9.00",
"wlvalues1": "3.50",
"qvalues1": "120.5",
"QPercent1": "45.2",
}
]
data = make_scraper(rows).fetch_water_data_for_date(TARGET)
p20 = [d for d in data if d["station_code"] == "P.20"]
assert len(p20) == 1
m = p20[0]
assert m["water_level"] == 3.5
assert m["discharge"] == 120.5
assert m["discharge_percent"] == 45.2
assert m["timestamp"] == datetime.datetime(2026, 7, 22, 9, 0)
assert m["station_name_en"] == "Ban Chiang Dao"
def test_discharge_asterisks_becomes_none(make_scraper):
rows = [{"hourlytime": "10.00", "wlvalues8": "4.20", "qvalues8": "***"}]
data = make_scraper(rows).fetch_water_data_for_date(TARGET)
p1 = [d for d in data if d["station_code"] == "P.1"][0]
assert p1["water_level"] == 4.2
assert p1["discharge"] is None
assert p1["discharge_percent"] is None
def test_hour_24_rolls_to_next_day_midnight(make_scraper):
rows = [{"hourlytime": "24.00", "wlvalues1": "3.00"}]
data = make_scraper(rows).fetch_water_data_for_date(TARGET)
assert data[0]["timestamp"] == datetime.datetime(2026, 7, 23, 0, 0)
def test_hours_1_to_23_stay_same_day(make_scraper):
rows = [
{"hourlytime": "1.00", "wlvalues1": "3.00"},
{"hourlytime": "23.00", "wlvalues1": "3.10"},
]
data = make_scraper(rows).fetch_water_data_for_date(TARGET)
times = sorted(d["timestamp"] for d in data)
assert times == [
datetime.datetime(2026, 7, 22, 1, 0),
datetime.datetime(2026, 7, 22, 23, 0),
]
def test_none_water_level_is_skipped(make_scraper):
rows = [{"hourlytime": "9.00", "wlvalues1": None, "wlvalues2": "2.5"}]
data = make_scraper(rows).fetch_water_data_for_date(TARGET)
codes = {d["station_code"] for d in data}
assert "P.20" not in codes # station 1 skipped (None water level)
assert "P.75" in codes # station 2 present
def test_out_of_range_and_empty_hours_skipped(make_scraper):
rows = [
{"hourlytime": "25.00", "wlvalues1": "3.0"},
{"hourlytime": "0.00", "wlvalues1": "3.0"},
{"hourlytime": "", "wlvalues1": "3.0"},
]
data = make_scraper(rows).fetch_water_data_for_date(TARGET)
assert data == []
def test_missing_rows_key_returns_empty(make_scraper):
scraper = make_scraper([])
scraper.session.post.return_value.json.return_value = {"unexpected": True}
assert scraper.fetch_water_data_for_date(TARGET) == []
tests/test_scraping_cycle.py
-307
View File
@@ -1,307 +0,0 @@
"""The RID scraping cycle must keep every reading of a late report.
RID's hourly report often lags 1-2 h in the evening. The cycle used to discard
a whole report unless it already held the current hour, and after 01:00 it
only fetched today's report, so the evening hours (and the "hour 24" =
00:00 reading) were lost for good. These tests pin the fixed behaviour with a
mocked HTTP session and DB adapter: no network, no database.
"""
import datetime
from unittest.mock import MagicMock
import pytest
import src.water_scraper_v3 as scraper_mod
from src.water_scraper_v3 import EnhancedWaterMonitorScraper as Scraper
TODAY = datetime.date(2026, 10, 17)
YESTERDAY = TODAY - datetime.timedelta(days=1)
def _thai(d: datetime.date) -> str:
return f"{d.day:02d}/{d.month:02d}/{d.year + 543}"
def _report(last_hour: int):
"""RID rows 1.00..24.00 with readings up to ``last_hour`` (rest null)."""
return [
{
"hourlytime": f"{h}.00",
"wlvalues8": 1.0 + h / 100 if h <= last_hour else None,
}
for h in range(1, 25)
]
@pytest.fixture
def make_scraper(monkeypatch):
monkeypatch.setattr(
scraper_mod.DataValidator,
"validate_measurements",
staticmethod(lambda m: m),
)
def _factory(reports, hours_by_day):
"""reports: {date: last published hour}; hours_by_day: DB contents."""
scraper = Scraper.__new__(Scraper) # bypass __init__ (no DB/network)
scraper.api_url = "https://example.invalid/api"
scraper.rate_limiter = MagicMock()
scraper.request_tracker = MagicMock()
scraper.station_config_path = "/nonexistent/stations.json"
scraper.station_mapping = scraper._load_station_mapping()
scraper.last_cycle = None
by_thai = {_thai(d): last for d, last in reports.items()}
def post(url, data, timeout):
response = MagicMock()
response.raise_for_status.return_value = None
last = by_thai.get(data["DW[TimeCurrent]"])
response.json.return_value = {
"rows": _report(last) if last is not None else []
}
return response
scraper.session = MagicMock()
scraper.session.post.side_effect = post
scraper.db_adapter = MagicMock()
scraper.db_adapter.get_recorded_hours_by_day.return_value = hours_by_day
scraper.db_adapter.save_measurements.return_value = True
return scraper
return _factory
def _fetched_dates(scraper):
return [
c.kwargs["data"]["DW[TimeCurrent]"] for c in scraper.session.post.call_args_list
]
def _saved_times(scraper):
saved = []
for c in scraper.db_adapter.save_measurements.call_args_list:
saved.extend(m["timestamp"] for m in c.args[0])
return sorted(set(saved))
def test_late_report_is_saved_but_not_current(make_scraper):
# 21:30, RID has only published up to 20:00 (the case that lost data)
scraper = make_scraper(
{TODAY: 20},
{YESTERDAY: set(range(24)), TODAY: set(range(0, 19))},
)
now = datetime.datetime(2026, 10, 17, 21, 30)
assert scraper.run_scraping_cycle(now=now) is False
saved = _saved_times(scraper)
assert datetime.datetime(2026, 10, 17, 19) in saved
assert datetime.datetime(2026, 10, 17, 20) in saved
assert scraper.last_cycle["latest_reading"] == datetime.datetime(2026, 10, 17, 20)
assert scraper.last_cycle["current"] is False
def test_current_hour_published_is_current(make_scraper):
scraper = make_scraper({TODAY: 21}, {YESTERDAY: set(range(24))})
now = datetime.datetime(2026, 10, 17, 21, 5)
assert scraper.run_scraping_cycle(now=now) is True
assert scraper.last_cycle["latest_reading"] == datetime.datetime(2026, 10, 17, 21)
def test_holes_in_yesterday_refetch_yesterdays_report(make_scraper):
# 02:10: yesterday's 22:00-23:00 and today's 00:00 (yesterday's hour 24)
# were published late; only yesterday's report still holds them
scraper = make_scraper(
{YESTERDAY: 24, TODAY: 2},
{YESTERDAY: set(range(22)), TODAY: {1}},
)
now = datetime.datetime(2026, 10, 17, 2, 10)
assert scraper.run_scraping_cycle(now=now) is True
assert _fetched_dates(scraper) == [_thai(YESTERDAY), _thai(TODAY)]
saved = _saved_times(scraper)
for hour in (
datetime.datetime(2026, 10, 16, 22),
datetime.datetime(2026, 10, 16, 23),
datetime.datetime(2026, 10, 17, 0),
):
assert hour in saved
def test_complete_yesterday_is_not_refetched(make_scraper):
scraper = make_scraper(
{TODAY: 15},
{YESTERDAY: set(range(24)), TODAY: set(range(15))},
)
scraper.run_scraping_cycle(now=datetime.datetime(2026, 10, 17, 15, 5))
assert _fetched_dates(scraper) == [_thai(TODAY)]
def test_before_one_am_uses_yesterdays_report(make_scraper):
# 00:20: today's report is empty; midnight is yesterday's "hour 24"
scraper = make_scraper({YESTERDAY: 24}, {YESTERDAY: set(range(24))})
assert scraper.run_scraping_cycle(now=datetime.datetime(2026, 10, 17, 0, 20))
assert _fetched_dates(scraper) == [_thai(YESTERDAY)]
assert scraper.last_cycle["latest_reading"] == datetime.datetime(2026, 10, 17, 0)
def test_nothing_published_returns_false_without_saving(make_scraper):
scraper = make_scraper({}, {YESTERDAY: set(range(24))})
assert (
scraper.run_scraping_cycle(now=datetime.datetime(2026, 10, 17, 9, 5)) is False
)
scraper.db_adapter.save_measurements.assert_not_called()
assert scraper.last_cycle["error"] == "no data fetched"
def test_gap_detection_failure_still_fetches_current_report(make_scraper):
scraper = make_scraper({TODAY: 9}, {})
scraper.db_adapter.get_recorded_hours_by_day.side_effect = RuntimeError("db down")
assert scraper.run_scraping_cycle(now=datetime.datetime(2026, 10, 17, 9, 5))
assert _fetched_dates(scraper) == [_thai(TODAY)]
class TestNextPollTime:
@pytest.fixture(autouse=True)
def _config(self, monkeypatch):
from src.config import Config
monkeypatch.setattr(Config, "SCRAPING_INTERVAL_HOURS", 1)
monkeypatch.setattr(Config, "RID_POLL_OFFSET_MINUTES", 5)
monkeypatch.setattr(Config, "RID_RETRY_MINUTES", 10)
def test_current_feed_waits_for_next_hour_plus_offset(self):
from src.web_api import _next_poll_time
now = datetime.datetime(2026, 10, 17, 21, 6)
assert _next_poll_time(now, True) == datetime.datetime(2026, 10, 17, 22, 5)
def test_behind_feed_retries_soon(self):
from src.web_api import _next_poll_time
now = datetime.datetime(2026, 10, 17, 21, 6)
assert _next_poll_time(now, False) == datetime.datetime(2026, 10, 17, 21, 16)
def test_retry_never_skips_the_hourly_poll(self):
from src.web_api import _next_poll_time
now = datetime.datetime(2026, 10, 17, 21, 58)
assert _next_poll_time(now, False) == datetime.datetime(2026, 10, 17, 22, 5)
def test_scraping_status_shared_through_file(tmp_path, monkeypatch):
from src import web_api
from src.config import Config
monkeypatch.setattr(Config, "SCRAPING_STATUS_FILE", str(tmp_path / "s.json"))
monkeypatch.setitem(web_api.app_state, "leader_lock", object())
monkeypatch.setitem(web_api.app_state, "is_scraping", False)
stats = dict(web_api.app_state["scraping_stats"])
stats.update(
total_runs=3,
successful_runs=2,
failed_runs=1,
feed_current=True,
latest_reading=datetime.datetime(2026, 10, 17, 21),
)
monkeypatch.setitem(web_api.app_state, "scraping_stats", stats)
web_api._publish_scraping_status()
shared = web_api._read_scraping_status()
assert shared["total_runs"] == 3
assert shared["latest_reading"] == "2026-10-17T21:00:00"
resp = web_api.ScrapingStatusResponse(
**{
k: v
for k, v in shared.items()
if k in web_api.ScrapingStatusResponse.model_fields
}
)
assert resp.latest_reading == datetime.datetime(2026, 10, 17, 21)
assert resp.feed_current is True
def test_loop_retries_while_behind_and_runs_hourly_work_once(monkeypatch):
"""Leader loop: 10-min retries while RID is behind; HII/dams/rain once per
clock hour; forecasts + ntfy only when newer readings arrive (or hourly)."""
import asyncio
from src import web_api
from src.config import Config
clock = datetime.datetime(2026, 10, 17, 21, 30)
class FixedClock(datetime.datetime):
@classmethod
def now(cls, tz=None):
return clock
class FakeScraper:
last_cycle = None
results = [
(False, datetime.datetime(2026, 10, 17, 20)),
(False, datetime.datetime(2026, 10, 17, 20)),
(True, datetime.datetime(2026, 10, 17, 21)),
]
def fill_data_gaps(self, days):
calls["gap_fill"] = days
return 0
def run_scraping_cycle(self):
current, latest = self.results.pop(0)
self.last_cycle = {"latest_reading": latest, "saved": 16}
return current
calls = {"hourly": 0, "precompute": 0, "notify": 0, "gap_fill": None}
sleeps = []
def counter(name):
async def _fn():
calls[name] += 1
return _fn
async def fake_sleep(seconds):
sleeps.append(seconds)
if len(sleeps) == 3:
raise asyncio.CancelledError
monkeypatch.setattr(web_api, "datetime", FixedClock)
monkeypatch.setattr(web_api.asyncio, "sleep", fake_sleep)
monkeypatch.setattr(web_api, "_collect_hourly_sources", counter("hourly"))
monkeypatch.setattr(web_api, "_precompute_forecasts", counter("precompute"))
monkeypatch.setattr(web_api, "_notify_transitions", counter("notify"))
monkeypatch.setattr(Config, "SCRAPING_INTERVAL_HOURS", 1)
monkeypatch.setattr(Config, "RID_POLL_OFFSET_MINUTES", 5)
monkeypatch.setattr(Config, "RID_RETRY_MINUTES", 10)
monkeypatch.setattr(Config, "GAP_FILL_DAYS_ON_START", 14)
monkeypatch.setitem(web_api.app_state, "scraper", FakeScraper())
monkeypatch.setitem(web_api.app_state, "leader_lock", None)
monkeypatch.setitem(web_api.app_state, "is_scraping", False)
monkeypatch.setitem(
web_api.app_state, "scraping_stats", dict(web_api.app_state["scraping_stats"])
)
asyncio.run(web_api.background_scraping_task())
assert calls == {"hourly": 1, "precompute": 2, "notify": 2, "gap_fill": 14}
assert sleeps == [600.0, 600.0, 2100.0] # retry, retry, then 22:05
stats = web_api.app_state["scraping_stats"]
assert (stats["total_runs"], stats["successful_runs"], stats["failed_runs"]) == (
3,
1,
2,
)
assert stats["latest_reading"] == datetime.datetime(2026, 10, 17, 21)
assert web_api.app_state["is_scraping"] is False
tests/test_station_management.py
-138
View File
@@ -1,138 +0,0 @@
#!/usr/bin/env python3
"""
Test script for station management API endpoints
"""
import requests
import json
def test_station_management():
"""Test the station management endpoints"""
base_url = "http://localhost:8000"
print("🧪 Testing Station Management API")
print("Make sure the API server is running: python run.py --web-api")
print()
# Test data for new station
new_station = {
"station_code": "P.TEST",
"thai_name": "สถานีทดสอบ",
"english_name": "Test Station",
"latitude": 18.7875,
"longitude": 99.0045,
"geohash": "w5q6uuhvfcfp25",
"status": "active"
}
try:
# 1. List existing stations
print("1. Listing existing stations...")
response = requests.get(f"{base_url}/stations")
if response.status_code == 200:
stations = response.json()
print(f"✅ Found {len(stations)} existing stations")
initial_count = len(stations)
else:
print(f"❌ Failed to list stations: {response.status_code}")
return
# 2. Create new station
print("\n2. Creating new test station...")
response = requests.post(
f"{base_url}/stations",
json=new_station,
headers={"Content-Type": "application/json"}
)
if response.status_code == 200:
created_station = response.json()
station_id = created_station["station_id"]
print(f"✅ Created station with ID: {station_id}")
print(f" Code: {created_station['station_code']}")
print(f" Name: {created_station['english_name']}")
else:
print(f"❌ Failed to create station: {response.status_code}")
print(f" Response: {response.text}")
return
# 3. Get specific station
print(f"\n3. Getting station details for ID {station_id}...")
response = requests.get(f"{base_url}/stations/{station_id}")
if response.status_code == 200:
station_details = response.json()
print(f"✅ Retrieved station details")
print(f" Thai name: {station_details['thai_name']}")
print(f" Coordinates: {station_details['latitude']}, {station_details['longitude']}")
else:
print(f"❌ Failed to get station: {response.status_code}")
# 4. Update station
print(f"\n4. Updating station {station_id}...")
update_data = {
"english_name": "Updated Test Station",
"thai_name": "สถานีทดสอบที่อัปเดต"
}
response = requests.put(
f"{base_url}/stations/{station_id}",
json=update_data,
headers={"Content-Type": "application/json"}
)
if response.status_code == 200:
updated_station = response.json()
print(f"✅ Updated station successfully")
print(f" New name: {updated_station['english_name']}")
else:
print(f"❌ Failed to update station: {response.status_code}")
print(f" Response: {response.text}")
# 5. Verify station count increased
print("\n5. Verifying station count...")
response = requests.get(f"{base_url}/stations")
if response.status_code == 200:
stations = response.json()
new_count = len(stations)
if new_count == initial_count + 1:
print(f"✅ Station count increased from {initial_count} to {new_count}")
else:
print(f"⚠️ Unexpected station count: {new_count} (expected {initial_count + 1})")
# 6. Delete test station
print(f"\n6. Deleting test station {station_id}...")
response = requests.delete(f"{base_url}/stations/{station_id}")
if response.status_code == 200:
result = response.json()
print(f"✅ Deleted station: {result['message']}")
else:
print(f"❌ Failed to delete station: {response.status_code}")
# 7. Verify station was deleted
print("\n7. Verifying station deletion...")
response = requests.get(f"{base_url}/stations/{station_id}")
if response.status_code == 404:
print("✅ Station successfully deleted (404 as expected)")
else:
print(f"⚠️ Station still exists: {response.status_code}")
# 8. Final station count check
response = requests.get(f"{base_url}/stations")
if response.status_code == 200:
stations = response.json()
final_count = len(stations)
if final_count == initial_count:
print(f"✅ Station count restored to original: {final_count}")
else:
print(f"⚠️ Station count mismatch: {final_count} (expected {initial_count})")
print("\n🎉 Station management tests completed!")
except requests.exceptions.ConnectionError:
print("❌ Connection failed - is the API server running?")
print("Start it with: python run.py --web-api")
except Exception as e:
print(f"❌ Test failed with error: {e}")
if __name__ == "__main__":
test_station_management()
tests/test_station_persistence.py
-77
View File
@@ -1,77 +0,0 @@
"""Assert-based tests for station-mapping persistence.
Station CRUD must survive restarts: the scraper loads its mapping from a
runtime-writable JSON file (falling back to bundled defaults) and writes it back
atomically. These tests exercise that load/save behaviour without constructing a
full scraper (which would open network/DB connections).
"""
from src.water_scraper_v3 import EnhancedWaterMonitorScraper as Scraper
def _bare_scraper(config_path):
"""A scraper instance with only the station-config attribute set.
Bypasses __init__ so no database/HTTP connection is attempted.
"""
scraper = Scraper.__new__(Scraper)
scraper.station_config_path = config_path
return scraper
def test_loads_bundled_defaults_when_runtime_file_absent(tmp_path):
scraper = _bare_scraper(str(tmp_path / "does_not_exist.json"))
mapping = scraper._load_station_mapping()
assert len(mapping) == 16
assert mapping["8"]["code"] == "P.1"
assert mapping["8"]["english_name"] == "Nawarat Bridge"
def test_save_then_reload_roundtrips_including_thai(tmp_path):
path = str(tmp_path / "stations.json")
scraper = _bare_scraper(path)
scraper.station_mapping = {
"1": {
"code": "P.99",
"thai_name": "สถานีทดสอบ",
"english_name": "Test Station",
"latitude": 1.0,
"longitude": 2.0,
"geohash": None,
}
}
assert scraper.save_stations() is True
reloaded = _bare_scraper(path)._load_station_mapping()
assert reloaded == scraper.station_mapping
assert reloaded["1"]["thai_name"] == "สถานีทดสอบ"
def test_runtime_file_takes_precedence_over_defaults(tmp_path):
path = str(tmp_path / "stations.json")
writer = _bare_scraper(path)
writer.station_mapping = {"1": {"code": "ONLY"}}
assert writer.save_stations() is True
mapping = _bare_scraper(path)._load_station_mapping()
assert list(mapping.keys()) == ["1"]
assert mapping["1"]["code"] == "ONLY"
def test_save_returns_false_without_a_path():
scraper = _bare_scraper("")
scraper.station_mapping = {}
assert scraper.save_stations() is False
def test_save_is_atomic_no_tmp_left_behind(tmp_path):
path = tmp_path / "stations.json"
scraper = _bare_scraper(str(path))
scraper.station_mapping = {"1": {"code": "P.1"}}
assert scraper.save_stations() is True
assert path.exists()
# The temp file used during the atomic write must not remain.
assert not (tmp_path / "stations.json.tmp").exists()
tests/test_thaiwater.py
-75
View File
@@ -1,75 +0,0 @@
from unittest.mock import MagicMock
from src.thaiwater import ThaiWaterClient
SAMPLE_RESPONSE = {
"data": {
"50": {
"type": "FeatureCollection",
"features": [
{
"geometry": {"type": "Point", "coordinates": [98.635262, 19.638411]},
"properties": {
"id": "123",
"waterlevelDatetime": "2026-08-09T15:00:00+07:00",
"waterlevelMsl": 742.34,
"storagePercent": 38.57,
"diffWlBank": 1.87,
"riverName": "Ping River",
"station": {
"stationCode": "G07003-P.65",
"station": "Ban Muang Pok",
},
"agency": {"agencyShort": "RID"},
"basin": {"basin": "Ping"},
},
},
{
"geometry": {"type": "Point", "coordinates": [100.1, 18.1]},
"properties": {
"id": "999",
"station": {"stationCode": "N.1", "station": "Nan station"},
"basin": {"basin": "Nan"},
},
},
],
}
}
}
def test_fetch_ping_sensors_normalizes_and_filters_basin():
session = MagicMock()
response = session.get.return_value
response.json.return_value = SAMPLE_RESPONSE
response.raise_for_status.return_value = None
sensors = ThaiWaterClient(session=session, api_key="public-key").fetch_ping_sensors()
assert sensors == [
{
"id": "thaiwater:123",
"station_code": "P.65",
"station_name": "Ban Muang Pok",
"latitude": 19.638411,
"longitude": 98.635262,
"timestamp": "2026-08-09T15:00:00+07:00",
"water_level_msl": 742.34,
"bank_percent": 38.57,
"distance_to_bank": 1.87,
"river_name": "Ping River",
"agency": "RID",
"source": "ThaiWater",
}
]
session.get.assert_called_once()
assert session.get.call_args.kwargs["headers"]["x-api-key"] == "public-key"
def test_fetch_ping_sensors_skips_features_without_coordinates():
response_data = {"data": {"50": {"features": [{"geometry": None, "properties": {"basin": {"basin": "Ping"}}}]}}}
session = MagicMock()
session.get.return_value.json.return_value = response_data
assert ThaiWaterClient(session=session, api_key="key").fetch_ping_sensors() == []