fix: survive junk RID dam values — widen storage_pct, bound inserts
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 8s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s

Dam 100602 reports 87798% storage on some days, overflowing
NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct
is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/
MySQL tables) and every measure column is bounds-checked before insert
so out-of-capacity junk becomes NULL instead of a batch-killing error.
Rerunning the backfill repairs the days the overflow skipped.
This commit is contained in:
2026-08-13 10:50:58 +07:00
parent ba781465a9
commit 6af6fbe02c
2 changed files with 69 additions and 2 deletions
+29
View File
@@ -123,6 +123,35 @@ class TestStore:
def test_save_empty(self, store):
assert store.save([]) == 0
def test_junk_source_values_are_nulled_not_fatal(self, store):
# Real junk from 2019-01-05: dam 100602 reported 87798% storage,
# which overflowed NUMERIC(6,2) and discarded the whole 33-dam batch.
payload = _dams_payload()
payload["regions"][0]["dams"].append(
{
"DAM_ID": "100602",
"DAM_Name": "junk",
"DMD_Date": "2026-08-13",
"DMD_QUse": "343292.00",
"PERCENT_DMD_QUse": "87798.00",
"DMD_Inflow": "0.59",
"DMD_Outflow": "1e12", # beyond NUMERIC(10,2) -> NULL
}
)
assert store.save(parse_dam_records(payload)) == 3
from sqlalchemy import text
with store.engine.begin() as conn:
row = conn.execute(
text(
"SELECT storage_mcm, storage_pct, outflow_mcm "
"FROM rid_reservoir_daily WHERE dam_id = '100602'"
)
).fetchone()
assert float(row[0]) == 343292.00 # fits NUMERIC(10,2), kept raw
assert float(row[1]) == 87798.00 # fits widened NUMERIC(8,2)
assert row[2] is None # beyond capacity -> NULL, batch survives
def test_present_dates(self, store):
lo, hi = datetime.date(2026, 8, 1), datetime.date(2026, 8, 31)
assert store.present_dates(lo, hi) == set()