fix: survive junk RID dam values — widen storage_pct, bound inserts
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 8s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Documentation / Generate API Documentation (push) Successful in 9s
Documentation / Build Sphinx Documentation (push) Successful in 15s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 25s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 16s
Documentation / Validate Documentation (push) Failing after 8s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
Dam 100602 reports 87798% storage on some days, overflowing NUMERIC(6,2) and discarding the entire 33-dam daily batch. storage_pct is now NUMERIC(8,2) (auto-migrated on connect for existing Postgres/ MySQL tables) and every measure column is bounds-checked before insert so out-of-capacity junk becomes NULL instead of a batch-killing error. Rerunning the backfill repairs the days the overflow skipped.
This commit is contained in:
@@ -123,6 +123,35 @@ class TestStore:
|
||||
def test_save_empty(self, store):
|
||||
assert store.save([]) == 0
|
||||
|
||||
def test_junk_source_values_are_nulled_not_fatal(self, store):
|
||||
# Real junk from 2019-01-05: dam 100602 reported 87798% storage,
|
||||
# which overflowed NUMERIC(6,2) and discarded the whole 33-dam batch.
|
||||
payload = _dams_payload()
|
||||
payload["regions"][0]["dams"].append(
|
||||
{
|
||||
"DAM_ID": "100602",
|
||||
"DAM_Name": "junk",
|
||||
"DMD_Date": "2026-08-13",
|
||||
"DMD_QUse": "343292.00",
|
||||
"PERCENT_DMD_QUse": "87798.00",
|
||||
"DMD_Inflow": "0.59",
|
||||
"DMD_Outflow": "1e12", # beyond NUMERIC(10,2) -> NULL
|
||||
}
|
||||
)
|
||||
assert store.save(parse_dam_records(payload)) == 3
|
||||
from sqlalchemy import text
|
||||
|
||||
with store.engine.begin() as conn:
|
||||
row = conn.execute(
|
||||
text(
|
||||
"SELECT storage_mcm, storage_pct, outflow_mcm "
|
||||
"FROM rid_reservoir_daily WHERE dam_id = '100602'"
|
||||
)
|
||||
).fetchone()
|
||||
assert float(row[0]) == 343292.00 # fits NUMERIC(10,2), kept raw
|
||||
assert float(row[1]) == 87798.00 # fits widened NUMERIC(8,2)
|
||||
assert row[2] is None # beyond capacity -> NULL, batch survives
|
||||
|
||||
def test_present_dates(self, store):
|
||||
lo, hi = datetime.date(2026, 8, 1), datetime.date(2026, 8, 31)
|
||||
assert store.present_dates(lo, hi) == set()
|
||||
|
||||
Reference in New Issue
Block a user