perf: backfill one dam per request via the api/dam range endpoint
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 1m6s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Generate API Documentation (push) Successful in 12s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 4s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 34s
Documentation / Validate Documentation (push) Failing after 10s
Documentation / Build Sphinx Documentation (push) Successful in 22s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 1m6s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Documentation / Generate API Documentation (push) Successful in 12s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 4s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 34s
Documentation / Validate Documentation (push) Failing after 10s
Documentation / Build Sphinx Documentation (push) Successful in 22s
GET app.rid.go.th/reservoir/api/dam?dam_id&date_start&date_end returns a single dam's whole date range in one response — Mae Ngat's 2009-today archive is ~4 chunked requests instead of the ~2,900 one-day POSTs the all-dams path needs. Field names differ from api/dams and are mapped in parse_dam_range_records, verified equal on spot-checked dates; the range endpoint also carries DMD_ULevel, the reservoir level in m MSL that api/dams stopped publishing after ~2013. scripts/backfill_rid_reservoir.py defaults to the fast per-dam path (--all-dams keeps the full-fleet crawl, --refresh rewrites stored days). Already-stored dates are still skipped, junk values are still bounded, and the consecutive-failure abort still applies.
This commit is contained in:
@@ -1,14 +1,20 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Backfill rid_reservoir_daily with RID large-dam history (Mae Ngat et al.).
|
||||
|
||||
One request per day against app.rid.go.th/reservoir/api/dams (archive reaches
|
||||
back to at least 2009). Days already stored are skipped, so reruns only fetch
|
||||
what is missing — safe alongside the hourly live collector, and a rerun
|
||||
repairs holes left by transient failures.
|
||||
Two paths, both idempotent and both skipping what is already stored, so a
|
||||
rerun repairs holes left by transient failures and is safe alongside the
|
||||
hourly live collector:
|
||||
|
||||
--dam-id (default: Mae Ngat) one dam, whole range, via api/dam — a handful
|
||||
of requests for the entire 2009-today archive
|
||||
--all-dams all ~35 dams, one request per calendar day via
|
||||
api/dams — thousands of requests, ~25 minutes
|
||||
|
||||
Usage:
|
||||
uv run scripts/backfill_rid_reservoir.py # missing days since 2018-08-01
|
||||
uv run scripts/backfill_rid_reservoir.py --start 2015-01-01
|
||||
uv run scripts/backfill_rid_reservoir.py # Mae Ngat since 2018-08-01
|
||||
uv run scripts/backfill_rid_reservoir.py --start 2009-01-01 # full archive
|
||||
uv run scripts/backfill_rid_reservoir.py --refresh # rewrite stored days too
|
||||
uv run scripts/backfill_rid_reservoir.py --all-dams --start 2015-01-01
|
||||
uv run scripts/backfill_rid_reservoir.py --db-url postgresql://...
|
||||
"""
|
||||
|
||||
@@ -21,7 +27,12 @@ import sys
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
|
||||
from src.config import Config
|
||||
from src.rid_reservoir import RidReservoirStore, backfill
|
||||
from src.rid_reservoir import (
|
||||
MAE_NGAT_DAM_ID,
|
||||
RidReservoirStore,
|
||||
backfill,
|
||||
backfill_dam,
|
||||
)
|
||||
|
||||
DEFAULT_START = datetime.date(2018, 8, 1) # start of the water_measurements grid
|
||||
|
||||
@@ -34,6 +45,27 @@ def main(argv=None) -> int:
|
||||
parser.add_argument("--end", type=datetime.date.fromisoformat, default=None)
|
||||
parser.add_argument("--db-url", default=None)
|
||||
parser.add_argument("--throttle", type=float, default=0.4)
|
||||
parser.add_argument(
|
||||
"--dam-id",
|
||||
default=MAE_NGAT_DAM_ID,
|
||||
help="dam to backfill via the fast range endpoint (default Mae Ngat)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--all-dams",
|
||||
action="store_true",
|
||||
help="every dam, one request per calendar day (slow full-fleet path)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--chunk-days",
|
||||
type=int,
|
||||
default=1830,
|
||||
help="days per range request; the endpoint imposes no limit of its own",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--refresh",
|
||||
action="store_true",
|
||||
help="re-fetch days already stored (adds level_msl to api/dams rows)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
logging.basicConfig(
|
||||
@@ -59,10 +91,35 @@ def main(argv=None) -> int:
|
||||
return 1
|
||||
end = args.end or datetime.date.today()
|
||||
span_days = (end - args.start).days + 1
|
||||
missing = span_days - len(store.present_dates(args.start, end))
|
||||
saved = backfill(store, args.start, end, throttle_seconds=args.throttle)
|
||||
print(f"backfilled {saved} dam-day rows ({missing} days were missing)")
|
||||
return 0 if saved or missing == 0 else 1
|
||||
dam_id = None if args.all_dams else args.dam_id
|
||||
missing = span_days - len(store.present_dates(args.start, end, dam_id=dam_id))
|
||||
if args.all_dams:
|
||||
saved = backfill(store, args.start, end, throttle_seconds=args.throttle)
|
||||
print(f"backfilled {saved} dam-day rows ({missing} days were missing)")
|
||||
return 0 if saved or missing == 0 else 1
|
||||
|
||||
stats = {}
|
||||
saved = backfill_dam(
|
||||
store,
|
||||
dam_id=args.dam_id,
|
||||
start=args.start,
|
||||
end=end,
|
||||
chunk_days=args.chunk_days,
|
||||
throttle_seconds=max(args.throttle, 1.0),
|
||||
skip_present=not args.refresh,
|
||||
stats=stats,
|
||||
)
|
||||
still_missing = span_days - len(
|
||||
store.present_dates(args.start, end, dam_id=args.dam_id)
|
||||
)
|
||||
print(
|
||||
f"backfilled {saved} dam-day rows for dam {args.dam_id} "
|
||||
f"(requests: {stats.get('requests', 0)}, {missing} days were missing, "
|
||||
f"{still_missing} never published by the source)"
|
||||
)
|
||||
# A rerun saves nothing once the archive is complete — only a real
|
||||
# transport/database failure is an error here.
|
||||
return 1 if stats.get("aborted") or stats.get("failures") else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user