fix: RID collector keeps late readings instead of losing every evening
CI / Format & lint (push) Successful in 29s
CI / Test suite (push) Successful in 50s
Docs / Validate documentation (push) Successful in 23s
Security / Dependency vulnerabilities (push) Successful in 1m34s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m27s

RID's hourly report often lags 1-2 h in the evening. The cycle discarded the
whole report unless it already held the current hour, the web-API loop then
slept a full hour, and after 01:00 only today's report was fetched. Result:
the 19:00-24:00 readings were dropped most nights (P.1 lost 00:00 on 23 of
the last 60 days, 13 evening hours this week), the feed went stale from
~21:00 until morning and ntfy ping-status sent a stale/recovered pair daily.

- run_scraping_cycle() upserts every reading the report has; "current hour
  present" now only decides the retry cadence (returned as before).
- Yesterday's report is re-fetched while the store has holes in it (hours
  22-24 are only reachable through that day's report).
- Leader loop polls at :05 (RID_POLL_OFFSET_MINUTES) and retries every
  RID_RETRY_MINUTES (10) while RID is behind; HII/dams/rain once per clock
  hour; forecast precompute + ntfy when newer readings arrive or hourly.
- Startup re-fetches holes from the last GAP_FILL_DAYS_ON_START (14) days.
- /scraping/status: the leader shares its stats via a JSON file, so any
  worker reports them (was always 0 runs); adds feed_current,
  latest_reading, readings_saved.
This commit is contained in:
grabowski committed 2026-10-17 21:45:09 +07:00
1 parent 32399f1899
commit 04ebdaa931
7 files changed
+607 -166

No files matched your search

+5
View File
@@ -53,6 +53,11 @@ THAIWATER_API_KEY=
# Data Collection Settings
SCRAPING_INTERVAL_HOURS=1
# RID poll minute past the hour; retry interval while RID is behind
RID_POLL_OFFSET_MINUTES=5
RID_RETRY_MINUTES=10
# Days of RID holes the leader re-fetches at startup (0 = off)
GAP_FILL_DAYS_ON_START=14
REQUEST_TIMEOUT=30
MAX_RETRIES=3
RETRY_DELAY_SECONDS=60