fix: RID collector keeps late readings instead of losing every evening
CI / Format & lint (push) Successful in 29s
CI / Test suite (push) Successful in 50s
Docs / Validate documentation (push) Successful in 23s
Security / Dependency vulnerabilities (push) Successful in 1m34s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m27s
CI / Format & lint (push) Successful in 29s
CI / Test suite (push) Successful in 50s
Docs / Validate documentation (push) Successful in 23s
Security / Dependency vulnerabilities (push) Successful in 1m34s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m27s
RID's hourly report often lags 1-2 h in the evening. The cycle discarded the whole report unless it already held the current hour, the web-API loop then slept a full hour, and after 01:00 only today's report was fetched. Result: the 19:00-24:00 readings were dropped most nights (P.1 lost 00:00 on 23 of the last 60 days, 13 evening hours this week), the feed went stale from ~21:00 until morning and ntfy ping-status sent a stale/recovered pair daily. - run_scraping_cycle() upserts every reading the report has; "current hour present" now only decides the retry cadence (returned as before). - Yesterday's report is re-fetched while the store has holes in it (hours 22-24 are only reachable through that day's report). - Leader loop polls at :05 (RID_POLL_OFFSET_MINUTES) and retries every RID_RETRY_MINUTES (10) while RID is behind; HII/dams/rain once per clock hour; forecast precompute + ntfy when newer readings arrive or hourly. - Startup re-fetches holes from the last GAP_FILL_DAYS_ON_START (14) days. - /scraping/status: the leader shares its stats via a JSON file, so any worker reports them (was always 0 runs); adds feed_current, latest_reading, readings_saved.
This commit is contained in:
1 parent
32399f1899
commit
04ebdaa931
7 files changed
+607
-166
No files matched your search
@@ -69,6 +69,13 @@ make test # pytest, synthetic data, no network
|
||||
make format # black + isort (the CI contract)
|
||||
```
|
||||
|
||||
RID collection in `--web-api` mode: polled at :05 past each hour
|
||||
(`RID_POLL_OFFSET_MINUTES`); while RID has not published the current hour yet,
|
||||
retried every `RID_RETRY_MINUTES` (10). Every poll stores all readings the report
|
||||
has, and yesterday's report is re-fetched while it still has holes (RID's evening
|
||||
hours often arrive 1-2 h late). On startup the leader re-fetches holes from the
|
||||
last `GAP_FILL_DAYS_ON_START` (14) days; for older history run `--fill-gaps all`.
|
||||
|
||||
## API
|
||||
|
||||
Read-only, no key, JSON. Base URL `https://water.buildfor.life`; timestamps are
|
||||
|
||||
Reference in new issue
Block a user