feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s

rust/collector replaces the collection half of the Python web app: RID hourly
gauges (with the late-report semantics: store everything, retry every 10 min
while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain +
water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and
upsert semantics, one UNNEST statement per table, Bangkok wall-clock time
regardless of the server time zone.

Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and
instead re-forecast + notify when the newest RID reading in the database
advances (polled every FORECAST_POLL_SECONDS) and once per clock hour;
/scraping/status then serves the collector's status file; /scrape/trigger
answers 409.

Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against
PostgreSQL); live parity run of both collectors into two empty databases:
all 9 tables row-for-row identical (scripts/collector_parity.py); the Python
API served health, measurements, stats, status and model forecasts from a
database filled only by ping-collector. 177 Python tests pass.

reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which
rustls refuses.

Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service;
CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service,
release artifact).
This commit is contained in:
grabowski committed 2026-10-17 22:20:02 +07:00
1 parent e3694d7591
commit 4e619aa2ae
26 files changed
+5836 -15

No files matched your search

+14 -1
View File
@@ -142,8 +142,20 @@ class Config:
# On startup the collection leader re-fetches RID reports with holes from
# the last N days (0 = off). Older history: `run.py --fill-gaps all`.
GAP_FILL_DAYS_ON_START = int(os.getenv("GAP_FILL_DAYS_ON_START", "14"))
# Leader -> other workers hand-off for /scraping/status (default: temp dir)
# Leader -> other workers hand-off for /scraping/status (default: temp dir).
# With ping-collector, point both services at the same file inside the app
# dir (the systemd units use PrivateTmp).
SCRAPING_STATUS_FILE = os.getenv("SCRAPING_STATUS_FILE", "")
# false: data collection runs in the Rust ping-collector service
# (rust/collector); this app then only serves, forecasts and notifies,
# re-forecasting when the newest RID reading advances (polled every
# FORECAST_POLL_SECONDS) and at least once per clock hour.
COLLECTION_ENABLED = os.getenv("COLLECTION_ENABLED", "true").lower() in (
"1",
"true",
"yes",
)
FORECAST_POLL_SECONDS = int(os.getenv("FORECAST_POLL_SECONDS", "60"))
# Logging settings
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
@@ -293,6 +305,7 @@ class Config:
"RID_POLL_OFFSET_MINUTES": cls.RID_POLL_OFFSET_MINUTES,
"RID_RETRY_MINUTES": cls.RID_RETRY_MINUTES,
"GAP_FILL_DAYS_ON_START": cls.GAP_FILL_DAYS_ON_START,
"COLLECTION_ENABLED": cls.COLLECTION_ENABLED,
"LOG_LEVEL": cls.LOG_LEVEL,
"LOG_FILE": cls.LOG_FILE,
"DATA_RETENTION_DAYS": cls.DATA_RETENTION_DAYS,