feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
rust/collector replaces the collection half of the Python web app: RID hourly gauges (with the late-report semantics: store everything, retry every 10 min while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain + water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and upsert semantics, one UNNEST statement per table, Bangkok wall-clock time regardless of the server time zone. Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and instead re-forecast + notify when the newest RID reading in the database advances (polled every FORECAST_POLL_SECONDS) and once per clock hour; /scraping/status then serves the collector's status file; /scrape/trigger answers 409. Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against PostgreSQL); live parity run of both collectors into two empty databases: all 9 tables row-for-row identical (scripts/collector_parity.py); the Python API served health, measurements, stats, status and model forecasts from a database filled only by ping-collector. 177 Python tests pass. reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which rustls refuses. Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service; CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service, release artifact).
This commit is contained in:
1 parent
e3694d7591
commit
4e619aa2ae
26 files changed
+5836
-15
No files matched your search
+14
-1
@@ -142,8 +142,20 @@ class Config:
|
||||
# On startup the collection leader re-fetches RID reports with holes from
|
||||
# the last N days (0 = off). Older history: `run.py --fill-gaps all`.
|
||||
GAP_FILL_DAYS_ON_START = int(os.getenv("GAP_FILL_DAYS_ON_START", "14"))
|
||||
# Leader -> other workers hand-off for /scraping/status (default: temp dir)
|
||||
# Leader -> other workers hand-off for /scraping/status (default: temp dir).
|
||||
# With ping-collector, point both services at the same file inside the app
|
||||
# dir (the systemd units use PrivateTmp).
|
||||
SCRAPING_STATUS_FILE = os.getenv("SCRAPING_STATUS_FILE", "")
|
||||
# false: data collection runs in the Rust ping-collector service
|
||||
# (rust/collector); this app then only serves, forecasts and notifies,
|
||||
# re-forecasting when the newest RID reading advances (polled every
|
||||
# FORECAST_POLL_SECONDS) and at least once per clock hour.
|
||||
COLLECTION_ENABLED = os.getenv("COLLECTION_ENABLED", "true").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
)
|
||||
FORECAST_POLL_SECONDS = int(os.getenv("FORECAST_POLL_SECONDS", "60"))
|
||||
|
||||
# Logging settings
|
||||
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
|
||||
@@ -293,6 +305,7 @@ class Config:
|
||||
"RID_POLL_OFFSET_MINUTES": cls.RID_POLL_OFFSET_MINUTES,
|
||||
"RID_RETRY_MINUTES": cls.RID_RETRY_MINUTES,
|
||||
"GAP_FILL_DAYS_ON_START": cls.GAP_FILL_DAYS_ON_START,
|
||||
"COLLECTION_ENABLED": cls.COLLECTION_ENABLED,
|
||||
"LOG_LEVEL": cls.LOG_LEVEL,
|
||||
"LOG_FILE": cls.LOG_FILE,
|
||||
"DATA_RETENTION_DAYS": cls.DATA_RETENTION_DAYS,
|
||||
|
||||
Reference in new issue
Block a user