feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s

rust/collector replaces the collection half of the Python web app: RID hourly
gauges (with the late-report semantics: store everything, retry every 10 min
while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain +
water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and
upsert semantics, one UNNEST statement per table, Bangkok wall-clock time
regardless of the server time zone.

Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and
instead re-forecast + notify when the newest RID reading in the database
advances (polled every FORECAST_POLL_SECONDS) and once per clock hour;
/scraping/status then serves the collector's status file; /scrape/trigger
answers 409.

Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against
PostgreSQL); live parity run of both collectors into two empty databases:
all 9 tables row-for-row identical (scripts/collector_parity.py); the Python
API served health, measurements, stats, status and model forecasts from a
database filled only by ping-collector. 177 Python tests pass.

reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which
rustls refuses.

Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service;
CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service,
release artifact).
This commit is contained in:
grabowski committed 2026-10-17 22:20:02 +07:00
1 parent e3694d7591
commit 4e619aa2ae
26 files changed
+5836 -15

No files matched your search

+23 -11
View File
@@ -1,6 +1,7 @@
# A Rust rewrite: assessment
Status: evaluation, nothing decided. Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
Status: **in progress.** Phase 1 (collector) is built and verified, deploy pending; see
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
## Short version
@@ -76,23 +77,34 @@ This slows model iteration, which is the most active part of the repo.
| One static binary; no venv on the server for the API | Server still needs Python + sklearn for the monthly retrain |
| Same stack as proplan | Feature work = Python + Rust change every time |
## Plan, if pursued (each phase ships on its own)
## Plan (each phase ships on its own)
| Phase | Scope | Status |
| --- | --- | --- |
| 0 | Spike: P.1 features + HGB inference | done, bit-identical |
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | built; live parity with Python on all 9 tables; deploy pending |
| 2 | Forecast crate + ntfy | next |
| 3 | Read API | |
| 4 | Cleanup | |
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
1. **Collector + notifier daemon** (~1.5k LOC): RID (the fixed semantics: store
late readings, retry while behind, re-fetch yesterday's holes), HII, dams,
Open-Meteo, ntfy, `notification_state`. The Python API runs with collection
off and becomes a stateless reader. Shadow one week first: compare stored
hours against RID upstream.
2. **Forecast crate** (~2k LOC): all 15 stations, dam/HII/rain inputs, heuristic
fallback, probabilities and stages, `forecast_history`; parity CI for every
station on a flood window and a dry window.
1. **Collector daemon** (`rust/collector`, ~1.5k LOC incl. tests): RID with the
fixed semantics (store late readings, retry while behind, re-fetch
yesterday's holes), HII, dams, Open-Meteo, into the same tables. The Python
API runs with `COLLECTION_ENABLED=false` and re-forecasts when the newest
RID reading in the database advances. ntfy moved to phase 2: the outlook
alert needs the forecasts, which are still Python. Cutover and rollback
are one command each (`scripts/install_collector.sh`).
2. **Forecast crate + ntfy** (~2k LOC): all 15 stations, dam/HII/rain inputs,
heuristic fallback, probabilities and stages, `forecast_history`,
`notification_state`; parity CI for every station on a flood window and a
dry window. The Python API then only serves.
3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
then a Caddy switch. Python keeps only training/evaluation.
4. **Cleanup:** drop the multi-backend adapters, demo databases and the
Python web stack.
Open questions before phase 1: is Matrix alerting still used? Must `/docs`
Open questions before phase 3: is Matrix alerting still used? Must `/docs`
(OpenAPI UI) stay? Is SQLite dev mode needed? Where does the monthly retrain
run if the server stops having a Python environment for the API?