feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s
rust/collector replaces the collection half of the Python web app: RID hourly gauges (with the late-report semantics: store everything, retry every 10 min while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain + water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and upsert semantics, one UNNEST statement per table, Bangkok wall-clock time regardless of the server time zone. Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and instead re-forecast + notify when the newest RID reading in the database advances (polled every FORECAST_POLL_SECONDS) and once per clock hour; /scraping/status then serves the collector's status file; /scrape/trigger answers 409. Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against PostgreSQL); live parity run of both collectors into two empty databases: all 9 tables row-for-row identical (scripts/collector_parity.py); the Python API served health, measurements, stats, status and model forecasts from a database filled only by ping-collector. 177 Python tests pass. reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which rustls refuses. Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service; CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service, release artifact).
This commit is contained in:
1 parent
e3694d7591
commit
4e619aa2ae
26 files changed
+5836
-15
No files matched your search
+23
-11
@@ -1,6 +1,7 @@
|
||||
# A Rust rewrite: assessment
|
||||
|
||||
Status: evaluation, nothing decided. Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
|
||||
Status: **in progress.** Phase 1 (collector) is built and verified, deploy pending; see
|
||||
[rust/README.md](../rust/README.md). Spike: [spikes/rust-forecast](../spikes/rust-forecast/README.md).
|
||||
|
||||
## Short version
|
||||
|
||||
@@ -76,23 +77,34 @@ This slows model iteration, which is the most active part of the repo.
|
||||
| One static binary; no venv on the server for the API | Server still needs Python + sklearn for the monthly retrain |
|
||||
| Same stack as proplan | Feature work = Python + Rust change every time |
|
||||
|
||||
## Plan, if pursued (each phase ships on its own)
|
||||
## Plan (each phase ships on its own)
|
||||
|
||||
| Phase | Scope | Status |
|
||||
| --- | --- | --- |
|
||||
| 0 | Spike: P.1 features + HGB inference | done, bit-identical |
|
||||
| 1 | `ping-collector`: RID, HII, dams, Open-Meteo | built; live parity with Python on all 9 tables; deploy pending |
|
||||
| 2 | Forecast crate + ntfy | next |
|
||||
| 3 | Read API | |
|
||||
| 4 | Cleanup | |
|
||||
|
||||
0. **Spike: done.** P.1 features + HGB inference, bit-identical.
|
||||
1. **Collector + notifier daemon** (~1.5k LOC): RID (the fixed semantics: store
|
||||
late readings, retry while behind, re-fetch yesterday's holes), HII, dams,
|
||||
Open-Meteo, ntfy, `notification_state`. The Python API runs with collection
|
||||
off and becomes a stateless reader. Shadow one week first: compare stored
|
||||
hours against RID upstream.
|
||||
2. **Forecast crate** (~2k LOC): all 15 stations, dam/HII/rain inputs, heuristic
|
||||
fallback, probabilities and stages, `forecast_history`; parity CI for every
|
||||
station on a flood window and a dry window.
|
||||
1. **Collector daemon** (`rust/collector`, ~1.5k LOC incl. tests): RID with the
|
||||
fixed semantics (store late readings, retry while behind, re-fetch
|
||||
yesterday's holes), HII, dams, Open-Meteo, into the same tables. The Python
|
||||
API runs with `COLLECTION_ENABLED=false` and re-forecasts when the newest
|
||||
RID reading in the database advances. ntfy moved to phase 2: the outlook
|
||||
alert needs the forecasts, which are still Python. Cutover and rollback
|
||||
are one command each (`scripts/install_collector.sh`).
|
||||
2. **Forecast crate + ntfy** (~2k LOC): all 15 stations, dam/HII/rain inputs,
|
||||
heuristic fallback, probabilities and stages, `forecast_history`,
|
||||
`notification_state`; parity CI for every station on a flood window and a
|
||||
dry window. The Python API then only serves.
|
||||
3. **Read API** (~2k LOC): routes recorded from the Python API as golden files,
|
||||
then a Caddy switch. Python keeps only training/evaluation.
|
||||
4. **Cleanup:** drop the multi-backend adapters, demo databases and the
|
||||
Python web stack.
|
||||
|
||||
Open questions before phase 1: is Matrix alerting still used? Must `/docs`
|
||||
Open questions before phase 3: is Matrix alerting still used? Must `/docs`
|
||||
(OpenAPI UI) stay? Is SQLite dev mode needed? Where does the monthly retrain
|
||||
run if the server stops having a Python environment for the API?
|
||||
|
||||
|
||||
Reference in new issue
Block a user