feat(rust): ping-collector, phase 1 of the Rust rewrite
CI / Format & lint (push) Successful in 19s
CI / Test suite (push) Successful in 47s
Docs / Validate documentation (push) Successful in 19s
Rust / fmt, clippy, test, build (push) Successful in 7m35s
Security / Dependency vulnerabilities (push) Successful in 1m37s
Security / Static analysis (push) Successful in 17s
Security / License report (push) Successful in 1m30s

rust/collector replaces the collection half of the Python web app: RID hourly
gauges (with the late-report semantics: store everything, retry every 10 min
while RID is behind, re-fetch yesterday's holes, startup gap fill), HII rain +
water level, RID dams (today + yesterday) and Open-Meteo rain. Same tables and
upsert semantics, one UNNEST statement per table, Bangkok wall-clock time
regardless of the server time zone.

Python side: COLLECTION_ENABLED=false makes the API leader stop collecting and
instead re-forecast + notify when the newest RID reading in the database
advances (polled every FORECAST_POLL_SECONDS) and once per clock hour;
/scraping/status then serves the collector's status file; /scrape/trigger
answers 409.

Verified: 20 Rust tests (parsers, gap/schedule logic, database tests against
PostgreSQL); live parity run of both collectors into two empty databases:
all 9 tables row-for-row identical (scripts/collector_parity.py); the Python
API served health, measurements, stats, status and model forecasts from a
database filled only by ping-collector. 177 Python tests pass.

reqwest uses native-tls: app.rid.go.th offers only TLS 1.2 CBC suites, which
rustls refuses.

Deploy/cutover/rollback: scripts/install_collector.sh, scripts/ping-collector.service;
CI: .gitea/workflows/rust.yml (fmt, clippy, tests with a PostgreSQL service,
release artifact).
This commit is contained in:
grabowski committed 2026-10-17 22:20:02 +07:00
1 parent e3694d7591
commit 4e619aa2ae
26 files changed
+5836 -15

No files matched your search

+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# Install the Rust collector (rust/collector, `ping-collector`) and switch
# data collection over to it, or back.
#
# sudo bash scripts/install_collector.sh # install binary + unit, dry-run check
# sudo bash scripts/install_collector.sh --cutover # install, then Rust collects:
# # COLLECTION_ENABLED=false, start
# # ping-collector, restart water-monitor
# sudo bash scripts/install_collector.sh --rollback # Python collects again
#
# The binary comes from BIN=/path/to/ping-collector (e.g. the "ping-collector-
# linux-x86_64" CI artifact) or from rust/target/release/ping-collector, built
# beforehand as a normal user:
# cargo build --release --locked --manifest-path rust/Cargo.toml
# (needs rustup, pkg-config and libssl-dev).
set -euo pipefail
APP_DIR="${APP_DIR:-/opt/thailand-water-monitor}"
ENV_FILE="${APP_DIR}/.env"
STATUS_FILE="${APP_DIR}/collector-status.json"
REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MODE="${1:-install}"
log() { printf '\033[1;32m==>\033[0m %s\n' "$*"; }
die() { printf '\033[1;31m[error]\033[0m %s\n' "$*" >&2; exit 1; }
[ "$(id -u)" -eq 0 ] || die "run as root (sudo)"
[ -f "$ENV_FILE" ] || die "$ENV_FILE not found"
set_env() {
local key="$1" value="$2"
if grep -q "^${key}=" "$ENV_FILE"; then
sed -i "s|^${key}=.*|${key}=${value}|" "$ENV_FILE"
else
printf '%s=%s\n' "$key" "$value" >>"$ENV_FILE"
fi
}
if [ "$MODE" = "--rollback" ]; then
log "Stopping ping-collector; the Python app collects again"
systemctl disable --now ping-collector.service 2>/dev/null || true
set_env COLLECTION_ENABLED true
systemctl restart water-monitor.service
log "Done. Check: journalctl -u water-monitor -n 50"
exit 0
fi
case "$MODE" in install | --cutover) ;; *) die "unknown option: $MODE" ;; esac
BIN="${BIN:-${REPO_DIR}/rust/target/release/ping-collector}"
[ -x "$BIN" ] || die "no binary at $BIN: build it (see header) or pass BIN=/path/to/ping-collector"
log "Installing $BIN -> ${APP_DIR}/bin/ping-collector ($("$BIN" --version))"
install -D -m 0755 -o root -g root "$BIN" "${APP_DIR}/bin/ping-collector"
install -m 0644 "${REPO_DIR}/scripts/ping-collector.service" /etc/systemd/system/ping-collector.service
systemctl daemon-reload
# Both services read and write /scraping/status through this file (inside
# the app dir: the units use PrivateTmp).
set_env SCRAPING_STATUS_FILE "$STATUS_FILE"
touch "$STATUS_FILE"
chown "$(stat -c %U:%G "$APP_DIR")" "$STATUS_FILE"
log "Dry run against the configured database (writes nothing)"
(cd "$APP_DIR" && "${APP_DIR}/bin/ping-collector" --env-file "$ENV_FILE" once --dry-run)
if [ "$MODE" = "--cutover" ]; then
log "Cutover: ping-collector collects, the Python app forecasts and notifies"
set_env COLLECTION_ENABLED false
systemctl enable --now ping-collector.service
systemctl restart water-monitor.service
sleep 20
systemctl --no-pager --lines=0 status ping-collector.service || true
journalctl -u ping-collector -n 20 --no-pager || true
log "Rollback if needed: sudo bash scripts/install_collector.sh --rollback"
else
log "Installed, not started. Switch over with: sudo bash scripts/install_collector.sh --cutover"
fi