feat(rid): screen one-hour spikes; refetch and redo-from to repair the store
CI / Format & lint (push) Successful in 15s
CI / Test suite (push) Successful in 20s
Docs / Validate documentation (push) Successful in 9s
Rust / fmt, clippy, test, build (push) Successful in 3m6s

RID's report sometimes carries a wrong value for one hour and corrects it
later. P.1 read 10.6 m at 2026-10-25 00:00 between 1.04 and 1.07 (RID now
says 1.06); the store kept it, because a report is re-fetched only while it
has holes, and the model issued a 10.8 m forecast for that hour.

- rid::judge / rid::screen: a reading that leaves the previous one (within
  3 h) by more than 1 m (2 m on the flashy tributaries P.76, P.77, P.81,
  P.82, P.84, P.85, P.87) while the next reading comes back near the
  previous level is a spike: not stored, a stored copy is deleted. Limbs
  and steps the next reading confirms are kept; the newest reading waits
  for the next one when it jumps more than twice the limit. On 2018-2026
  history: 16 spikes found (slipped decimals, sign flips, -5/-6/-9 m
  placeholders), no genuine change rejected.
- Every store goes through it (rid::store: run_cycle, fill-gaps, refetch).
- `ping-collector refetch FROM [TO]`: re-fetch reports regardless of holes
  (corrects revised readings, fills the discharge of level-only evening
  hours, removes stored spikes) and print the catch-up command when
  readings changed.
- `catch-up-forecasts --redo-from T`: delete the model forecasts issued
  from T on, then recompute them.
- Tests: judge/screen unit tests; a DB test of refetch, the live hold and
  the forecast deletion. Checked end to end against RID on a scratch DB.
- Docs: DATA_SOURCES (spikes, level-only hours), FLOOD_FORECASTING,
  rust/README.
This commit is contained in:
grabowski committed 2026-10-26 15:04:01 +07:00
1 parent a610abcdb8
commit 9bf904b23c
8 files changed
+684 -15

No files matched your search

+56
View File
@@ -237,6 +237,62 @@ pub async fn save_rid(pool: &PgPool, readings: &[Reading], stations: &Stations)
Ok(rows.len())
}
/// Stored RID levels of report columns `nums` in [start, end], for the
/// screening (rid::screen).
pub async fn rid_readings_between(
pool: &PgPool,
nums: &[u32],
start: NaiveDateTime,
end: NaiveDateTime,
) -> Result<Vec<crate::rid::StoredReading>> {
let rows: Vec<(NaiveDateTime, i32, f64, Option<f64>)> = sqlx::query_as(
"SELECT timestamp, station_id, water_level::float8, discharge::float8
FROM water_measurements
WHERE station_id = ANY($1) AND timestamp >= $2 AND timestamp <= $3
AND water_level IS NOT NULL",
)
.bind(nums.iter().map(|&n| n as i32).collect::<Vec<_>>())
.bind(start)
.bind(end)
.fetch_all(pool)
.await
.context("reading stored RID levels")?;
Ok(rows
.into_iter()
.map(|(t, n, l, q)| (t, n as u32, l, q))
.collect())
}
/// Delete stored RID readings (timestamp, report column).
pub async fn delete_rid_readings(pool: &PgPool, keys: &[(NaiveDateTime, u32)]) -> Result<usize> {
if keys.is_empty() {
return Ok(0);
}
let r = sqlx::query(
"DELETE FROM water_measurements w
USING UNNEST($1::timestamp[], $2::int4[]) AS k(ts, sid)
WHERE w.timestamp = k.ts AND w.station_id = k.sid",
)
.bind(keys.iter().map(|(t, _)| *t).collect::<Vec<_>>())
.bind(keys.iter().map(|(_, n)| *n as i32).collect::<Vec<_>>())
.execute(pool)
.await
.context("deleting RID spikes")?;
Ok(r.rows_affected() as usize)
}
/// Delete the model rows of forecast_history issued at or after `from`, so
/// a catch-up recomputes them (after the readings they were computed from
/// changed).
pub async fn delete_model_forecasts_since(pool: &PgPool, from: NaiveDateTime) -> Result<usize> {
let r = sqlx::query("DELETE FROM forecast_history WHERE as_of >= $1 AND source = 'model'")
.bind(from)
.execute(pool)
.await
.context("deleting forecast_history rows")?;
Ok(r.rows_affected() as usize)
}
/// Day -> hours (0-23) that have at least one RID reading, for [start, end].
pub async fn recorded_hours_by_day(
pool: &PgPool,