feat(rid): screen one-hour spikes; refetch and redo-from to repair the store
RID's report sometimes carries a wrong value for one hour and corrects it later. P.1 read 10.6 m at 2026-10-25 00:00 between 1.04 and 1.07 (RID now says 1.06); the store kept it, because a report is re-fetched only while it has holes, and the model issued a 10.8 m forecast for that hour. - rid::judge / rid::screen: a reading that leaves the previous one (within 3 h) by more than 1 m (2 m on the flashy tributaries P.76, P.77, P.81, P.82, P.84, P.85, P.87) while the next reading comes back near the previous level is a spike: not stored, a stored copy is deleted. Limbs and steps the next reading confirms are kept; the newest reading waits for the next one when it jumps more than twice the limit. On 2018-2026 history: 16 spikes found (slipped decimals, sign flips, -5/-6/-9 m placeholders), no genuine change rejected. - Every store goes through it (rid::store: run_cycle, fill-gaps, refetch). - `ping-collector refetch FROM [TO]`: re-fetch reports regardless of holes (corrects revised readings, fills the discharge of level-only evening hours, removes stored spikes) and print the catch-up command when readings changed. - `catch-up-forecasts --redo-from T`: delete the model forecasts issued from T on, then recompute them. - Tests: judge/screen unit tests; a DB test of refetch, the live hold and the forecast deletion. Checked end to end against RID on a scratch DB. - Docs: DATA_SOURCES (spikes, level-only hours), FLOOD_FORECASTING, rust/README.
This commit is contained in:
1 parent
a610abcdb8
commit
9bf904b23c
8 files changed
+684
-15
No files matched your search
@@ -237,6 +237,62 @@ pub async fn save_rid(pool: &PgPool, readings: &[Reading], stations: &Stations)
|
||||
Ok(rows.len())
|
||||
}
|
||||
|
||||
/// Stored RID levels of report columns `nums` in [start, end], for the
|
||||
/// screening (rid::screen).
|
||||
pub async fn rid_readings_between(
|
||||
pool: &PgPool,
|
||||
nums: &[u32],
|
||||
start: NaiveDateTime,
|
||||
end: NaiveDateTime,
|
||||
) -> Result<Vec<crate::rid::StoredReading>> {
|
||||
let rows: Vec<(NaiveDateTime, i32, f64, Option<f64>)> = sqlx::query_as(
|
||||
"SELECT timestamp, station_id, water_level::float8, discharge::float8
|
||||
FROM water_measurements
|
||||
WHERE station_id = ANY($1) AND timestamp >= $2 AND timestamp <= $3
|
||||
AND water_level IS NOT NULL",
|
||||
)
|
||||
.bind(nums.iter().map(|&n| n as i32).collect::<Vec<_>>())
|
||||
.bind(start)
|
||||
.bind(end)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.context("reading stored RID levels")?;
|
||||
Ok(rows
|
||||
.into_iter()
|
||||
.map(|(t, n, l, q)| (t, n as u32, l, q))
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Delete stored RID readings (timestamp, report column).
|
||||
pub async fn delete_rid_readings(pool: &PgPool, keys: &[(NaiveDateTime, u32)]) -> Result<usize> {
|
||||
if keys.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let r = sqlx::query(
|
||||
"DELETE FROM water_measurements w
|
||||
USING UNNEST($1::timestamp[], $2::int4[]) AS k(ts, sid)
|
||||
WHERE w.timestamp = k.ts AND w.station_id = k.sid",
|
||||
)
|
||||
.bind(keys.iter().map(|(t, _)| *t).collect::<Vec<_>>())
|
||||
.bind(keys.iter().map(|(_, n)| *n as i32).collect::<Vec<_>>())
|
||||
.execute(pool)
|
||||
.await
|
||||
.context("deleting RID spikes")?;
|
||||
Ok(r.rows_affected() as usize)
|
||||
}
|
||||
|
||||
/// Delete the model rows of forecast_history issued at or after `from`, so
|
||||
/// a catch-up recomputes them (after the readings they were computed from
|
||||
/// changed).
|
||||
pub async fn delete_model_forecasts_since(pool: &PgPool, from: NaiveDateTime) -> Result<usize> {
|
||||
let r = sqlx::query("DELETE FROM forecast_history WHERE as_of >= $1 AND source = 'model'")
|
||||
.bind(from)
|
||||
.execute(pool)
|
||||
.await
|
||||
.context("deleting forecast_history rows")?;
|
||||
Ok(r.rows_affected() as usize)
|
||||
}
|
||||
|
||||
/// Day -> hours (0-23) that have at least one RID reading, for [start, end].
|
||||
pub async fn recorded_hours_by_day(
|
||||
pool: &PgPool,
|
||||
|
||||
Reference in new issue
Block a user