fix(forecast): no nightly gaps in the issued model forecasts
CI / Format & lint (push) Successful in 28s
CI / Test suite (push) Successful in 1m4s
Docs / Validate documentation (push) Successful in 17s
Rust / fmt, clippy, test, build (push) Successful in 13m1s

The P.1 chart's "24 h peak, as issued" line had a gap every night and a
dip at 21:00. Two causes:

- A run forecasts from the newest reading only. RID publishes 22:00..06:00
  in one batch in the morning (and now and then one hour late), so those
  hours were never forecast from. After each run the collector now
  forecasts the hours of the last 48 h (two weeks on the first run after a
  start) that have a RID reading but no model forecast, each as a run at
  that hour would have: gauges, HII fill and dam state up to that hour.
  Rain is fetched once, so a late hour's rain_fc24 partly holds rain that
  has since fallen (as in training). Only missing (hour, station) pairs are
  written. `ping-collector catch-up-forecasts --hours N` goes further back.
- Overnight each hourly run re-issues the 21:00 reading, after 6 h as the
  heuristic, which overwrote the 21:00 model row. A heuristic row no longer
  replaces a model row in forecast_history.

The dashboard draws only model rows on the issued line.

Checked on a local copy of 18 days of RID data: the catch-up of an hour
matches `ping-collector forecast --now <hour>`; 288 hours took 20 s
(debug build). Tests for both database rules.
This commit is contained in:
grabowski committed 2026-10-20 15:21:48 +07:00
1 parent cf10f2e4c8
commit 180d95c6bd
10 files changed
+311 -3

No files matched your search

+35 -1
View File
@@ -267,6 +267,34 @@ pub async fn earliest_rid_timestamp(pool: &PgPool) -> Result<Option<NaiveDateTim
)
}
/// (hour, station) pairs in `[start, end)` where one of `codes` has a RID
/// level but no model forecast: hours that arrived together with a later
/// one (RID's overnight batch, a late report), so no run forecast from them.
pub async fn hours_without_model_forecast(
pool: &PgPool,
codes: &[String],
start: NaiveDateTime,
end: NaiveDateTime,
) -> Result<Vec<(NaiveDateTime, String)>> {
sqlx::query_as(
"SELECT DISTINCT date_trunc('hour', m.timestamp) AS h, s.station_code::text
FROM water_measurements m JOIN stations s ON s.id = m.station_id
WHERE m.timestamp >= $1 AND m.timestamp < $2 AND m.water_level IS NOT NULL
AND s.station_code = ANY($3)
AND NOT EXISTS (
SELECT 1 FROM forecast_history f
WHERE f.station_code = s.station_code
AND f.as_of = date_trunc('hour', m.timestamp) AND f.source = 'model')
ORDER BY 1, 2",
)
.bind(start)
.bind(end)
.bind(codes)
.fetch_all(pool)
.await
.context("finding hours without a model forecast")
}
pub async fn latest_rid_timestamp(pool: &PgPool) -> Result<Option<NaiveDateTime>> {
Ok(
sqlx::query_scalar("SELECT MAX(timestamp) FROM water_measurements")
@@ -665,6 +693,11 @@ pub async fn latest_rid_readings(pool: &PgPool) -> Result<Vec<LatestReading>> {
}
/// `ForecastHistoryStore.save_rows`: one row per (as_of, station, horizon).
/// Store issued forecasts; a later run's row for the same hour replaces the
/// earlier one, except that a heuristic row never replaces a model row. The
/// hourly run re-issues the newest reading's hour while RID is silent
/// overnight, and after 6 h (`stale_after_h`) it does so as the heuristic:
/// without the exception the 21:00 model forecast was overwritten each night.
pub async fn save_forecast_history(
pool: &PgPool,
rows: &[ping_forecast::Forecast],
@@ -683,7 +716,8 @@ pub async fn save_forecast_history(
ON CONFLICT (as_of, station_code, horizon_hours) DO UPDATE SET
predicted_max_level = EXCLUDED.predicted_max_level, p_warning = EXCLUDED.p_warning,
p_danger = EXCLUDED.p_danger, current_level = EXCLUDED.current_level,
model_version = EXCLUDED.model_version, source = EXCLUDED.source",
model_version = EXCLUDED.model_version, source = EXCLUDED.source
WHERE forecast_history.source IS DISTINCT FROM 'model' OR EXCLUDED.source = 'model'",
)
.bind(rows.iter().map(|r| r.as_of).collect::<Vec<_>>())
.bind(