feat(rid): screen one-hour spikes; refetch and redo-from to repair the store
CI / Format & lint (push) Successful in 15s
CI / Test suite (push) Successful in 20s
Docs / Validate documentation (push) Successful in 9s
Rust / fmt, clippy, test, build (push) Successful in 3m6s

RID's report sometimes carries a wrong value for one hour and corrects it
later. P.1 read 10.6 m at 2026-10-25 00:00 between 1.04 and 1.07 (RID now
says 1.06); the store kept it, because a report is re-fetched only while it
has holes, and the model issued a 10.8 m forecast for that hour.

- rid::judge / rid::screen: a reading that leaves the previous one (within
  3 h) by more than 1 m (2 m on the flashy tributaries P.76, P.77, P.81,
  P.82, P.84, P.85, P.87) while the next reading comes back near the
  previous level is a spike: not stored, a stored copy is deleted. Limbs
  and steps the next reading confirms are kept; the newest reading waits
  for the next one when it jumps more than twice the limit. On 2018-2026
  history: 16 spikes found (slipped decimals, sign flips, -5/-6/-9 m
  placeholders), no genuine change rejected.
- Every store goes through it (rid::store: run_cycle, fill-gaps, refetch).
- `ping-collector refetch FROM [TO]`: re-fetch reports regardless of holes
  (corrects revised readings, fills the discharge of level-only evening
  hours, removes stored spikes) and print the catch-up command when
  readings changed.
- `catch-up-forecasts --redo-from T`: delete the model forecasts issued
  from T on, then recompute them.
- Tests: judge/screen unit tests; a DB test of refetch, the live hold and
  the forecast deletion. Checked end to end against RID on a scratch DB.
- Docs: DATA_SOURCES (spikes, level-only hours), FLOOD_FORECASTING,
  rust/README.
This commit is contained in:
grabowski committed 2026-10-26 15:04:01 +07:00
1 parent a610abcdb8
commit 9bf904b23c
8 files changed
+684 -15

No files matched your search

+20 -1
View File
@@ -23,10 +23,29 @@ The only source persisted to the database and used by the ML pipeline.
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports) |
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports with holes, `refetch FROM [TO]` any reports) |
Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
**Spikes.** The report sometimes carries a wrong value for one hour and corrects
it later: a slipped decimal point (P.1 10.6 m between 1.04 and 1.07 on 2026-10-25
00:00, later 1.06; P.85 0.101 for 1.1), a flipped sign (P.4A ±1.18, P.67 ±0.6) or
a placeholder (−5, −6, −9, 50 m). The collector re-fetches a report only while it
has holes, so a later correction never arrived. Since 2026-10-26 every batch is
screened before it is stored (`rid::judge`, `rid::screen`): a reading that leaves
the previous one (within 3 h) by more than 1 m (2 m on the flashy tributaries
P.76, P.77, P.81, P.82, P.84, P.85, P.87) while the next reading comes back near
the previous level is not stored, and a stored copy is deleted. A rising or falling
limb, or a step the next reading confirms, is kept. The newest reading of a gauge
waits for the next one when it jumps more than twice the limit. On the 2018–2026
history the rule finds 16 spikes and would have rejected no genuine change (the
largest genuine 1-h changes: 1.1 m on the mainstem, 3.1 m at P.76). Two wrong hours
in a row are not caught.
Evening hours often arrive level-only (no discharge) and gain the discharge
later; the API's history fills a missing discharge with 20 × level (an old
placeholder, kept for compatibility). `refetch` brings in RID's discharge.
---
## 2. ThaiWater / HII ecosystem
+4 -1
View File
@@ -643,7 +643,10 @@ Open-Meteo's estimate of rain that has since fallen, as in training, where
the forward rain is observed rain. Such rows are written only where no model
row exists; `created_at` tells them apart (more than an hour after `as_of`).
`ping-collector catch-up-forecasts --hours N` does the same further back (up
to 60 days).
to 60 days); with `--redo-from T` it first deletes the model forecasts issued
from T on, to recompute them after `ping-collector refetch` corrected the
readings they came from (a bad reading reaches the features for 72 h through
`roll_max_72`).
A model row in `forecast_history` is never replaced by a heuristic one. While
RID is silent overnight each hourly run re-issues the 21:00 reading, and once