feat(rid): screen one-hour spikes; refetch and redo-from to repair the store
RID's report sometimes carries a wrong value for one hour and corrects it later. P.1 read 10.6 m at 2026-10-25 00:00 between 1.04 and 1.07 (RID now says 1.06); the store kept it, because a report is re-fetched only while it has holes, and the model issued a 10.8 m forecast for that hour. - rid::judge / rid::screen: a reading that leaves the previous one (within 3 h) by more than 1 m (2 m on the flashy tributaries P.76, P.77, P.81, P.82, P.84, P.85, P.87) while the next reading comes back near the previous level is a spike: not stored, a stored copy is deleted. Limbs and steps the next reading confirms are kept; the newest reading waits for the next one when it jumps more than twice the limit. On 2018-2026 history: 16 spikes found (slipped decimals, sign flips, -5/-6/-9 m placeholders), no genuine change rejected. - Every store goes through it (rid::store: run_cycle, fill-gaps, refetch). - `ping-collector refetch FROM [TO]`: re-fetch reports regardless of holes (corrects revised readings, fills the discharge of level-only evening hours, removes stored spikes) and print the catch-up command when readings changed. - `catch-up-forecasts --redo-from T`: delete the model forecasts issued from T on, then recompute them. - Tests: judge/screen unit tests; a DB test of refetch, the live hold and the forecast deletion. Checked end to end against RID on a scratch DB. - Docs: DATA_SOURCES (spikes, level-only hours), FLOOD_FORECASTING, rust/README.
This commit is contained in:
1 parent
a610abcdb8
commit
9bf904b23c
8 files changed
+684
-15
No files matched your search
+20
-1
@@ -23,10 +23,29 @@ The only source persisted to the database and used by the ML pipeline.
|
||||
| Params | `DW[UtokID]=1`, `DW[BasinID]=6` (Ping), `DW[TimeCurrent]=<Buddhist-calendar date>`, `rows=100` |
|
||||
| Variables | Water level (m, gauge datum), discharge (m³/s, `'***'` = malformed), discharge % of channel capacity |
|
||||
| Stations | 16 P-series gauges (P.1 anchor at Nawarat Bridge; see `src/data/stations.json`) |
|
||||
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports) |
|
||||
| Client | `rust/collector/src/rid.rs` (`ping-collector`; `fill-gaps` re-fetches old reports with holes, `refetch FROM [TO]` any reports) |
|
||||
|
||||
Human-facing page: <https://hyd-app-db.rid.go.th/hydro1h.html>
|
||||
|
||||
**Spikes.** The report sometimes carries a wrong value for one hour and corrects
|
||||
it later: a slipped decimal point (P.1 10.6 m between 1.04 and 1.07 on 2026-10-25
|
||||
00:00, later 1.06; P.85 0.101 for 1.1), a flipped sign (P.4A ±1.18, P.67 ±0.6) or
|
||||
a placeholder (−5, −6, −9, 50 m). The collector re-fetches a report only while it
|
||||
has holes, so a later correction never arrived. Since 2026-10-26 every batch is
|
||||
screened before it is stored (`rid::judge`, `rid::screen`): a reading that leaves
|
||||
the previous one (within 3 h) by more than 1 m (2 m on the flashy tributaries
|
||||
P.76, P.77, P.81, P.82, P.84, P.85, P.87) while the next reading comes back near
|
||||
the previous level is not stored, and a stored copy is deleted. A rising or falling
|
||||
limb, or a step the next reading confirms, is kept. The newest reading of a gauge
|
||||
waits for the next one when it jumps more than twice the limit. On the 2018–2026
|
||||
history the rule finds 16 spikes and would have rejected no genuine change (the
|
||||
largest genuine 1-h changes: 1.1 m on the mainstem, 3.1 m at P.76). Two wrong hours
|
||||
in a row are not caught.
|
||||
|
||||
Evening hours often arrive level-only (no discharge) and gain the discharge
|
||||
later; the API's history fills a missing discharge with 20 × level (an old
|
||||
placeholder, kept for compatibility). `refetch` brings in RID's discharge.
|
||||
|
||||
---
|
||||
|
||||
## 2. ThaiWater / HII ecosystem
|
||||
|
||||
Reference in new issue
Block a user