B4L/Northern-Thailand-Ping-River-Monitor
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity
Files
d621aa9ce70043209b022795c6445b903def4933
Northern-Thailand-Ping-Rive…/models/eval_2026-09-12.json
T
grabowski d621aa9ce7
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 41s
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 17s
Details
Documentation / Validate Documentation (push) Failing after 16s
Details
Documentation / Generate API Documentation (push) Successful in 11s
Details
Documentation / Build Sphinx Documentation (push) Successful in 18s
Details
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Details
Documentation / Documentation Summary (push) Successful in 3s
Details
eval: quantile heads and fc48 on top of hgb-v3 (rejected/deferred); HII gauge-rain aggregate 
Rolling-origin harness gains rise_rain_quantile, rise_rain_quantile_uw,
rise_rain_qsigma (L2 point + quantile sigma) and rise_rain_fc48, all
opt-in, plus --from-cache for reproducible offline reruns. Results in
models/eval_2026-09-12*.json, write-up in docs/FLOOD_FORECASTING.md:

- quantile point prediction: better MAE, worse first-alert lead at 5 of
  11 events (P.103 2022-08-14 +6h -> +1h) -> rejected
- quantile sigma only: Brier within noise (0.0031 -> 0.0029) -> not worth 3x heads
- rain_fc48: neutral everywhere except 2024-10-03 P.1 (+21h -> +72h), n=1
  -> deferred to after the 2026 season

src/ml/hii_rain.py: catchment-mean hourly rain from the ~130 HII gauges in
the upper-Ping box and a 24h-sum comparison against Open-Meteo. Not a
training feature (table exists only since 2026-08-11, no archive); exposed
at GET /api/hii/rainfall/catchment so the two sources' agreement is on
record by the time a fold can test it.

data._read_cache now skips non-station files in models/cache/ (the shared
dir also holds rain_openmeteo / dam_* caches, which crashed the reader).
scripts/summarize_eval.py prints per-variant lead/peak-error tables.
2026-09-11 21:55:37 +02:00

723 lines
18 KiB
JSON
Raw Blame History

[
{
"station": "P.1",
"warn_thr": 3.7,
"folds": [
{
"year": 2021,
"n_train": 20024,
"n_test": 4392,
"events": [],
"variants": {
"rise_rain": {
"mae": 0.07375926701460789,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.07332598842242062,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.07383022350644125,
"mae_above_2p5": null,
"brier_warn": 1.0850721383440065e-12,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.06977345537342049,
"mae_above_2p5": null,
"brier_warn": 7.128994064266462e-16,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2022,
"n_train": 28784,
"n_test": 4392,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"peak_ts": "2022-10-03T15:00:00",
"peak_level": 4.65
}
],
"variants": {
"rise_rain": {
"mae": 0.08452847354970178,
"mae_above_2p5": 0.25082252888260664,
"brier_warn": 0.004300908725927739,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 5.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.8173954245046406
}
],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.08369773912557228,
"mae_above_2p5": 0.24702357745371217,
"brier_warn": 0.004325741658698973,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 5.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.8114190118860223
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.08230165237819607,
"mae_above_2p5": 0.2599241552494541,
"brier_warn": 0.004191550822623699,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 3.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.693734826616603
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.08181109208140971,
"mae_above_2p5": 0.2639358812546371,
"brier_warn": 0.004581181754314636,
"events": [
{
"crossing": "2022-10-02T19:00:00",
"lead_h": 2.0,
"peak_level": 4.65,
"peak_pred_24h_before": 3.620470606696475
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2023,
"n_train": 37539,
"n_test": 4392,
"events": [],
"variants": {
"rise_rain": {
"mae": 0.07696637416933236,
"mae_above_2p5": 0.10643655855133666,
"brier_warn": 1.919860722404625e-15,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.07708101918232377,
"mae_above_2p5": 0.09974894450126857,
"brier_warn": 1.7028444765622288e-15,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.06968496247786034,
"mae_above_2p5": 0.10210094973854984,
"brier_warn": 1.0987601008721297e-09,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.06726603345661021,
"mae_above_2p5": 0.09716644572204487,
"brier_warn": 2.7085590803510675e-09,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2024,
"n_train": 46323,
"n_test": 4392,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"peak_ts": "2024-09-26T02:00:00",
"peak_level": 4.93
},
{
"crossing": "2024-10-03T09:00:00",
"peak_ts": "2024-10-05T12:00:00",
"peak_level": 5.3
}
],
"variants": {
"rise_rain": {
"mae": 0.0890019157543212,
"mae_above_2p5": 0.2093245927883516,
"brier_warn": 0.005826169840715695,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 10.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.817519939833057
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 21.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.546588884631041
}
],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.08875916089292855,
"mae_above_2p5": 0.20975114218621593,
"brier_warn": 0.006061154593275248,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 11.0,
"peak_level": 4.93,
"peak_pred_24h_before": 4.800924141216692
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 72.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.577164984770105
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.08341835741803395,
"mae_above_2p5": 0.18688839374651373,
"brier_warn": 0.003188229521816099,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 17.0,
"peak_level": 4.93,
"peak_pred_24h_before": 5.058029430632501
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 21.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.3311787370709975
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.08412336465267245,
"mae_above_2p5": 0.19173218504592401,
"brier_warn": 0.0034722638888286116,
"events": [
{
"crossing": "2024-09-24T17:00:00",
"lead_h": 15.0,
"peak_level": 4.93,
"peak_pred_24h_before": 5.0118284217314
},
{
"crossing": "2024-10-03T09:00:00",
"lead_h": 21.0,
"peak_level": 5.3,
"peak_pred_24h_before": 5.3520431553190155
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2025,
"n_train": 55083,
"n_test": 4392,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"peak_ts": "2025-09-27T22:00:00",
"peak_level": 3.93
}
],
"variants": {
"rise_rain": {
"mae": 0.11202835461695825,
"mae_above_2p5": 0.2456782496767171,
"brier_warn": 0.005178356039745929,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 2.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.11083068059655521,
"mae_above_2p5": 0.23787013498709667,
"brier_warn": 0.005228043825289021,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 2.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.10372526043263834,
"mae_above_2p5": 0.23846363852091013,
"brier_warn": 0.005898259026876986,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 1.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.23
}
],
"false_alarm_episodes": 1
},
"rise_rain_quantile_uw": {
"mae": 0.10267267834487567,
"mae_above_2p5": 0.2507951319537275,
"brier_warn": 0.005020467408392599,
"events": [
{
"crossing": "2025-09-27T18:00:00",
"lead_h": 2.0,
"peak_level": 3.93,
"peak_pred_24h_before": 3.2417205711942434
}
],
"false_alarm_episodes": 1
}
}
}
]
},
{
"station": "P.103",
"warn_thr": 5.95,
"folds": [
{
"year": 2021,
"n_train": 20009,
"n_test": 4392,
"events": [],
"variants": {
"rise_rain": {
"mae": 0.14438683486790602,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.14305613024043953,
"mae_above_2p5": null,
"brier_warn": 0.0,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.13512418754716052,
"mae_above_2p5": null,
"brier_warn": 6.937198832251013e-10,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.10270057685061172,
"mae_above_2p5": null,
"brier_warn": 6.565945930999852e-14,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2022,
"n_train": 28769,
"n_test": 4392,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"peak_ts": "2022-08-14T08:00:00",
"peak_level": 6.09
},
{
"crossing": "2022-10-02T16:00:00",
"peak_ts": "2022-10-03T16:00:00",
"peak_level": 7.54
}
],
"variants": {
"rise_rain": {
"mae": 0.14392334606606189,
"mae_above_2p5": 0.41075382386412596,
"brier_warn": 0.00764679988213082,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 6.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.825270553204425
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 9.0,
"peak_level": 7.54,
"peak_pred_24h_before": 7.194064117976157
}
],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.14284722696431765,
"mae_above_2p5": 0.39401132538096667,
"brier_warn": 0.007427375799450148,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 6.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.845896993132048
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 9.0,
"peak_level": 7.54,
"peak_pred_24h_before": 7.194661123502819
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.1458954690776336,
"mae_above_2p5": 0.4674051188369517,
"brier_warn": 0.00899991317044724,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 1.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.82714042795344
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 5.0,
"peak_level": 7.54,
"peak_pred_24h_before": 6.59871451008115
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.13203958422483053,
"mae_above_2p5": 0.4902773906613983,
"brier_warn": 0.008514527559601331,
"events": [
{
"crossing": "2022-08-14T04:00:00",
"lead_h": 4.0,
"peak_level": 6.09,
"peak_pred_24h_before": 4.895369771408423
},
{
"crossing": "2022-10-02T16:00:00",
"lead_h": 3.0,
"peak_level": 7.54,
"peak_pred_24h_before": 6.294082735853337
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2023,
"n_train": 37524,
"n_test": 4392,
"events": [],
"variants": {
"rise_rain": {
"mae": 0.16195301512514512,
"mae_above_2p5": null,
"brier_warn": 3.0208441147560167e-07,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.16334224973870387,
"mae_above_2p5": null,
"brier_warn": 1.1626187140381066e-08,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.14755024879522577,
"mae_above_2p5": null,
"brier_warn": 3.850104714388072e-05,
"events": [],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.10589805327026759,
"mae_above_2p5": null,
"brier_warn": 1.8896757611081001e-07,
"events": [],
"false_alarm_episodes": 0
}
}
},
{
"year": 2024,
"n_train": 46308,
"n_test": 4058,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"peak_ts": "2024-09-26T00:00:00",
"peak_level": 8.27
},
{
"crossing": "2024-09-30T03:00:00",
"peak_ts": "2024-09-30T06:00:00",
"peak_level": 5.99
},
{
"crossing": "2024-10-03T06:00:00",
"peak_ts": "2024-10-05T07:00:00",
"peak_level": 9.93
}
],
"variants": {
"rise_rain": {
"mae": 0.13901842325128816,
"mae_above_2p5": 0.2887416310966684,
"brier_warn": 0.0072040857753137046,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 19.0,
"peak_level": 8.27,
"peak_pred_24h_before": 8.212734363860193
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 9.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.47565489914091
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 55.0,
"peak_level": 9.93,
"peak_pred_24h_before": 8.780278464915938
}
],
"false_alarm_episodes": 0
},
"rise_rain_fc48": {
"mae": 0.13850733053801131,
"mae_above_2p5": 0.29817392926900865,
"brier_warn": 0.007928846574280278,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 19.0,
"peak_level": 8.27,
"peak_pred_24h_before": 8.166020251020889
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 10.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.524584037259288
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 69.0,
"peak_level": 9.93,
"peak_pred_24h_before": 8.595120917488185
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile": {
"mae": 0.12640546558686208,
"mae_above_2p5": 0.27944115421093924,
"brier_warn": 0.007730268539879654,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 20.0,
"peak_level": 8.27,
"peak_pred_24h_before": 8.336487732683672
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 4.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.45284915024346
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 69.0,
"peak_level": 9.93,
"peak_pred_24h_before": 8.575915226697406
}
],
"false_alarm_episodes": 0
},
"rise_rain_quantile_uw": {
"mae": 0.12480975852168202,
"mae_above_2p5": 0.2857575557415695,
"brier_warn": 0.005894928998647136,
"events": [
{
"crossing": "2024-09-24T10:00:00",
"lead_h": 21.0,
"peak_level": 8.27,
"peak_pred_24h_before": 8.444720326882825
},
{
"crossing": "2024-09-30T03:00:00",
"lead_h": 9.0,
"peak_level": 5.99,
"peak_pred_24h_before": 5.42
},
{
"crossing": "2024-10-03T06:00:00",
"lead_h": 32.0,
"peak_level": 9.93,
"peak_pred_24h_before": 8.781001847167515
}
],
"false_alarm_episodes": 0
}
}
},
{
"year": 2025,
"n_train": 54577,
"n_test": 4392,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"peak_ts": "2025-09-27T21:00:00",
"peak_level": 6.64
},
{
"crossing": "2025-10-03T06:00:00",
"peak_ts": "2025-10-03T12:00:00",
"peak_level": 6.14
}
],
"variants": {
"rise_rain": {
"mae": 0.17206685418009837,
"mae_above_2p5": 0.3840520085986099,
"brier_warn": 0.015824852891785323,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 6.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73043665401637
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 8.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.609919760117381
}
],
"false_alarm_episodes": 2
},
"rise_rain_fc48": {
"mae": 0.1722274451362234,
"mae_above_2p5": 0.3872216974630654,
"brier_warn": 0.016245727130063177,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 4.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.7312263707793685
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 8.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.685511824758637
}
],
"false_alarm_episodes": 2
},
"rise_rain_quantile": {
"mae": 0.15912275848686555,
"mae_above_2p5": 0.3845695612674703,
"brier_warn": 0.014797606329811499,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 11.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.73
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 10.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.58133012298031
}
],
"false_alarm_episodes": 2
},
"rise_rain_quantile_uw": {
"mae": 0.15385355845103685,
"mae_above_2p5": 0.41314645854725585,
"brier_warn": 0.016528116336109958,
"events": [
{
"crossing": "2025-09-26T06:00:00",
"lead_h": 9.0,
"peak_level": 6.64,
"peak_pred_24h_before": 5.741254234340573
},
{
"crossing": "2025-10-03T06:00:00",
"lead_h": 8.0,
"peak_level": 6.14,
"peak_pred_24h_before": 5.418105405306207
}
],
"false_alarm_episodes": 2
}
}
}
]
}
]
Reference in New Issue View Git Blame Copy Permalink