feat: --fill-gaps all repairs missing data across the whole DB range
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 29s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 14s
Documentation / Validate Documentation (push) Failing after 9s
Documentation / Generate API Documentation (push) Successful in 11s
Documentation / Build Sphinx Documentation (push) Successful in 17s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Cleanup (push) Successful in 1s
Documentation / Documentation Summary (push) Successful in 3s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Test Suite (3.11) (push) Failing after 29s
CI/CD Pipeline - Northern Thailand Ping River Monitor / Build Docker Image (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Integration Test with Services (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Staging (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Deploy to Production (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Performance Test (push) Skipped
CI/CD Pipeline - Northern Thailand Ping River Monitor / Code Quality (push) Successful in 14s
Documentation / Validate Documentation (push) Failing after 9s
Gap detection is now hour-granular: days with partial data (missing hourly slots) are re-fetched, not just days with no rows at all. The API's hour-24-is-next-midnight quirk is handled by re-fetching day D-1 when day D is missing its 00:00 slot. SQL adapters gain single-query range and recorded-hours lookups; non-SQL backends fall back to the old day-granular check.
This commit is contained in:
+23
-7
@@ -180,9 +180,12 @@ def run_continuous_monitoring():
|
||||
return True
|
||||
|
||||
|
||||
def run_gap_filling(days_back: int):
|
||||
"""Run gap filling for missing data"""
|
||||
logger.info(f"Checking for data gaps in the last {days_back} days...")
|
||||
def run_gap_filling(days_back: Optional[int]):
|
||||
"""Run gap filling for missing data (days_back=None scans the whole range)"""
|
||||
if days_back is None:
|
||||
logger.info("Checking for data gaps across the whole data range...")
|
||||
else:
|
||||
logger.info(f"Checking for data gaps in the last {days_back} days...")
|
||||
|
||||
try:
|
||||
# Validate configuration
|
||||
@@ -445,6 +448,7 @@ Examples:
|
||||
%(prog)s # Run continuous monitoring
|
||||
%(prog)s --web-api # Start web API server
|
||||
%(prog)s --fill-gaps 7 # Fill missing data for last 7 days
|
||||
%(prog)s --fill-gaps all # Fill missing data across the whole data range
|
||||
%(prog)s --update-data 2 # Update existing data for last 2 days
|
||||
%(prog)s --import-historical 2024-01-01 2024-01-31 # Import historical data
|
||||
%(prog)s --status # Show system status
|
||||
@@ -461,9 +465,11 @@ Examples:
|
||||
|
||||
parser.add_argument(
|
||||
"--fill-gaps",
|
||||
type=int,
|
||||
metavar="DAYS",
|
||||
help="Fill missing data gaps for the specified number of days back",
|
||||
metavar="DAYS|all",
|
||||
help=(
|
||||
"Fill missing data gaps for the specified number of days back, "
|
||||
"or 'all' to scan the entire data range in the database"
|
||||
),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
@@ -529,7 +535,17 @@ Examples:
|
||||
elif args.web_api:
|
||||
success = run_web_api()
|
||||
elif args.fill_gaps is not None:
|
||||
success = run_gap_filling(args.fill_gaps)
|
||||
if args.fill_gaps.lower() == "all":
|
||||
success = run_gap_filling(None)
|
||||
else:
|
||||
try:
|
||||
success = run_gap_filling(int(args.fill_gaps))
|
||||
except ValueError:
|
||||
logger.error(
|
||||
f"Invalid --fill-gaps value '{args.fill_gaps}': "
|
||||
"expected a number of days or 'all'"
|
||||
)
|
||||
sys.exit(1)
|
||||
elif args.update_data is not None:
|
||||
success = run_data_update(args.update_data)
|
||||
elif args.import_historical is not None:
|
||||
|
||||
Reference in New Issue
Block a user