Station CRUD via the API previously mutated the scraper's in-memory
station_mapping only, so changes were lost on restart (and the systemd
service auto-restarts).
- Extract the 130-line hardcoded station_mapping into bundled defaults at
src/data/stations.json; the scraper loads from a runtime-writable config
file (STATION_CONFIG_PATH, default stations.json) and falls back to the
bundled defaults to seed it.
- Add scraper.save_stations() with an atomic temp-file + os.replace write.
- create/update/delete station endpoints now persist and roll back the
in-memory change if the write fails; re-raise HTTPException so persistence
errors surface as real 500s instead of being swallowed.
- Backend-agnostic (works for the VictoriaMetrics deployment, which has no
relational stations table). Runtime stations.json is gitignored.
Also clears pre-existing flake8 debt in water_scraper_v3.py (unused imports,
long lines, duplicate logging import) and dedupes the User-Agent to
Config.USER_AGENT.
Features:
- Real-time water level monitoring for Ping River Basin (16 stations)
- Coverage from Chiang Dao to Nakhon Sawan in Northern Thailand
- FastAPI web interface with interactive dashboard and station management
- Multi-database support (SQLite, MySQL, PostgreSQL, InfluxDB, VictoriaMetrics)
- Comprehensive monitoring with health checks and metrics collection
- Docker deployment with Grafana integration
- Production-ready architecture with enterprise-grade observability
CI/CD & Automation:
- Complete Gitea Actions workflows for CI/CD, security, and releases
- Multi-Python version testing (3.9-3.12)
- Multi-architecture Docker builds (amd64, arm64)
- Daily security scanning and dependency monitoring
- Automated documentation generation
- Performance testing and validation
Production Ready:
- Type safety with Pydantic models and comprehensive type hints
- Data validation layer with range checking and error handling
- Rate limiting and request tracking for API protection
- Enhanced logging with rotation, colors, and performance metrics
- Station management API for dynamic CRUD operations
- Comprehensive documentation and deployment guides
Technical Stack:
- Python 3.9+ with FastAPI and Pydantic
- Multi-database architecture with adapter pattern
- Docker containerization with multi-stage builds
- Grafana dashboards for visualization
- Gitea Actions for CI/CD automation
- Enterprise monitoring and alerting
Ready for deployment to B4L infrastructure!
grabowski