B4L/Northern-Thailand-Ping-River-Monitor
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

chore: remove the Python service; Python is the training stack only

Browse Source 
Phase 4 of docs/RUST_REWRITE.md. Since 2026-10-18 ping-collector (rust/)
collects, forecasts, notifies (ntfy + Matrix) and serves the website/API;
nothing in production ran the Python service code any more.

Removed (~17k lines): the FastAPI app (web_api, schemas, health_check,
metrics, postgres_history, forecast_history, thaiwater), the Python RID
collector and CLI (water_scraper_v3, main, run.py), the notifier and Matrix
alerting (notify, alerting), the multi-backend adapters (SQLite, MySQL,
InfluxDB, VictoriaMetrics) with demo databases, models/validators/exceptions/
logging/rate limiter, ml/skill.py (served by Rust now), the migration tooling
(api_parity, collector_parity, e2e_notify, install_collector.sh,
gen_notify_cases.py, spikes/rust-forecast), Docker/compose/Grafana files,
sql/, the release workflow and the docs for all of that. Their tests went
with them.

Kept in Python: src/ml (training, evaluation harness, export; predict.py as
the reference the Rust forecaster is tested against), the HII and RID-dam
clients/stores for the archive backfills (their live collectors removed),
the backfill scripts, retrain.sh + timer, forecast_parity.py.

- PostgreSQL only: src/config.py is database_url() from POSTGRES_* / .env;
  the stores lost their sqlite/mysql branches (and the 2026-08 column
  migration). The database tests run against a throwaway schema from
  TEST_DATABASE_URL (tests/conftest.py pg_url, skipped without it), as the
  Rust tests do; CI gets a PostgreSQL service. 100 tests (66 without a DB).
- The notifier settings moved from src/notify.py into src/ml/export.py;
  flood_config.json regenerates byte-identical.
- tests/test_dashboard.py checks every path the dashboard fetches against
  ping-collector's router.
- Dependencies: fastapi, uvicorn, pydantic, influxdb, pymysql, psutil,
  schedule, sphinx, pyinstaller, mypy, pytest-asyncio dropped; python-dotenv
  1.1.1 explicit (it came in through uvicorn[standard]).
- Formatting contract extended to scripts/ and tests/ (reformatted once).
- scripts/install.sh is the one installer (fresh host or update, in place in
  a checkout without touching its ownership): binary, units, uv sync, .env
  defaults, removes the retired water-monitor.service and any old
  --alert-check cron job, dry run + forecast check, restart, readiness wait.
- Docs, CLAUDE.md, CONTRIBUTING.md, .env.example, Makefile, unit comments
  and Rust doc comments updated to the Rust service.
This commit is contained in:
grabowski committed 2026-10-18 17:15:07 +07:00
1 parent 330fdf7804
commit 7e642555b4
125 files changed
+1300 -19558
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CONTRIBUTING.md
+42 -268
View File
@@ -1,277 +1,51 @@
# Contributing to Northern Thailand Ping River Monitor
# Contributing
Thank you for your interest in contributing to the Northern Thailand Ping River Monitor! This document provides guidelines and information for contributors.
Issues and merge requests on
[git.b4l.co.th](https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor).
The repository has two halves: the service in Rust (`rust/`, see
[rust/README.md](rust/README.md)) and the model training/evaluation stack in Python
(`src/ml`, see [docs/FLOOD_FORECASTING.md](docs/FLOOD_FORECASTING.md)).
## 🚀 Getting Started
## Setup
### Prerequisites
- Python 3.9 or higher
- Git
- Basic knowledge of water monitoring systems
- Familiarity with FastAPI and time-series databases (optional)
- Python 3.11 and [uv](https://docs.astral.sh/uv/) (3.12+ has no pandas 2.0.3 wheels)
- Rust >= 1.88 from [rustup](https://rustup.rs), plus `pkg-config` and `libssl-dev` on Linux
- Docker (or any PostgreSQL 16) for the database tests
### Development Setup
```bash
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
cd Northern-Thailand-Ping-River-Monitor
make install # uv sync + pre-commit hooks
docker run -d --name ping-pg -e POSTGRES_PASSWORD=ping -e POSTGRES_DB=ping -p 55432:5432 postgres:16
export TEST_DATABASE_URL=postgres://postgres:ping@localhost:55432/ping
make check # format check, flake8, clippy, pytest, cargo test, Rust 1.88
```
1. **Clone the repository:**
```bash
git clone https://git.b4l.co.th/B4L/Northern-Thailand-Ping-River-Monitor.git
cd Northern-Thailand-Ping-River-Monitor
```
Without `TEST_DATABASE_URL` the database tests are skipped, the rest still run.
2. **Create a virtual environment:**
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
## Before a merge request
3. **Install dependencies:**
```bash
pip install -r requirements.txt
```
- `make format`, then `make check` green. CI runs the same (`.gitea/workflows/`).
- One logical change per commit, imperative subject (`fix(collector): ...`,
`feat(web): ...`, `docs: ...`).
- Dashboard changes (`src/static/dashboard.html`): every visible string in English
and Thai (`t()` table); a screenshot in the merge request.
- API changes: keep existing fields and formats; external clients read them. New
parameters go into the route's declaration in `rust/collector/src/web/params.rs`,
which also documents them at `/docs`.
- Model changes: evaluate with `scripts/evaluate_variants.py` (rolling origin, judged
on first-alert lead and false alarms) and record the result in
`docs/FLOOD_FORECASTING.md` section 5, whether it helped or not. A feature change
must be ported to `rust/forecast` in the same change (see CLAUDE.md for the
fixture and config regeneration steps).
- Never commit credentials, `.env`, trained models (`models/*.joblib`,
`models/*.json.gz`) or the server's private addresses.
4. **Run tests:**
```bash
python test_integration.py
```
## Useful commands
5. **Start the development server:**
```bash
python run.py --web-api
```
## 🔧 Development Guidelines
### Code Style
- Follow PEP 8 Python style guidelines
- Use type hints for all function parameters and return values
- Write descriptive docstrings for all classes and functions
- Keep functions focused and under 50 lines when possible
### Testing
- Write tests for new features
- Ensure all existing tests pass before submitting
- Test both success and error cases
- Include integration tests for API endpoints
### Documentation
- Update README.md for new features
- Add docstrings to all new functions and classes
- Update API documentation for new endpoints
- Include examples in documentation
## 📝 Contribution Process
### 1. Issue Creation
- Check existing issues before creating new ones
- Use clear, descriptive titles
- Provide detailed descriptions with steps to reproduce (for bugs)
- Include system information and logs when relevant
### 2. Pull Request Process
1. **Fork the repository**
2. **Create a feature branch:**
```bash
git checkout -b feature/your-feature-name
```
3. **Make your changes**
4. **Test thoroughly:**
```bash
python test_integration.py
python test_station_management.py
```
5. **Commit with clear messages:**
```bash
git commit -m "Add: New station validation feature"
```
6. **Push to your fork:**
```bash
git push origin feature/your-feature-name
```
7. **Create a pull request**
### Pull Request Guidelines
- Use clear, descriptive titles
- Provide detailed descriptions of changes
- Reference related issues
- Include screenshots for UI changes
- Ensure all tests pass
- Update documentation as needed
## 🎯 Areas for Contribution
### High Priority
- **Additional Database Support** - MongoDB, TimescaleDB
- **Authentication System** - User management and API keys
- **Real-time Alerts** - Email/SMS notifications for critical conditions
- **Mobile App** - React Native or Flutter interface
- **Advanced Analytics** - Trend analysis and forecasting
### Medium Priority
- **Bulk Station Operations** - Import/export multiple stations
- **Map Interface** - Visual station management
- **Data Export** - CSV, Excel format support
- **Performance Optimization** - Query optimization and caching
- **Internationalization** - Multi-language support
### Low Priority
- **Additional Visualizations** - Custom chart types
- **Report Generation** - Automated PDF reports
- **Integration APIs** - Third-party service connections
- **Advanced Logging** - Structured logging with ELK stack
## 🐛 Bug Reports
When reporting bugs, please include:
1. **Environment Information:**
- Operating system and version
- Python version
- Database type and version
- Browser (for web interface issues)
2. **Steps to Reproduce:**
- Clear, numbered steps
- Expected vs actual behavior
- Screenshots or error messages
3. **Additional Context:**
- Log files (sanitized of sensitive data)
- Configuration details
- Recent changes to the system
## 💡 Feature Requests
For feature requests, please provide:
1. **Use Case:** Why is this feature needed?
2. **Description:** What should the feature do?
3. **Acceptance Criteria:** How do we know it's complete?
4. **Priority:** How important is this feature?
5. **Alternatives:** Are there workarounds available?
## 🔒 Security
### Reporting Security Issues
- **DO NOT** create public issues for security vulnerabilities
- Email security issues to: [security contact]
- Include detailed information about the vulnerability
- Allow time for fixes before public disclosure
### Security Guidelines
- Never commit sensitive data (passwords, API keys)
- Use environment variables for configuration
- Validate all user inputs
- Follow OWASP security guidelines
- Keep dependencies updated
## 📊 Database Contributions
### Adding New Database Support
1. Create adapter in `src/database_adapters.py`
2. Implement all required methods
3. Add configuration options in `src/config.py`
4. Update documentation
5. Add integration tests
### Database Schema Changes
- Provide migration scripts
- Test with existing data
- Document breaking changes
- Update all database adapters
## 🌐 API Contributions
### Adding New Endpoints
1. Define Pydantic models in `src/models.py`
2. Implement endpoint in `src/web_api.py`
3. Add input validation
4. Include error handling
5. Write comprehensive tests
6. Update API documentation
### API Guidelines
- Use RESTful conventions
- Include proper HTTP status codes
- Provide clear error messages
- Support pagination for large datasets
- Include rate limiting considerations
## 📈 Performance Guidelines
### Optimization Areas
- Database query efficiency
- API response times
- Memory usage
- Concurrent request handling
- Data processing speed
### Monitoring
- Use built-in metrics collection
- Monitor response times
- Track error rates
- Watch resource usage
- Set up alerts for issues
## 🧪 Testing Guidelines
### Test Types
- **Unit Tests:** Individual function testing
- **Integration Tests:** Component interaction testing
- **API Tests:** Endpoint functionality testing
- **Performance Tests:** Load and stress testing
### Test Coverage
- Aim for >80% code coverage
- Test both success and failure paths
- Include edge cases
- Test with different database backends
## 📚 Documentation Standards
### Code Documentation
- Use clear, descriptive variable names
- Write comprehensive docstrings
- Include type hints
- Add inline comments for complex logic
### User Documentation
- Keep README.md updated
- Provide clear installation instructions
- Include usage examples
- Document configuration options
## 🤝 Community Guidelines
### Communication
- Be respectful and professional
- Help newcomers get started
- Share knowledge and experiences
- Provide constructive feedback
### Code Reviews
- Review code thoroughly
- Provide specific, actionable feedback
- Be open to suggestions
- Focus on code quality and maintainability
## 📞 Getting Help
### Resources
- **Documentation:** README.md and docs/
- **API Reference:** http://localhost:8000/docs
- **Issues:** GitHub Issues section
- **Discussions:** GitHub Discussions
### Contact
- **General Questions:** Create a GitHub issue
- **Security Issues:** [security contact]
- **Maintainer:** [maintainer contact]
## 🎉 Recognition
Contributors will be recognized in:
- README.md contributors section
- Release notes for significant contributions
- Special recognition for major features
Thank you for contributing to the Northern Thailand Ping River Monitor! Your contributions help improve water resource monitoring and management in the region.
```bash
make build && make serve # the website/API from your database on 127.0.0.1:8001
uv run python scripts/dev_proxy.py # the working-copy dashboard against the live API
uv run python scripts/evaluate_variants.py --stations P.1
uv run python -m src.ml.export --config # after changing thresholds or notifier settings
```