B4L/Northern-Thailand-Ping-River-Monitor
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

ci: green pipelines that check what exists; one formatting contract

Browse Source 
The Test Suite job failed on every push since the black check was added
because the tree had never been formatted, and pre-commit said 120
columns while CI ran black's default 88. pyproject.toml now carries
[tool.black] / [tool.isort] (88, black profile) as the single source;
pre-commit reads it; `make format` applied it (13 files, whitespace only,
146 insertions / 128 deletions, tests unchanged at 146 passed).

ci.yml: lint (black, isort, flake8 hard errors) + pytest. The Docker
registry push, VictoriaMetrics integration test, staging/production
deploy and Apache-Bench jobs were template scaffolding for hosts and
registries that do not exist; production is a systemd unit updated by
git pull. Removed rather than left permanently skipped.

docs.yml: the "Check markdown links" step curl'd every URL in every .md
and failed on localhost examples and the Tailscale IP, and the Sphinx
jobs built artifacts nobody read. Replaced by two checks that mean
something: relative links/images in README, CONTRIBUTING and docs/
resolve inside the repo, and the FastAPI OpenAPI schema exports with
the documented endpoints present (uploaded as an artifact).
This commit is contained in:
grabowski
2026-09-11 23:05:37 +02:00
parent 6f4a86edbb
commit 5ad8e4eac3
19 changed files with 290 additions and 807 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/ci.yml
+60 -327
View File
@@ -1,342 +1,75 @@
name: CI/CD Pipeline - Northern Thailand Ping River Monitor
name: CI
# What this checks, on every push and PR to master:
# 1. formatting contract (black + isort, config in pyproject.toml)
# 2. flake8 hard-error gate (syntax, undefined names)
# 3. the pytest suite (synthetic data, no DB/network; ~1 min)
# Docker build / staging / production / perf jobs from the original template
# were removed: there is no registry, no staging host, and production is a
# systemd unit deployed by `git pull` on the server (docs/FLOOD_FORECASTING.md
# section 6, scripts/install.sh). Re-add a job when the thing it deploys exists.
on:
push:
branches: [ master, develop ]
branches: [master, develop]
pull_request:
branches: [ master ]
branches: [master]
schedule:
# Run tests daily at 2 AM UTC
- cron: '0 2 * * *'
# daily, catches dependency drift / upstream API changes in the tests
- cron: "0 2 * * *"
workflow_dispatch:
env:
PYTHON_VERSION: '3.11'
REGISTRY: git.b4l.co.th
IMAGE_NAME: b4l/northern-thailand-ping-river-monitor
# GitHub token for better rate limits and authentication
GH_TOKEN: ${{ secrets.GH_TOKEN }}
PYTHON_VERSION: "3.11" # pandas 2.0.3 ships no 3.12 wheels; psycopg2-binary 2.9.9 breaks on 3.13
jobs:
# Test job
lint:
name: Format & lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install tools
run: |
python -m pip install --upgrade pip --root-user-action=ignore
pip install --root-user-action=ignore black==23.11.0 isort==5.12.0 flake8==6.1.0
- name: black
run: black --check --diff src/ *.py
- name: isort
run: isort --check-only --diff src/ *.py
- name: flake8 (errors only)
run: flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
test:
name: Test Suite
name: Test suite
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.11'] # pandas 2.0.3 ships no 3.12 wheels; widen after upgrading pandas
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
python -m pip install --upgrade pip --root-user-action=ignore
pip install --root-user-action=ignore -r requirements.txt
pip install --root-user-action=ignore -r requirements-dev.txt
- name: Lint with flake8
run: |
flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
flake8 src/ --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
- name: Type check with mypy (advisory)
run: |
# 86 pre-existing errors; blocking typing gate deferred until the debt is paid down
mypy src/ --ignore-missing-imports || true
- name: Format check with black
run: |
black --check src/ *.py
- name: Import sort check
run: |
isort --check-only src/ *.py
- name: Run integration tests
run: |
python tests/test_integration.py
- name: Run station management tests
run: |
python tests/test_station_management.py
- name: Test application startup
run: |
timeout 10s python run.py --test || true
- name: Security scan with bandit
run: |
bandit -r src/ -f json -o bandit-report.json || true
- name: Upload test artifacts
uses: actions/upload-artifact@v3
if: always()
with:
name: test-results-${{ matrix.python-version }}
path: |
bandit-report.json
*.log
- uses: actions/checkout@v4
# Code quality job
code-quality:
name: Code Quality
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip --root-user-action=ignore
pip install --root-user-action=ignore -r requirements-dev.txt
- name: Run safety check
run: |
safety check -r requirements.txt --json --output safety-report.json || true
- name: Run bandit security scan
run: |
bandit -r src/ -f json -o bandit-report.json || true
- name: Upload security reports
uses: actions/upload-artifact@v3
with:
name: security-reports
path: |
safety-report.json
bandit-report.json
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: pip
cache-dependency-path: |
requirements.txt
requirements-dev.txt
# Build Docker image
build:
name: Build Docker Image
runs-on: ubuntu-latest
needs: test
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITEA_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=sha,prefix={{branch}}-
type=raw,value=latest,enable={{is_default_branch}}
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
env:
GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
- name: Test Docker image
run: |
docker run --rm ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} python run.py --test
- name: Install dependencies
run: |
python -m pip install --upgrade pip --root-user-action=ignore
pip install --root-user-action=ignore -r requirements.txt
pip install --root-user-action=ignore pytest==7.4.3 pytest-asyncio==0.21.1
# Integration test with services
integration-test:
name: Integration Test with Services
runs-on: ubuntu-latest
needs: build
services:
victoriametrics:
image: victoriametrics/victoria-metrics:latest
ports:
- 8428:8428
options: >-
--health-cmd "wget --quiet --tries=1 --spider http://localhost:8428/health"
--health-interval 30s
--health-timeout 10s
--health-retries 3
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Wait for VictoriaMetrics
run: |
timeout 60s bash -c 'until curl -f http://localhost:8428/health; do sleep 2; done'
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip --root-user-action=ignore
pip install --root-user-action=ignore -r requirements.txt
- name: Test with VictoriaMetrics
env:
DB_TYPE: victoriametrics
VM_HOST: localhost
VM_PORT: 8428
run: |
python run.py --test
- name: Start API server
env:
DB_TYPE: victoriametrics
VM_HOST: localhost
VM_PORT: 8428
run: |
python run.py --web-api &
sleep 10
- name: Test API endpoints
run: |
curl -f http://localhost:8000/health
curl -f http://localhost:8000/stations
curl -f http://localhost:8000/metrics
# Deploy to staging (only on develop branch)
deploy-staging:
name: Deploy to Staging
runs-on: ubuntu-latest
needs: [test, build, integration-test]
if: github.ref == 'refs/heads/develop'
environment:
name: staging
url: https://staging.ping-river-monitor.b4l.co.th
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Deploy to staging
run: |
echo "Deploying to staging environment..."
# Add your staging deployment commands here
# Example: kubectl, docker-compose, or webhook call
- name: Health check staging
run: |
sleep 30
curl -f https://staging.ping-river-monitor.b4l.co.th/health
# Deploy to production (only on main branch, manual approval)
deploy-production:
name: Deploy to Production
runs-on: ubuntu-latest
needs: [test, build, integration-test]
if: github.ref == 'refs/heads/master'
environment:
name: production
url: https://ping-river-monitor.b4l.co.th
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Deploy to production
run: |
echo "Deploying to production environment..."
# Add your production deployment commands here
- name: Health check production
run: |
sleep 30
curl -f https://ping-river-monitor.b4l.co.th/health
- name: Notify deployment
run: |
echo "✅ Production deployment successful!"
echo "🌐 URL: https://ping-river-monitor.b4l.co.th"
echo "📊 Grafana: https://grafana.ping-river-monitor.b4l.co.th"
# Performance test (only on main branch)
performance-test:
name: Performance Test
runs-on: ubuntu-latest
needs: deploy-production
if: github.ref == 'refs/heads/master'
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Install Apache Bench
run: |
sudo apt-get update
sudo apt-get install -y apache2-utils
- name: Performance test API endpoints
run: |
# Test health endpoint
ab -n 100 -c 10 https://ping-river-monitor.b4l.co.th/health
# Test stations endpoint
ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/stations
# Test metrics endpoint
ab -n 50 -c 5 https://ping-river-monitor.b4l.co.th/metrics
# Cleanup old artifacts
cleanup:
name: Cleanup
runs-on: ubuntu-latest
if: always()
needs: [test, build, integration-test]
steps:
- name: Clean up old Docker images
run: |
echo "Cleaning up old Docker images..."
# Add cleanup commands for old images/artifacts
- name: pytest
env:
DB_TYPE: sqlite
run: pytest -q -p no:cacheprovider
.gitea/workflows/docs.yml
+83 -351
View File
@@ -1,367 +1,99 @@
name: Documentation
name: Docs
# Checks that the documentation the project actually ships stays consistent:
# - every relative link / image path in docs/*.md and README.md resolves
# inside the repo (external URLs are NOT fetched: localhost examples,
# rate-limited hosts and the Tailscale-era links made that gate permanently
# red, and a 200 on a curl --head proves nothing about a doc anyway)
# - the FastAPI app imports and its OpenAPI schema is exportable (that is
# the reference at https://water.buildfor.life/docs)
# The previous Sphinx/apidoc jobs produced artifacts nobody read and were
# removed. Reference docs live in docs/*.md; the public overview is at
# https://buildfor.life/docs/tooling/ping-river-monitor/.
on:
push:
branches: [ master, develop ]
branches: [master, develop]
paths:
- 'docs/**'
- 'README.md'
- 'CONTRIBUTING.md'
- 'src/**/*.py'
- "docs/**"
- "README.md"
- "CONTRIBUTING.md"
- "src/web_api.py"
- "src/schemas.py"
- ".gitea/workflows/docs.yml"
pull_request:
paths:
- 'docs/**'
- 'README.md'
- 'CONTRIBUTING.md'
- "docs/**"
- "README.md"
- "CONTRIBUTING.md"
workflow_dispatch:
env:
PYTHON_VERSION: '3.11'
PYTHON_VERSION: "3.11"
jobs:
# Validate documentation
validate-docs:
name: Validate Documentation
docs:
name: Validate documentation
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install documentation tools
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
pip install markdown-link-check || true
- name: Check markdown links
run: |
echo "🔗 Checking markdown links..."
find . -name "*.md" -not -path "./.git/*" -not -path "./node_modules/*" | while read file; do
echo "Checking $file"
# Basic link validation (you can enhance this)
grep -o 'http[s]*://[^)]*' "$file" | while read url; do
if curl -s --head "$url" | head -n 1 | grep -q "200 OK"; then
echo "✅ $url"
else
echo "❌ $url (in $file)"
fi
done
done
- name: Validate README structure
run: |
echo "📋 Validating README structure..."
required_sections=(
"# Northern Thailand Ping River Monitor"
"## Features"
"## Quick Start"
"## Installation"
"## Usage"
"## API Endpoints"
"## Docker"
"## Contributing"
"## License"
)
for section in "${required_sections[@]}"; do
if grep -q "$section" README.md; then
echo "✅ Found: $section"
else
echo "❌ Missing: $section"
fi
done
- name: Check documentation completeness
run: |
echo "📚 Checking documentation completeness..."
# Check if all Python modules have docstrings
python -c "
import ast
import os
def check_docstrings(filepath):
with open(filepath, 'r', encoding='utf-8') as f:
tree = ast.parse(f.read())
missing_docstrings = []
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef)):
if not ast.get_docstring(node):
missing_docstrings.append(f'{node.name} in {filepath}')
return missing_docstrings
all_missing = []
for root, dirs, files in os.walk('src'):
for file in files:
if file.endswith('.py') and not file.startswith('__'):
filepath = os.path.join(root, file)
missing = check_docstrings(filepath)
all_missing.extend(missing)
if all_missing:
print('⚠️ Missing docstrings:')
for item in all_missing[:10]: # Show first 10
print(f' - {item}')
if len(all_missing) > 10:
print(f' ... and {len(all_missing) - 10} more')
else:
print('✅ All functions and classes have docstrings')
"
- uses: actions/checkout@v4
# Generate API documentation
generate-api-docs:
name: Generate API Documentation
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Generate OpenAPI spec
run: |
echo "📝 Generating OpenAPI specification..."
python -c "
import json
import sys
sys.path.insert(0, 'src')
try:
from web_api import app
openapi_spec = app.openapi()
with open('openapi.json', 'w') as f:
json.dump(openapi_spec, f, indent=2)
print('✅ OpenAPI spec generated: openapi.json')
except Exception as e:
print(f'❌ Failed to generate OpenAPI spec: {e}')
"
- name: Generate API documentation
run: |
echo "📖 Generating API documentation..."
# Create API documentation from OpenAPI spec
if [ -f openapi.json ]; then
cat > api-docs.md << 'EOF'
# API Documentation
This document describes the REST API endpoints for the Northern Thailand Ping River Monitor.
## Base URL
- Production: `https://ping-river-monitor.b4l.co.th`
- Staging: `https://staging.ping-river-monitor.b4l.co.th`
- Development: `http://localhost:8000`
## Authentication
Currently, the API does not require authentication. This may change in future versions.
## Endpoints
EOF
# Extract endpoints from OpenAPI spec
python -c "
import json
with open('openapi.json', 'r') as f:
spec = json.load(f)
for path, methods in spec.get('paths', {}).items():
for method, details in methods.items():
print(f'### {method.upper()} {path}')
print()
print(details.get('summary', 'No description available'))
print()
if 'parameters' in details:
print('**Parameters:**')
for param in details['parameters']:
print(f'- `{param[\"name\"]}` ({param.get(\"in\", \"query\")}): {param.get(\"description\", \"No description\")}')
print()
print('---')
print()
" >> api-docs.md
echo "✅ API documentation generated: api-docs.md"
fi
- name: Upload documentation artifacts
uses: actions/upload-artifact@v3
with:
name: documentation-${{ github.run_number }}
path: |
openapi.json
api-docs.md
- name: Relative links and images resolve
run: |
python3 - <<'PY'
import re, sys, pathlib
root = pathlib.Path(".")
files = [root / "README.md", root / "CONTRIBUTING.md", *root.glob("docs/**/*.md")]
link = re.compile(r"!?\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
bad = []
for md in files:
if not md.exists():
continue
for m in link.finditer(md.read_text(encoding="utf-8")):
target = m.group(1)
if target.startswith(("http://", "https://", "mailto:", "#")):
continue
path = target.split("#", 1)[0]
if not path:
continue
resolved = (md.parent / path).resolve()
if not resolved.exists():
bad.append(f"{md}: {target}")
if bad:
print("Broken relative links:")
print("\n".join(" " + b for b in bad))
sys.exit(1)
print(f"checked {len(files)} files, all relative links resolve")
PY
# Build Sphinx documentation
build-sphinx-docs:
name: Build Sphinx Documentation
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITEA_TOKEN }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install sphinx sphinx-rtd-theme sphinx-autodoc-typehints
- name: Create Sphinx configuration
run: |
mkdir -p docs/sphinx
cat > docs/sphinx/conf.py << 'EOF'
import os
import sys
sys.path.insert(0, os.path.abspath('../../src'))
project = 'Northern Thailand Ping River Monitor'
copyright = '2025, Ping River Monitor Team'
author = 'Ping River Monitor Team'
version = '3.1.3'
release = '3.1.3'
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
'sphinx.ext.napoleon',
'sphinx_autodoc_typehints',
]
templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']
autodoc_default_options = {
'members': True,
'member-order': 'bysource',
'special-members': '__init__',
'undoc-members': True,
'exclude-members': '__weakref__'
}
EOF
cat > docs/sphinx/index.rst << 'EOF'
Northern Thailand Ping River Monitor Documentation
================================================
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
EOF
- name: Generate module documentation
run: |
cd docs/sphinx
sphinx-apidoc -o . ../../src
- name: Build documentation
run: |
cd docs/sphinx
sphinx-build -b html . _build/html
- name: Upload Sphinx documentation
uses: actions/upload-artifact@v3
with:
name: sphinx-docs-${{ github.run_number }}
path: docs/sphinx/_build/html/
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: pip
cache-dependency-path: requirements.txt
# Documentation summary
docs-summary:
name: Documentation Summary
runs-on: ubuntu-latest
needs: [validate-docs, generate-api-docs, build-sphinx-docs]
if: always()
steps:
- name: Generate documentation summary
run: |
echo "# 📚 Documentation Build Summary" > docs-summary.md
echo "" >> docs-summary.md
echo "**Build Date:** $(date -u)" >> docs-summary.md
echo "**Repository:** ${{ github.repository }}" >> docs-summary.md
echo "**Commit:** ${{ github.sha }}" >> docs-summary.md
echo "" >> docs-summary.md
echo "## 📊 Results" >> docs-summary.md
echo "" >> docs-summary.md
if [ "${{ needs.validate-docs.result }}" = "success" ]; then
echo "- ✅ **Documentation Validation**: Passed" >> docs-summary.md
else
echo "- ❌ **Documentation Validation**: Failed" >> docs-summary.md
fi
if [ "${{ needs.generate-api-docs.result }}" = "success" ]; then
echo "- ✅ **API Documentation**: Generated" >> docs-summary.md
else
echo "- ❌ **API Documentation**: Failed" >> docs-summary.md
fi
if [ "${{ needs.build-sphinx-docs.result }}" = "success" ]; then
echo "- ✅ **Sphinx Documentation**: Built" >> docs-summary.md
else
echo "- ❌ **Sphinx Documentation**: Failed" >> docs-summary.md
fi
echo "" >> docs-summary.md
echo "## 🔗 Available Documentation" >> docs-summary.md
echo "" >> docs-summary.md
echo "- [README.md](../README.md)" >> docs-summary.md
echo "- [API Documentation](../docs/)" >> docs-summary.md
echo "- [Contributing Guide](../CONTRIBUTING.md)" >> docs-summary.md
cat docs-summary.md
- name: Upload documentation summary
uses: actions/upload-artifact@v3
with:
name: docs-summary-${{ github.run_number }}
path: docs-summary.md
- name: Install dependencies
run: |
python -m pip install --upgrade pip --root-user-action=ignore
pip install --root-user-action=ignore -r requirements.txt
- name: OpenAPI schema exports
env:
DB_TYPE: sqlite
run: |
python - <<'PY'
import json
from src.web_api import app
spec = app.openapi()
paths = sorted(spec["paths"])
required = {"/forecast", "/measurements/latest", "/measurements/history/{station_code}", "/stations", "/api/stats", "/health"}
missing = required - set(paths)
assert not missing, f"documented endpoints missing from the app: {missing}"
json.dump(spec, open("openapi.json", "w"), indent=1)
print(f"{len(paths)} paths; schema written to openapi.json")
PY
- uses: actions/upload-artifact@v3
with:
name: openapi-${{ github.run_number }}
path: openapi.json