Brilliant Solution Implemented:
- Create dedicated Docker network (ci_net) for container communication
- Use container name resolution (ping-river-monitor-test:8000)
- Separate curl container for probing (curlimages/curl:8.10.1)
- Clean separation of concerns and reliable networking
Key Improvements:
- set -euo pipefail for strict error handling
- Container name resolution instead of IP detection
- Dedicated curl container on same network
- Cleaner probe() function for reusability
- Better error messages and debugging
Network Architecture:
1. ci_net: Custom Docker network
2. ping-river-monitor-test: App container on ci_net
3. curlimages/curl: Probe container on ci_net (ephemeral)
4. Direct container-to-container communication
Fallback Strategy:
- Primary: Container name resolution on ci_net
- Fallback: Host gateway probing via published port
- Comprehensive coverage of networking scenarios
This should definitively resolve all networking issues!
Connection Methods (in order of preference):
1. Container IP direct connection (172.17.0.x:8000)
2. Docker exec from inside container (127.0.0.1:8000)
3. Host networking fallback (127.0.0.1:8080)
Addresses Exit Code 28 (Timeout):
- Container IP connection was timing out in CI environment
- Docker exec bypasses network isolation issues
- Multiple fallback methods ensure reliability
Improved Error Handling:
- Shorter timeouts (5s max, 3s connect) for faster fallback
- Clear method identification in logs
- Graceful degradation through connection methods
Why Docker Exec Should Work:
- Runs curl from inside the target container
- No network isolation between runner and app container
- Direct access to 127.0.0.1:8000 (internal)
- Most reliable method in containerized CI environments
Should resolve timeout issues and provide reliable health checks
Root Cause Identified:
- Gitea runner runs inside docker.gitea.com/runner-images:ubuntu-latest
- App container runs as sibling container, not accessible via localhost:8080
- Port mapping works for host access, but not container-to-container
Networking Solution:
- Get container IP with: docker inspect ping-river-monitor-test
- Connect directly to container IP:8000 (internal port)
- Fallback to localhost:8080 if IP detection fails
- Bypasses localhost networking issues in containerized CI
Updated Health Checks:
- Use container IP for direct communication
- Test internal port 8000 instead of mapped port 8080
- More reliable in containerized CI environments
- Better debugging with container IP logging
Should resolve curl connection failures in Gitea CI environment
🔍 Enhanced Debugging:
- Show HTTP response codes and response bodies
- Remove -f flag that was causing curl to fail on valid responses
- Add detailed logging for each endpoint test
- Show container logs on failures
🌐 Improved Health Check Logic:
- Check HTTP code = 200 AND response body exists
- Use curl -w to capture HTTP status codes
- Parse response and status separately
- More tolerant of response format variations
🧪 Better API Endpoint Testing:
- Test each endpoint individually with status reporting
- Show specific HTTP codes for each endpoint
- Clear success/failure messages per endpoint
- Exit only on actual HTTP errors
🎯 Addresses CI-Specific Issues:
- Local testing shows endpoints work correctly
- CI environment may have different curl behavior
- More detailed output will help identify root cause
- Removes false failures from -f flag sensitivity
Should resolve curl failures despite HTTP 200 responses
🌐 Network Fix:
- Change localhost to 127.0.0.1 for all health check URLs
- Prevents IPv6 resolution issues in CI environment
- Ensures consistent IPv4 connectivity to container
🔍 Debugging Improvements:
- Check if container is running with docker ps
- Show recent container logs before health checks
- Better troubleshooting information for failures
📋 Updated Endpoints:
- http://127.0.0.1:8080/health
- http://127.0.0.1:8080/docs
- http://127.0.0.1:8080/stations
- http://127.0.0.1:8080/metrics✅ Should resolve curl connection failures to localhost
🌐 Network Fix:
- Change localhost to 127.0.0.1 for all health check URLs
- Prevents IPv6 resolution issues in CI environment
- Ensures consistent IPv4 connectivity to container
🔍 Debugging Improvements:
- Check if container is running with docker ps
- Show recent container logs before health checks
- Better troubleshooting information for failures
📋 Updated Endpoints:
- http://127.0.0.1:8080/health
- http://127.0.0.1:8080/docs
- http://127.0.0.1:8080/stations
- http://127.0.0.1:8080/metrics✅ Should resolve curl connection failures to localhost
Dockerfile Fixes:
- Copy Python packages to /home/appuser/.local instead of /root/.local
- Create appuser home directory before copying packages
- Update PATH to use /home/appuser/.local/bin
- Set proper ownership of .local directory for appuser
- Ensure appuser has access to installed Python packages
Problem Solved:
- Container was failing with 'ModuleNotFoundError: No module named requests'
- appuser couldn't access packages installed in /root/.local
- Python dependencies now properly accessible to non-root user
Docker container should now start successfully with all dependencies
Release Workflow Changes:
- Replace production deployment with local container testing
- Spin up Docker container on same machine (port 8080)
- Run comprehensive health checks against local container
- Test all API endpoints (health, docs, stations, metrics)
- Clean up test container after validation
Removed Redundant Validation:
- Remove validate-release job (redundant with local testing)
- Consolidate all testing into deploy-release job
- Update notification dependencies (validate-release deploy-release)
- Remove external URL dependencies
Benefits:
- No external production system required
- Safer testing approach (isolated container)
- Comprehensive API validation before any real deployment
- Container logs available for debugging
- Ready-to-deploy image verification
Workflow now tests locally and confirms image is ready for production
2025-08-13 11:27:38 +07:00
3 changed files with 259 additions and 234 deletions
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
