2 Commits

Author SHA1 Message Date
4336e99e0c Implement elegant Docker networking solution for health checks
Some checks failed
Release - Northern Thailand Ping River Monitor / Create Release (push) Failing after 17s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.10) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.11) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.12) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Build (3.9) (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Build Release Images (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Security Scan (push) Has been skipped
Release - Northern Thailand Ping River Monitor / Test Release Deployment (push) Has been skipped
Security & Dependency Updates / Dependency Security Scan (push) Successful in 2m9s
Security & Dependency Updates / License Compliance (push) Successful in 15s
Security & Dependency Updates / Check for Dependency Updates (push) Successful in 20s
Security & Dependency Updates / Code Quality Metrics (push) Successful in 16s
Release - Northern Thailand Ping River Monitor / Notify Release (push) Successful in 1s
Security & Dependency Updates / Security Summary (push) Successful in 7s
Brilliant Solution Implemented:
- Create dedicated Docker network (ci_net) for container communication
- Use container name resolution (ping-river-monitor-test:8000)
- Separate curl container for probing (curlimages/curl:8.10.1)
- Clean separation of concerns and reliable networking

 Key Improvements:
- set -euo pipefail for strict error handling
- Container name resolution instead of IP detection
- Dedicated curl container on same network
- Cleaner probe() function for reusability
- Better error messages and debugging

 Network Architecture:
1. ci_net: Custom Docker network
2. ping-river-monitor-test: App container on ci_net
3. curlimages/curl: Probe container on ci_net (ephemeral)
4. Direct container-to-container communication

 Fallback Strategy:
- Primary: Container name resolution on ci_net
- Fallback: Host gateway probing via published port
- Comprehensive coverage of networking scenarios

 This should definitively resolve all networking issues!
2025-08-13 17:03:03 +07:00
455259a852 Add multi-method connection strategy for container health checks
Some checks failed
Release - Northern Thailand Ping River Monitor / Create Release (push) Successful in 5s
Security & Dependency Updates / Dependency Security Scan (push) Successful in 33s
Security & Dependency Updates / License Compliance (push) Successful in 13s
Security & Dependency Updates / Check for Dependency Updates (push) Successful in 19s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.10) (push) Successful in 16s
Security & Dependency Updates / Security Summary (push) Successful in 7s
Security & Dependency Updates / Code Quality Metrics (push) Successful in 20s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.11) (push) Successful in 15s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.12) (push) Successful in 14s
Release - Northern Thailand Ping River Monitor / Test Release Build (3.9) (push) Successful in 17s
Release - Northern Thailand Ping River Monitor / Security Scan (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Build Release Images (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Test Release Deployment (push) Has been cancelled
Release - Northern Thailand Ping River Monitor / Notify Release (push) Has been cancelled
Connection Methods (in order of preference):
1. Container IP direct connection (172.17.0.x:8000)
2. Docker exec from inside container (127.0.0.1:8000)
3. Host networking fallback (127.0.0.1:8080)

 Addresses Exit Code 28 (Timeout):
- Container IP connection was timing out in CI environment
- Docker exec bypasses network isolation issues
- Multiple fallback methods ensure reliability

 Improved Error Handling:
- Shorter timeouts (5s max, 3s connect) for faster fallback
- Clear method identification in logs
- Graceful degradation through connection methods

 Why Docker Exec Should Work:
- Runs curl from inside the target container
- No network isolation between runner and app container
- Direct access to 127.0.0.1:8000 (internal)
- Most reliable method in containerized CI environments

 Should resolve timeout issues and provide reliable health checks
2025-08-13 16:51:34 +07:00

View File

@@ -185,110 +185,97 @@ jobs:
- name: Deploy to production (Local Test)
run: |
set -euo pipefail
echo "🚀 Testing ${{ needs.create-release.outputs.version }} deployment locally..."
# Create a dedicated network so we can resolve by container name
docker network create ci_net || true
# Pull the built image
docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.create-release.outputs.version }}
# Stop any existing containers
docker stop ping-river-monitor-test || true
docker rm ping-river-monitor-test || true
# Start the container for testing
# Stop & remove any existing container
docker rm -f ping-river-monitor-test 2>/dev/null || true
# Start the container on the user-defined network
docker run -d \
--name ping-river-monitor-test \
--network ci_net \
-p 8080:8000 \
-e LOG_LEVEL=INFO \
-e DB_TYPE=sqlite \
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.create-release.outputs.version }}
echo "✅ Container started for testing"
- name: Health check after deployment
run: |
set -euo pipefail
echo "⏳ Waiting for application to start..."
sleep 30
echo "🔍 Running health checks against local container..."
# Check if container is running
docker ps | grep ping-river-monitor-test || echo "⚠️ Container not found in docker ps"
# Check container logs for any startup issues
echo "📋 Recent container logs:"
docker logs --tail 10 ping-river-monitor-test || true
# Wait for the application to be ready with more robust checking
echo "🔍 Testing application readiness..."
# Pull a curl-only image for probing (keeps your app image slim)
docker pull curlimages/curl:8.10.1
# Helper: curl via a sibling container on the SAME Docker network
probe() {
local url="$1"
docker run --rm --network ci_net curlimages/curl:8.10.1 \
-sS --max-time 5 --connect-timeout 3 -w "HTTP_CODE:%{http_code}" "$url" || true
}
# Wait for /health (up to ~3m 45s)
for i in {1..15}; do
echo " Attempt $i/15: Testing health endpoint..."
echo "🔍 Attempt $i/15: checking http://ping-river-monitor-test:8000/health"
resp="$(probe http://ping-river-monitor-test:8000/health)"
code="$(echo "$resp" | sed -n 's/.*HTTP_CODE:\([0-9]\+\).*/\1/p')"
body="$(echo "$resp" | sed 's/HTTP_CODE:[0-9]*$//')"
# Test health endpoint with container networking
echo "Testing health endpoint..."
echo "HTTP: ${code:-<none>} | Body: ${body:-<empty>}"
# Get the container's IP address for direct communication
CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ping-river-monitor-test)
echo "Container IP: $CONTAINER_IP"
# Test using container IP directly (port 8000 inside container)
if [ -n "$CONTAINER_IP" ]; then
response=$(curl -s --max-time 10 --connect-timeout 5 -w "HTTP_CODE:%{http_code}" http://$CONTAINER_IP:8000/health)
else
# Fallback to localhost if IP detection fails
response=$(curl -s --max-time 10 --connect-timeout 5 -w "HTTP_CODE:%{http_code}" http://127.0.0.1:8080/health)
fi
http_code=$(echo "$response" | grep -o "HTTP_CODE:[0-9]*" | cut -d: -f2)
response_body=$(echo "$response" | sed 's/HTTP_CODE:[0-9]*$//')
echo "HTTP Code: $http_code"
echo "Response Body: $response_body"
if [ "$http_code" = "200" ] && [ -n "$response_body" ]; then
echo "✅ Health endpoint responding successfully!"
if [ "${code:-}" = "200" ] && [ -n "${body:-}" ]; then
echo "✅ Health endpoint responding successfully"
break
else
echo "❌ Health check failed (HTTP: $http_code), waiting 15 seconds..."
# Show what's happening with the container
echo "Container status:"
docker ps | grep ping-river-monitor-test || echo "Container not found"
echo "Recent container logs:"
docker logs --tail 5 ping-river-monitor-test || true
sleep 15
fi
done
# Test API endpoints with container networking
echo "🧪 Testing API endpoints..."
# Get container IP for direct communication
CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ping-river-monitor-test)
echo "Using container IP: $CONTAINER_IP"
endpoints=("health" "docs" "stations" "metrics")
for endpoint in "${endpoints[@]}"; do
echo "Testing /$endpoint..."
# Use container IP if available, otherwise fallback to localhost
if [ -n "$CONTAINER_IP" ]; then
response=$(curl -s --max-time 10 -w "HTTP_CODE:%{http_code}" http://$CONTAINER_IP:8000/$endpoint)
else
response=$(curl -s --max-time 10 -w "HTTP_CODE:%{http_code}" http://127.0.0.1:8080/$endpoint)
fi
http_code=$(echo "$response" | grep -o "HTTP_CODE:[0-9]*" | cut -d: -f2)
echo "❌ Not ready yet. Showing recent logs…"
docker logs --tail 20 ping-river-monitor-test || true
sleep 15
if [ "$http_code" = "200" ]; then
echo "✅ /$endpoint: OK (HTTP $http_code)"
else
echo "❌ /$endpoint: FAILED (HTTP $http_code)"
echo "Response: $(echo "$response" | sed 's/HTTP_CODE:[0-9]*$//')"
if [ "$i" -eq 15 ]; then
echo "❌ Health never reached 200. Failing."
exit 1
fi
done
echo "🧪 Testing API endpoints…"
endpoints=("health" "docs" "stations" "metrics")
for ep in "${endpoints[@]}"; do
url="http://ping-river-monitor-test:8000/$ep"
resp="$(probe "$url")"
code="$(echo "$resp" | sed -n 's/.*HTTP_CODE:\([0-9]\+\).*/\1/p')"
if [ "${code:-}" = "200" ]; then
echo "✅ /$ep: OK"
else
echo "❌ /$ep: FAILED (HTTP ${code:-<none>})"
echo "Response: $(echo "$resp" | sed 's/HTTP_CODE:[0-9]*$//')"
exit 1
fi
done
echo "✅ All health checks passed!"
- name: (Fallback) Probe via host-published port
if: always()
run: |
set -euo pipefail
# In case you also want to verify the host-published port from inside the job container:
HOST_GATEWAY="$(ip route | awk '/default/ {print $3}')"
echo "🔎 Host gateway is $HOST_GATEWAY — probing http://$HOST_GATEWAY:8080/health"
docker run --rm curlimages/curl:8.10.1 \
-sS --max-time 5 --connect-timeout 3 -w "HTTP_CODE:%{http_code}\n" \
"http://$HOST_GATEWAY:8080/health" || true
- name: Container logs and cleanup
if: always()
run: |