What causes "Service '{service_name}' health check failed" in ruvnet/RuView?

Grep the API server logs for 'Error checking service health for ' — the logged exception is the true cause

ruvnet/RuView · error · HTTPException

Service '{service_name}' health check failed

Error message

Service '{service_name}' health check failed

What it means

check_service_health raises 503 "Service '<name>' health check failed" from the generic except handler when calling service.get_status() itself raises an unexpected exception (as opposed to returning an unhealthy dict). The original exception is logged via logger.error before the 503 is raised, so the log holds the real cause.

Source

Thrown at archive/v1/src/api/dependencies.py:291

                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
                detail=f"Service '{service_name}' not available"
            )
        
        # Check service health
        status_info = await service.get_status()
        if status_info.get("status") != "healthy":
            raise HTTPException(
                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
                detail=f"Service '{service_name}' is unhealthy: {status_info.get('error', 'Unknown error')}"
            )
        
        return True
        
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Error checking service health for {service_name}: {e}")
        raise HTTPException(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            detail=f"Service '{service_name}' health check failed"
        )


# Rate limiting dependencies
async def check_rate_limit(
    request: Request,
    current_user: Optional[Dict[str, Any]] = Depends(get_current_user)
) -> bool:
    """Check rate limiting status."""
    settings = get_settings()
    
    # Skip if rate limiting is disabled
    if not settings.enable_rate_limiting:
        return True
    
    # Rate limiting is handled by middleware

View on GitHub (pinned to 4685618388)

Solutions

Grep the API server logs for 'Error checking service health for <name>' — the logged exception is the true cause
Fix or restart the failing service so get_status() completes again
Harden get_status() implementations to report degraded status instead of raising
Retry the health check after remediation; treat 503 as transient

Defensive patterns

Strategy: retry

Try / catch

try:
    r = client.get(f'/api/health/{service_name}')
    r.raise_for_status()
except httpx.HTTPStatusError as e:
    if e.response.status_code == 503 and 'health check failed' in e.response.json().get('detail', ''):
        # get_status() raised: transient broken state, retry with backoff;
        # the server log line 'Error checking service health for ...' has the cause
        await asyncio.sleep(backoff)
        r = client.get(f'/api/health/{service_name}')
    raise

Prevention

Correlate client 503s with the server log's 'Error checking service health' entries
Make get_status() implementations return degraded status instead of raising
Use bounded retries with exponential backoff; escalate if the failure persists

When it happens

Trigger: get_status() throwing because a dependency client is in a broken state (connection closed, None where an object is expected), a bug in the service's status code path, or an async operation failing mid-probe.

Common situations: Service crashed internally but app.state still holds the dead object; status code paths untested against missing optional dependencies; race between service shutdown and a health probe.

Related errors

AI-assisted analysis of ruvnet/RuView@4685618388 (2026-08-16). Data as JSON: /api/errors/39eb364db8682381. Report an issue: GitHub.