What causes "VLLMException - {original_exception.message}" in BerriAI/litellm?

Verify the vLLM server process/container is alive and the port matches api_base

BerriAI/litellm · critical · APIConnectionError

VLLMException - {original_exception.message}

Error message

VLLMException - {original_exception.message}

What it means

litellm maps a vLLM error with status_code == 0 to litellm.APIConnectionError. Status 0 means no HTTP response at all - the TCP connection to the vLLM server failed (refused, reset, or DNS failure).

Source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1849

            message=f"OllamaException: {original_exception}",
            llm_provider="ollama",
            model=model,
        )


def _map_vllm_exception(
    *,
    model: str,
    original_exception: _ProviderHTTPException,
    custom_llm_provider: str,
    error_str: str,
    exception_type: str,
    exception_provider: str,
    extra_information: str,
) -> None:
    if hasattr(original_exception, "status_code"):
        if original_exception.status_code == 0:
            raise APIConnectionError(
                message=f"VLLMException - {original_exception.message}",
                llm_provider="vllm",
                model=model,
                request=getattr(original_exception, "request", None),
            )


def _map_azure_exception(
    *,
    model: str,
    original_exception: _ProviderHTTPException,
    custom_llm_provider: str,
    error_str: str,
    exception_type: str,
    exception_provider: str,
    extra_information: str,
) -> None:
    message = get_error_message(error_obj=original_exception)

View on GitHub (pinned to 6c2dcb801b)

Solutions

Verify the vLLM server process/container is alive and the port matches api_base
Health-check the endpoint: curl http://<host>:<port>/health (vLLM) or /v1/models
Fix the base URL passed to litellm (api_base= including http:// and port)
If OOM: raise memory limits or reduce gpu_memory_utilization / tensor_parallel size
Add readiness probes so traffic only routes to ready vLLM replicas

Example fix

# before
litellm.completion(model='vllm/meta-llama/Llama-3-8B', messages=msgs,
                    api_base='http://localhost:8000')  # server crashed
# after
import requests
base = 'http://localhost:8000'
requests.get(f'{base}/health', timeout=5).raise_for_status()  # gate the call
litellm.completion(model='vllm/meta-llama/Llama-3-8B', messages=msgs, api_base=base)

Defensive patterns

Strategy: validation

Validate before calling

import requests

def vllm_ready(base: str) -> bool:
    try:
        return requests.get(f'{base}/health', timeout=3).status_code == 200
    except requests.RequestException:
        return False

Type guard

import litellm

def is_vllm_conn_error(e: Exception) -> bool:
    return isinstance(e, litellm.APIConnectionError) and 'VLLMException' in str(e)

Try / catch

if not vllm_ready(base):
    raise RuntimeError('vLLM not ready')
try:
    litellm.completion(...)
except litellm.APIConnectionError:
    restart_or_failover()

Prevention

Run vLLM under a supervisor with restart policy
Set memory requests/limits so the model never OOMs at load
Health-check /health before sending user traffic

When it happens

Trigger: Calling model='vllm/...' (or openai/ against a vLLM base) when the vLLM server process is down, the api_base URL/port is wrong, the pod crashed (OOM during model load), or a network policy blocks the connection. The client raises a connection-level exception with status_code 0 and this mapper converts it.

Common situations: vLLM workers OOM-killed while loading large models, k8s service endpoints not ready, wrong OPENAI_API_BASE in env, or healthcheck-less deployments routing to dead replicas.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/cbb5ca8ccdbc02bb. Report an issue: GitHub.