BerriAI/litellm · error · OpenAIError

error_message

Error message

error_message

What it means

Some OpenAI-compatible backends (vLLM, sglang) return HTTP 200 SSE streams whose individual data chunks carry an error payload like {"error": {"message": ..., "code": 400}}. LiteLLM's chunk_parser extracts that embedded error (message text; code if a 400-599 int, else 500) and raises OpenAIError mid-stream with that status and message.

Source

Thrown at litellm/llms/openai/chat/gpt_transformation.py:794

        """OpenAI-compatible backends (vLLM, sglang) can return an HTTP 200
        stream whose body carries an error payload, e.g.
        ``data: {"error": {"message": "...", "code": 400}}``."""
        error: Final = chunk.get("error")
        if not error:
            return None
        if not isinstance(error, dict):
            return str(error), 500
        message: Final = error.get("message")
        code: Final = error.get("code")
        status_code: Final = code if isinstance(code, int) and 400 <= code < 600 else 500
        return (message if isinstance(message, str) else json.dumps(error)), status_code

    def chunk_parser(self, chunk: dict) -> ModelResponseStream:
        try:
            error_details: Final = self._extract_error_from_chunk(chunk)
            if error_details is not None:
                error_message, error_status_code = error_details
                raise OpenAIError(
                    status_code=error_status_code,
                    message=error_message,
                )
            choices = chunk.get("choices", [])
            choices = self._map_reasoning_to_reasoning_content(choices)

            kwargs: Final[dict[str, Any]] = {
                "id": chunk.get("id"),
                "object": "chat.completion.chunk",
                "created": chunk.get("created"),
                "model": chunk.get("model"),
                "choices": choices,
            }
            if "usage" in chunk and chunk["usage"] is not None:
                kwargs["usage"] = chunk["usage"]
            return ModelResponseStream(**kwargs)
        except Exception as e:
            raise e

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Read the message — it is the backend's own error text and states the real cause.
2. For context-length errors, reduce input tokens/max_tokens; for preemption/OOM, reduce concurrency or increase GPU memory quota.
3. Retry with backoff for transient backend preemptions; pin the model (e.g. via sticky routing) if a router swaps models mid-stream.
4. Handle mid-stream failures in your consumer: errors can arrive after several successful chunks.

def stream_or_fallback(model, messages, **kw):
    try:
        return list(litellm.completion(model=model, messages=messages, stream=True, **kw))
    except litellm.exceptions.OpenAIError as e:
        if getattr(e, "status_code", None) == 400 and kw.get("stream"):
            # in-band stream error (vLLM/sglang style): retry non-streaming once
            return [litellm.completion(model=model, messages=messages, **kw)]
        raise

chunks = []
try:
    for chunk in litellm.completion(model="openai/m", messages=msgs, stream=True):
        chunks.append(chunk)
except litellm.exceptions.OpenAIError as e:
    # error arrived mid-stream with HTTP 200 already sent
    log.error("in-band stream error status=%s: %s", getattr(e, "status_code", "?"), e)
    raise

When it happens

Trigger: Streaming from vLLM/sglang (or any openai/ prefixed endpoint) when the server hits an error after the stream starts: model unloaded, request cancelled server-side, max token limit exceeded, or invalid request detected late. The chunk contains an 'error' object rather than 'choices'.

Common situations: vLLM backend crashing or preempting requests under load, served model swapped behind a router mid-stream, context length overruns detected after streaming began, or aggregated servers returning in-band errors.

Related errors

• Braintrust API error: {e.response.text}
• Failed to connect to Braintrust API: {str(e)}
• data_json.get("error")
• Ollama Error - {chunk}
• Unclassified keys in {PRICES_PATH.name}: {', '.join(unclassi

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/bc25e3963afc51ae. Report an issue: GitHub.