BerriAI/litellm · critical · MidStreamFallbackError

litellm.MidStreamFallbackError: {message}

Error message

litellm.MidStreamFallbackError: {message}

What it means

LiteLLM wraps any non-client-error exception raised mid-stream (on the anthropic_messages / mid-stream fallback path) in MidStreamFallbackError so the Router can attempt a fallback model. Non-retriable 4xx client errors (except 429, which stays retriable) are re-raised directly instead; the error carries the partial generated_content, the original exception, and is_pre_first_chunk so callers know whether anything was already streamed to the client.

Source

Thrown at litellm/litellm_core_utils/streaming_handler.py:2340

                    status_code: Final = getattr(response, "status_code", None)
                    if status_code is not None:
                        return int(status_code)
                except Exception:
                    pass
            return None

        mapped_status_code: Final = _normalize_status_code(mapped_exception)
        original_status_code: Final = _normalize_status_code(e)

        # Raise non-retriable client errors directly (skip fallback).
        # Exception: 429 (rate-limit) IS retriable/transient — allow it
        # through so the Router can switch to a different model group.
        if mapped_status_code is not None and 400 <= mapped_status_code < 500 and mapped_status_code != 429:
            raise mapped_exception
        if original_status_code is not None and 400 <= original_status_code < 500 and original_status_code != 429:
            raise mapped_exception

        raise MidStreamFallbackError(
            message=str(mapped_exception),
            model=self.model,
            llm_provider=self.custom_llm_provider or "anthropic",
            original_exception=mapped_exception,
            generated_content=self.response_uptil_now,
            is_pre_first_chunk=not self.sent_first_chunk,
        )

    @staticmethod
    def _strip_sse_data_from_chunk(chunk: str | None) -> str | None:
        """
        Strips the 'data: ' prefix from Server-Sent Events (SSE) chunks.

        Some providers like sagemaker send it as `data:`, need to handle both

        SSE messages are prefixed with 'data: ' which is part of the protocol,
        not the actual content from the LLM. This method removes that prefix
        and returns the actual content.

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Catch MidStreamFallbackError and retry via the Router against a healthy fallback model group; verify the fallback deployments exist and are reachable.
2. Increase litellm timeout / stream_timeout - mid-stream timeouts are a frequent root cause.
3. If the underlying error is 4xx (auth, bad request), fix the root cause (api_key, model name, request body) - those bypass fallback by design.
4. Use .original_exception, .is_pre_first_chunk and .generated_content to decide whether partial content can be salvaged.

Example fix

# before
resp = await client.acompletion(model="anthropic/claude-3-5-sonnet", messages=msgs, stream=True)

# after
from litellm.types.utils import MidStreamFallbackError
try:
    resp = await router.acompletion(model="claude", messages=msgs, stream=True)
    async for e in resp: process(e)
except MidStreamFallbackError as e:
    log.warning("primary died pre_first_chunk=%s content_len=%d cause=%s",
                e.is_pre_first_chunk, len(e.generated_content or ""), e.original_exception)

from litellm.types.utils import MidStreamFallbackError

def is_midstream_fallback(e: BaseException) -> bool:
    return isinstance(e, MidStreamFallbackError)

from litellm.types.utils import MidStreamFallbackError
try:
    async for event in stream: process(event)
except MidStreamFallbackError as e:
    if e.is_pre_first_chunk:
        result = await router.acompletion(model=fallback_model, messages=msgs, stream=True)
    else:
        salvage(e.generated_content)  # partial content already streamed

When it happens

Trigger: Streaming with fallbacks enabled (Router fallback models or mid-stream fallback) when the primary deployment throws a server-side error (5xx, timeout, connection reset) after the stream started; a 429 mid-stream also lands here so the Router can switch model groups.

Common situations: Primary Azure/Anthropic deployment times out or returns 500 mid-generation; connection resets behind corporate proxies; rate limits hit mid-stream while using Router fallbacks; fallback model groups misconfigured so the fallback also fails.

Related errors

• Redis circuit breaker is open — skipping {name}
• {self.custom_llm_provider} raised a streaming error - finish
• Error receiving chunk from stream: {e}
• Failed to decode JSON from chunk: {chunk}
• Braintrust API error: {e.response.text}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/1b4f57dfbb7256e8. Report an issue: GitHub.