{"record":{"id":"1b4f57dfbb7256e8","repo":"BerriAI/litellm","slug":"litellm-midstreamfallbackerror-message","errorCode":null,"errorMessage":"litellm.MidStreamFallbackError: {message}","messagePattern":"litellm\\.MidStreamFallbackError: (.+?)","errorType":"exception","errorClass":"MidStreamFallbackError","httpStatus":503,"severity":"critical","filePath":"litellm/litellm_core_utils/streaming_handler.py","lineNumber":2340,"sourceCode":"                    status_code: Final = getattr(response, \"status_code\", None)\n                    if status_code is not None:\n                        return int(status_code)\n                except Exception:\n                    pass\n            return None\n\n        mapped_status_code: Final = _normalize_status_code(mapped_exception)\n        original_status_code: Final = _normalize_status_code(e)\n\n        # Raise non-retriable client errors directly (skip fallback).\n        # Exception: 429 (rate-limit) IS retriable/transient — allow it\n        # through so the Router can switch to a different model group.\n        if mapped_status_code is not None and 400 <= mapped_status_code < 500 and mapped_status_code != 429:\n            raise mapped_exception\n        if original_status_code is not None and 400 <= original_status_code < 500 and original_status_code != 429:\n            raise mapped_exception\n\n        raise MidStreamFallbackError(\n            message=str(mapped_exception),\n            model=self.model,\n            llm_provider=self.custom_llm_provider or \"anthropic\",\n            original_exception=mapped_exception,\n            generated_content=self.response_uptil_now,\n            is_pre_first_chunk=not self.sent_first_chunk,\n        )\n\n    @staticmethod\n    def _strip_sse_data_from_chunk(chunk: str | None) -> str | None:\n        \"\"\"\n        Strips the 'data: ' prefix from Server-Sent Events (SSE) chunks.\n\n        Some providers like sagemaker send it as `data:`, need to handle both\n\n        SSE messages are prefixed with 'data: ' which is part of the protocol,\n        not the actual content from the LLM. This method removes that prefix\n        and returns the actual content.","sourceCodeStart":2322,"sourceCodeEnd":2358,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/litellm_core_utils/streaming_handler.py#L2322-L2358","documentation":"LiteLLM wraps any non-client-error exception raised mid-stream (on the anthropic_messages / mid-stream fallback path) in MidStreamFallbackError so the Router can attempt a fallback model. Non-retriable 4xx client errors (except 429, which stays retriable) are re-raised directly instead; the error carries the partial generated_content, the original exception, and is_pre_first_chunk so callers know whether anything was already streamed to the client.","triggerScenarios":"Streaming with fallbacks enabled (Router fallback models or mid-stream fallback) when the primary deployment throws a server-side error (5xx, timeout, connection reset) after the stream started; a 429 mid-stream also lands here so the Router can switch model groups.","commonSituations":"Primary Azure/Anthropic deployment times out or returns 500 mid-generation; connection resets behind corporate proxies; rate limits hit mid-stream while using Router fallbacks; fallback model groups misconfigured so the fallback also fails.","solutions":["Catch MidStreamFallbackError and retry via the Router against a healthy fallback model group; verify the fallback deployments exist and are reachable.","Increase litellm timeout / stream_timeout - mid-stream timeouts are a frequent root cause.","If the underlying error is 4xx (auth, bad request), fix the root cause (api_key, model name, request body) - those bypass fallback by design.","Use .original_exception, .is_pre_first_chunk and .generated_content to decide whether partial content can be salvaged."],"exampleFix":"# before\nresp = await client.acompletion(model=\"anthropic/claude-3-5-sonnet\", messages=msgs, stream=True)\n\n# after\nfrom litellm.types.utils import MidStreamFallbackError\ntry:\n    resp = await router.acompletion(model=\"claude\", messages=msgs, stream=True)\n    async for e in resp: process(e)\nexcept MidStreamFallbackError as e:\n    log.warning(\"primary died pre_first_chunk=%s content_len=%d cause=%s\",\n                e.is_pre_first_chunk, len(e.generated_content or \"\"), e.original_exception)","handlingStrategy":"fallback","validationCode":null,"typeGuard":"from litellm.types.utils import MidStreamFallbackError\n\ndef is_midstream_fallback(e: BaseException) -> bool:\n    return isinstance(e, MidStreamFallbackError)","tryCatchPattern":"from litellm.types.utils import MidStreamFallbackError\ntry:\n    async for event in stream: process(event)\nexcept MidStreamFallbackError as e:\n    if e.is_pre_first_chunk:\n        result = await router.acompletion(model=fallback_model, messages=msgs, stream=True)\n    else:\n        salvage(e.generated_content)  # partial content already streamed","preventionTips":["Configure fallbacks on the Router (fallbacks=[{\"primary\": [\"backup\"]}]).","Set stream_timeout above the provider's typical time-to-first-token plus generation length.","Test the fallback path with a deliberately broken primary deployment."],"tags":["streaming","fallback","router","transient","network"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}