BerriAI/litellm · error · ServiceUnavailableError

SagemakerException - {original_exception.message}

Error message

SagemakerException - {original_exception.message}

What it means

litellm maps a SageMaker failure carrying status_code 500 to litellm.ServiceUnavailableError with 'SagemakerException - <original message>'. The SageMaker runtime (or the model server behind the endpoint) returned an internal server error: the container crashed on the payload, the model process OOMed, or invoke_endpoint hit an unexpected server-side fault.

Source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1003

        raise BadRequestError(
            message="SagemakerException - the value of 'n' must be > 0 and <= 2 for sagemaker endpoints",
            model=model,
            llm_provider="sagemaker",
            response=getattr(original_exception, "response", None),
        )
    elif (
        "`inputs` tokens + `max_new_tokens` must be <=" in error_str
        or "instance type with more CPU capacity or memory" in error_str
    ):
        raise ContextWindowExceededError(
            message=f"SagemakerException - {error_str}",
            model=model,
            llm_provider="sagemaker",
            response=getattr(original_exception, "response", None),
        )
    elif hasattr(original_exception, "status_code"):
        if original_exception.status_code == 500:
            raise ServiceUnavailableError(
                message=f"SagemakerException - {original_exception.message}",
                llm_provider=custom_llm_provider,
                model=model,
                response=httpx.Response(
                    status_code=500,
                    request=httpx.Request(method="POST", url="https://api.openai.com/v1/"),
                ),
            )
        elif original_exception.status_code == 401:
            raise AuthenticationError(
                message=f"SagemakerException - {original_exception.message}",
                llm_provider=custom_llm_provider,
                model=model,
                response=getattr(original_exception, "response", None),
            )
        elif original_exception.status_code == 400:
            raise BadRequestError(
                message=f"SagemakerException - {original_exception.message}",

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Retry with backoff — a single 500 is often transient (num_retries on the call or Router)
2. Check endpoint CloudWatch logs (/aws/sagemaker/Endpoints/<endpoint>) for the container stack trace
3. Fix or harden the inference handler if the trace shows an unhandled exception on your payload shape
4. Scale the endpoint (instance size/count) if OOM or saturation recurs

Example fix

# before
resp = litellm.completion(model="sagemaker/my-endpoint", messages=msgs)

# after
resp = litellm.completion(model="sagemaker/my-endpoint", messages=msgs, num_retries=3)

import litellm

def is_endpoint_500(e: Exception) -> bool:
    return isinstance(e, litellm.ServiceUnavailableError)

import litellm
try:
    resp = litellm.completion(model="sagemaker/ep", messages=msgs)
except litellm.ServiceUnavailableError:
    # container-side failure: retry a bounded number of times, then alert
    resp = litellm.completion(model="sagemaker/ep", messages=msgs, num_retries=3)

When it happens

Trigger: Calling a 'sagemaker/' model where the endpoint's container returns 500: payload deserialization crashing the custom inference code, OOM kills of the model process, broken model artifacts after a bad deploy, or transient runtime faults. The mapped error attaches a synthetic httpx.Response(500) as a placeholder, not the real SageMaker response.

Common situations: Custom inference scripts raising unhandled exceptions; under-provisioned instances under concurrent load; endpoints mid-update while traffic flows; malformed input that bypasses validation and crashes handler code.

Related errors

• litellm.BadRequestError: SagemakerException - {error_str}
• SagemakerException - the value of 'n' must be > 0 and <= 2 f
• SagemakerException - {error_str}
• Internal Server Error
• Invalid content moderation response: {redacted_text}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/7f36ae512f99b100. Report an issue: GitHub.