BerriAI/litellm · error · ContextWindowExceededError

SagemakerException - {error_str}

Error message

SagemakerException - {error_str}

What it means

When the SageMaker error string contains '`inputs` tokens + `max_new_tokens` must be <=' or 'instance type with more CPU capacity or memory', litellm raises litellm.ContextWindowExceededError with 'SagemakerException - <error>'. The hosted HuggingFace endpoint has a hard token limit: your input length plus requested generation length overflows it, or the payload exceeds the instance's memory.

Source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:995

    if "Unable to locate credentials" in error_str:
        raise BadRequestError(
            message=f"litellm.BadRequestError: SagemakerException - {error_str}",
            model=model,
            llm_provider="sagemaker",
            response=getattr(original_exception, "response", None),
        )
    elif "Input validation error: `best_of` must be > 0 and <= 2" in error_str:
        raise BadRequestError(
            message="SagemakerException - the value of 'n' must be > 0 and <= 2 for sagemaker endpoints",
            model=model,
            llm_provider="sagemaker",
            response=getattr(original_exception, "response", None),
        )
    elif (
        "`inputs` tokens + `max_new_tokens` must be <=" in error_str
        or "instance type with more CPU capacity or memory" in error_str
    ):
        raise ContextWindowExceededError(
            message=f"SagemakerException - {error_str}",
            model=model,
            llm_provider="sagemaker",
            response=getattr(original_exception, "response", None),
        )
    elif hasattr(original_exception, "status_code"):
        if original_exception.status_code == 500:
            raise ServiceUnavailableError(
                message=f"SagemakerException - {original_exception.message}",
                llm_provider=custom_llm_provider,
                model=model,
                response=httpx.Response(
                    status_code=500,
                    request=httpx.Request(method="POST", url="https://api.openai.com/v1/"),
                ),
            )
        elif original_exception.status_code == 401:
            raise AuthenticationError(

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Reduce max_tokens and/or truncate the input so inputs + max_new_tokens fits the model's context
2. Count tokens with the same tokenizer the endpoint uses before sending
3. Add chunking/summarization upstream so prompts stay under budget
4. If the memory variant fires, redeploy the endpoint on a larger instance type

Example fix

# before
resp = litellm.completion(model="sagemaker/my-endpoint", messages=msgs, max_tokens=2000)

# after
from litellm import token_counter
MAX_CTX = 4096
budget = MAX_CTX - 512  # reserve for generation
if token_counter(model="sagemaker/my-endpoint", messages=msgs) > budget:
    msgs = truncate_messages(msgs, budget)
resp = litellm.completion(model="sagemaker/my-endpoint", messages=msgs, max_tokens=512)

from litellm import token_counter

CTX_LIMIT = 4096  # of the deployed sagemaker model
GEN_RESERVE = 512

def fits_context(messages, model: str) -> bool:
    return token_counter(model=model, messages=messages) + GEN_RESERVE <= CTX_LIMIT

import litellm

def is_context_exceeded(e: Exception) -> bool:
    return isinstance(e, litellm.ContextWindowExceededError)

import litellm
try:
    resp = litellm.completion(model="sagemaker/ep", messages=msgs, max_tokens=512)
except litellm.ContextWindowExceededError:
    # shrink deterministically: halve the prompt and retry once
    msgs = trim_to_half(msgs)
    resp = litellm.completion(model="sagemaker/ep", messages=msgs, max_tokens=256)

When it happens

Trigger: Long prompts on a sagemaker/ completion call with a large max_tokens such that inputs+max_new_tokens cross the deployed model's context limit; tokenizing differently from the endpoint so your count underestimates; payloads too large for the chosen instance type (the memory variant of the message).

Common situations: RAG pipelines stuffing many retrieved chunks into the prompt; keeping OpenAI-sized 128k habits against a 4k-context model on SageMaker; deploying small instances (ml.m5.xlarge) for large models.

Related errors

• BedrockException: Context Window Error - {error_str}
• litellm.BadRequestError: SagemakerException - {error_str}
• SagemakerException - the value of 'n' must be > 0 and <= 2 f
• ContextWindowExceededError: {exception_provider} - {message}
• {custom_llm_provider.capitalize()}Exception: Context Window

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/0bda91deef2e028d. Report an issue: GitHub.