{"record":{"id":"0bda91deef2e028d","repo":"BerriAI/litellm","slug":"sagemakerexception-error-str","errorCode":null,"errorMessage":"SagemakerException - {error_str}","messagePattern":"SagemakerException - (.+?)","errorType":"exception","errorClass":"ContextWindowExceededError","httpStatus":400,"severity":"error","filePath":"litellm/litellm_core_utils/exception_mapping_utils.py","lineNumber":995,"sourceCode":"    if \"Unable to locate credentials\" in error_str:\n        raise BadRequestError(\n            message=f\"litellm.BadRequestError: SagemakerException - {error_str}\",\n            model=model,\n            llm_provider=\"sagemaker\",\n            response=getattr(original_exception, \"response\", None),\n        )\n    elif \"Input validation error: `best_of` must be > 0 and <= 2\" in error_str:\n        raise BadRequestError(\n            message=\"SagemakerException - the value of 'n' must be > 0 and <= 2 for sagemaker endpoints\",\n            model=model,\n            llm_provider=\"sagemaker\",\n            response=getattr(original_exception, \"response\", None),\n        )\n    elif (\n        \"`inputs` tokens + `max_new_tokens` must be <=\" in error_str\n        or \"instance type with more CPU capacity or memory\" in error_str\n    ):\n        raise ContextWindowExceededError(\n            message=f\"SagemakerException - {error_str}\",\n            model=model,\n            llm_provider=\"sagemaker\",\n            response=getattr(original_exception, \"response\", None),\n        )\n    elif hasattr(original_exception, \"status_code\"):\n        if original_exception.status_code == 500:\n            raise ServiceUnavailableError(\n                message=f\"SagemakerException - {original_exception.message}\",\n                llm_provider=custom_llm_provider,\n                model=model,\n                response=httpx.Response(\n                    status_code=500,\n                    request=httpx.Request(method=\"POST\", url=\"https://api.openai.com/v1/\"),\n                ),\n            )\n        elif original_exception.status_code == 401:\n            raise AuthenticationError(","sourceCodeStart":977,"sourceCodeEnd":1013,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/litellm_core_utils/exception_mapping_utils.py#L977-L1013","documentation":"When the SageMaker error string contains '`inputs` tokens + `max_new_tokens` must be <=' or 'instance type with more CPU capacity or memory', litellm raises litellm.ContextWindowExceededError with 'SagemakerException - <error>'. The hosted HuggingFace endpoint has a hard token limit: your input length plus requested generation length overflows it, or the payload exceeds the instance's memory.","triggerScenarios":"Long prompts on a sagemaker/ completion call with a large max_tokens such that inputs+max_new_tokens cross the deployed model's context limit; tokenizing differently from the endpoint so your count underestimates; payloads too large for the chosen instance type (the memory variant of the message).","commonSituations":"RAG pipelines stuffing many retrieved chunks into the prompt; keeping OpenAI-sized 128k habits against a 4k-context model on SageMaker; deploying small instances (ml.m5.xlarge) for large models.","solutions":["Reduce max_tokens and/or truncate the input so inputs + max_new_tokens fits the model's context","Count tokens with the same tokenizer the endpoint uses before sending","Add chunking/summarization upstream so prompts stay under budget","If the memory variant fires, redeploy the endpoint on a larger instance type"],"exampleFix":"# before\nresp = litellm.completion(model=\"sagemaker/my-endpoint\", messages=msgs, max_tokens=2000)\n\n# after\nfrom litellm import token_counter\nMAX_CTX = 4096\nbudget = MAX_CTX - 512  # reserve for generation\nif token_counter(model=\"sagemaker/my-endpoint\", messages=msgs) > budget:\n    msgs = truncate_messages(msgs, budget)\nresp = litellm.completion(model=\"sagemaker/my-endpoint\", messages=msgs, max_tokens=512)","handlingStrategy":"validation","validationCode":"from litellm import token_counter\n\nCTX_LIMIT = 4096  # of the deployed sagemaker model\nGEN_RESERVE = 512\n\ndef fits_context(messages, model: str) -> bool:\n    return token_counter(model=model, messages=messages) + GEN_RESERVE <= CTX_LIMIT","typeGuard":"import litellm\n\ndef is_context_exceeded(e: Exception) -> bool:\n    return isinstance(e, litellm.ContextWindowExceededError)","tryCatchPattern":"import litellm\ntry:\n    resp = litellm.completion(model=\"sagemaker/ep\", messages=msgs, max_tokens=512)\nexcept litellm.ContextWindowExceededError:\n    # shrink deterministically: halve the prompt and retry once\n    msgs = trim_to_half(msgs)\n    resp = litellm.completion(model=\"sagemaker/ep\", messages=msgs, max_tokens=256)","preventionTips":["Count tokens with the endpoint's tokenizer and reserve headroom for max_new_tokens","Truncate/summarize RAG context upstream of the LLM call","Store each sagemaker model's context limit in config and validate before sending"],"tags":["sagemaker","aws","context-window","tokens","bad-request"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}