{"record":{"id":"34c34e2d729209a0","repo":"BerriAI/litellm","slug":"litellm-error-unable-to-parse-sagemaker-raw-respo","errorCode":null,"errorMessage":"LiteLLM Error: Unable to parse sagemaker RAW RESPONSE {json.dumps(completion_response)}","messagePattern":"LiteLLM Error: Unable to parse sagemaker RAW RESPONSE (.+?)","errorType":"exception","errorClass":"SagemakerError","httpStatus":500,"severity":"error","filePath":"litellm/llms/sagemaker/completion/transformation.py","lineNumber":232,"sourceCode":"        ## RESPONSE OBJECT\n        try:\n            if isinstance(completion_response, list):\n                completion_response_choices = completion_response[0]\n            else:\n                completion_response_choices = completion_response\n            completion_output = \"\"\n            if \"generation\" in completion_response_choices:\n                completion_output += completion_response_choices[\"generation\"]\n            elif \"generated_text\" in completion_response_choices:\n                completion_output += completion_response_choices[\"generated_text\"]\n\n            # check if the prompt template is part of output, if so - filter it out\n            if completion_output.startswith(prompt) and \"<s>\" in prompt:\n                completion_output = completion_output.replace(prompt, \"\", 1)\n\n            model_response.choices[0].message.content = completion_output\n        except Exception:\n            raise SagemakerError(\n                message=f\"LiteLLM Error: Unable to parse sagemaker RAW RESPONSE {json.dumps(completion_response)}\",\n                status_code=500,\n            )\n\n        ## CALCULATING USAGE - baseten charges on time, not tokens - have some mapping of cost here.\n        prompt_tokens: Final = token_counter(\n            text=prompt, count_response_tokens=True\n        )  # doesn't apply any default token count from openai's chat template\n        completion_tokens: Final = token_counter(\n            text=model_response[\"choices\"][0][\"message\"].get(\"content\", \"\"),\n            count_response_tokens=True,\n        )\n\n        model_response.created = int(time.time())\n        model_response.model = model\n        usage: Final = Usage(\n            prompt_tokens=prompt_tokens,\n            completion_tokens=completion_tokens,","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/llms/sagemaker/completion/transformation.py#L214-L250","documentation":"While transforming a SageMaker completion response, LiteLLM tries to read the generated text from the first choice under the keys 'generation' or 'generated_text' (HF TGI/JumpStart conventions). If that traversal raises (missing keys, unexpected JSON shape, non-dict choices), it rethrows as SagemakerError 500 with the raw response JSON dumped into the message.","triggerScenarios":"The deployed SageMaker model returns a JSON body that does not contain a top-level list whose first element has 'generation' or 'generated_text' - e.g. LMI/TensorRT-LLM containers returning {'outputs': ...}, custom inference scripts, or an error page serialized as JSON.","commonSituations":"Pointing the sagemaker/ model string at a non-HuggingFace container; JumpStart model versions whose serving schema changed; a custom inference.py returning {'text': ...} instead of [{'generated_text': ...}]; endpoints that return 200 with an embedded error object.","solutions":["Set litellm.set_verbose = True (or inspect the dumped JSON in the message) to see the actual response schema.","If the body uses a different key (e.g. 'outputs'), adjust the container's inference code to return [{'generated_text': '<text>'}] or use a provider config that matches the container.","Confirm you are using the correct model string; some containers expose both /invoke and /generate routes with different schemas.","Test the endpoint directly with boto3 invoke_endpoint to see its raw output before routing through LiteLLM."],"exampleFix":"# custom SageMaker inference code - before\ndef transform(body):\n    return {'outputs': [text]}\n# after (HF/JumpStart convention LiteLLM expects)\ndef transform(body):\n    return [{'generated_text': text}]","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from litellm import SagemakerError\n\ntry:\n    resp = litellm.completion(model='sagemaker/ep', messages=msgs)\nexcept SagemakerError as e:\n    if 'Unable to parse sagemaker RAW RESPONSE' in str(e):\n        # raw body is embedded after the prefix - log it and alert\n        log.exception('schema drift on sagemaker endpoint: %s', e.message)\n    raise","preventionTips":["Pin JumpStart/container versions so serving schemas do not drift under you.","Add a canary request after each endpoint update and assert the response parses.","Test the endpoint's raw output with boto3 in CI before wiring it into LiteLLM."],"tags":["sagemaker","response-parsing","huggingface","completion"],"backgroundTag":"unexpected-response-schema","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}