BerriAI/litellm · error · SagemakerError

LiteLLM Error: Unable to parse sagemaker RAW RESPONSE {json.

Error message

LiteLLM Error: Unable to parse sagemaker RAW RESPONSE {json.dumps(completion_response)}

What it means

While transforming a SageMaker completion response, LiteLLM tries to read the generated text from the first choice under the keys 'generation' or 'generated_text' (HF TGI/JumpStart conventions). If that traversal raises (missing keys, unexpected JSON shape, non-dict choices), it rethrows as SagemakerError 500 with the raw response JSON dumped into the message.

Source

Thrown at litellm/llms/sagemaker/completion/transformation.py:232

        ## RESPONSE OBJECT
        try:
            if isinstance(completion_response, list):
                completion_response_choices = completion_response[0]
            else:
                completion_response_choices = completion_response
            completion_output = ""
            if "generation" in completion_response_choices:
                completion_output += completion_response_choices["generation"]
            elif "generated_text" in completion_response_choices:
                completion_output += completion_response_choices["generated_text"]

            # check if the prompt template is part of output, if so - filter it out
            if completion_output.startswith(prompt) and "<s>" in prompt:
                completion_output = completion_output.replace(prompt, "", 1)

            model_response.choices[0].message.content = completion_output
        except Exception:
            raise SagemakerError(
                message=f"LiteLLM Error: Unable to parse sagemaker RAW RESPONSE {json.dumps(completion_response)}",
                status_code=500,
            )

        ## CALCULATING USAGE - baseten charges on time, not tokens - have some mapping of cost here.
        prompt_tokens: Final = token_counter(
            text=prompt, count_response_tokens=True
        )  # doesn't apply any default token count from openai's chat template
        completion_tokens: Final = token_counter(
            text=model_response["choices"][0]["message"].get("content", ""),
            count_response_tokens=True,
        )

        model_response.created = int(time.time())
        model_response.model = model
        usage: Final = Usage(
            prompt_tokens=prompt_tokens,
            completion_tokens=completion_tokens,

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Set litellm.set_verbose = True (or inspect the dumped JSON in the message) to see the actual response schema.
2. If the body uses a different key (e.g. 'outputs'), adjust the container's inference code to return [{'generated_text': '<text>'}] or use a provider config that matches the container.
3. Confirm you are using the correct model string; some containers expose both /invoke and /generate routes with different schemas.
4. Test the endpoint directly with boto3 invoke_endpoint to see its raw output before routing through LiteLLM.

Example fix

# custom SageMaker inference code - before
def transform(body):
    return {'outputs': [text]}
# after (HF/JumpStart convention LiteLLM expects)
def transform(body):
    return [{'generated_text': text}]

from litellm import SagemakerError

try:
    resp = litellm.completion(model='sagemaker/ep', messages=msgs)
except SagemakerError as e:
    if 'Unable to parse sagemaker RAW RESPONSE' in str(e):
        # raw body is embedded after the prefix - log it and alert
        log.exception('schema drift on sagemaker endpoint: %s', e.message)
    raise

When it happens

Trigger: The deployed SageMaker model returns a JSON body that does not contain a top-level list whose first element has 'generation' or 'generated_text' - e.g. LMI/TensorRT-LLM containers returning {'outputs': ...}, custom inference scripts, or an error page serialized as JSON.

Common situations: Pointing the sagemaker/ model string at a non-HuggingFace container; JumpStart model versions whose serving schema changed; a custom inference.py returning {'text': ...} instead of [{'generated_text': ...}]; endpoints that return 200 with an embedded error object.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Failed to parse response: {e}
• Unexpected response format. Expected list or dict with 'embe
• HF response not in expected format - {embeddings}
• No audio part found in the response
• Mavvrik FOCUS destination: response missing 'url' field: {re

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/34c34e2d729209a0. Report an issue: GitHub.