BerriAI/litellm · error

Failed to decode JSON from chunk

Error message

Failed to decode JSON from chunk: {chunk}

What it means

In Databricks streaming, each SSE data line is json.loads-ed in __next__ and handed to chunk_parser, which validates it by constructing litellm.ModelResponseStream(**chunk) inside a try block. If that construction fails with json.JSONDecodeError (malformed embedded JSON strings inside the chunk), it is re-raised as ValueError naming the chunk. Note: a plain undecodable SSE line is handled separately (logged, skipped as an empty chunk), so this error specifically means the dict decoded but its contents failed JSON validation.

Solutions

1. Log the chunk embedded in the message and compare it field-by-field with the OpenAI streaming chunk schema
2. Confirm the endpoint is OpenAI-compatible (chat completions streaming shape) and reached via the expected databricks provider path
3. Retry with stream=False to see whether the non-streaming response from the same endpoint is well-formed
4. Upgrade LiteLLM; if the chunk is valid per Databricks docs, report it with the logged payload

try:
    for chunk in litellm.completion(model='databricks/my-ep', messages=msgs, stream=True):
        handle(chunk)
except Exception as e:
    if 'Failed to decode JSON from chunk' in str(e):
        log.error('malformed stream chunk: %s', str(e)[:2000])
        resp = litellm.completion(model='databricks/my-ep', messages=msgs, stream=False)
        handle(resp)
    else:
        raise

When it happens

Trigger: A Databricks stream chunk whose fields contain malformed JSON text (e.g. tool-call arguments or usage delivered as invalid JSON strings), or a chunk shape that breaks ModelResponseStream's parsing — from a non-OpenAI-compatible serving endpoint, a gateway mutating chunks, or a corrupted stream body.

Common situations: Custom/external model endpoints emitting non-standard chunk payloads; middleboxes rewriting SSE frames; schema drift between Databricks serving and LiteLLM's expected OpenAI-compatible chunk.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• KeyError: , Got unexpected response from Databricks
• AgentCore: Failed to read/parse JSON response body
• Error receiving chunk from stream
• Unable to get json response
• A2A send_message_streaming failed: no response received…

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/729fe053ff326dd2. Report an issue: GitHub.

Appendix: source

Thrown at litellm/llms/databricks/streaming_utils.py:90

            usage_chunk: Final[Usage | None] = getattr(processed_chunk, "usage", None)
            if usage_chunk is not None:
                usage = ChatCompletionUsageBlock(
                    prompt_tokens=usage_chunk.prompt_tokens,
                    completion_tokens=usage_chunk.completion_tokens,
                    total_tokens=usage_chunk.total_tokens,
                )

            return GenericStreamingChunk(
                text=text,
                tool_use=tool_use,
                is_finished=is_finished,
                finish_reason=finish_reason,
                usage=usage,
                index=0,
            )
        except json.JSONDecodeError:
            raise ValueError(f"Failed to decode JSON from chunk: {chunk}")

    # Sync iterator
    def __iter__(self):
        self.response_iterator = self.streaming_response
        return self

    def __next__(self):
        if not hasattr(self, "response_iterator"):
            self.response_iterator = self.streaming_response
        try:
            chunk = self.response_iterator.__next__()
        except StopIteration:
            raise StopIteration
        except ValueError as e:
            raise RuntimeError(f"Error receiving chunk from stream: {e}")

        try:
            chunk = litellm.CustomStreamWrapper._strip_sse_data_from_chunk(chunk) or ""

View on GitHub (pinned to 77b7c6c40c)