BerriAI/litellm · error · ValueError

Failed to decode JSON from chunk: {chunk}

Error message

Failed to decode JSON from chunk: {chunk}

What it means

Thrown by the Anthropic streaming chunk parser when json.loads fails on a chunk it was handed. LiteLLM's Anthropic streaming handler parses each SSE data payload as JSON before converting it to a GenericStreamingChunk; if the payload is not valid JSON, the json.JSONDecodeError is re-raised as this ValueError. It almost always means the bytes coming off the HTTP stream are not a well-formed Anthropic SSE JSON event.

Source

Thrown at litellm/llms/anthropic/chat/handler.py:977

                        index=index,
                        delta=Delta(
                            content=text,
                            tool_calls=[tool_use] if tool_use is not None else None,
                            provider_specific_fields=(provider_specific_fields if provider_specific_fields else None),
                            thinking_blocks=(thinking_blocks if thinking_blocks else None),
                            reasoning_content=reasoning_content,
                        ),
                        finish_reason=finish_reason,
                    )
                ],
                usage=usage,
                id=self.response_id,
            )

            return returned_chunk

        except json.JSONDecodeError:
            raise ValueError(f"Failed to decode JSON from chunk: {chunk}")

    def _handle_json_mode_chunk(
        self, text: str, tool_use: ChatCompletionToolCallChunk | None
    ) -> tuple[str, ChatCompletionToolCallChunk | None]:
        """
        If JSON mode is enabled, convert the tool call to a message.

        Anthropic returns the JSON schema as part of the tool call
        OpenAI returns the JSON schema as part of the content, this handles placing it in the content

        Tool streaming follows Anthropic's fine-grained streaming pattern:
        - content_block_start: Contains complete tool info (id, name, empty arguments)
        - content_block_delta: Contains argument deltas (partial_json)
        - content_block_stop: Signals end of tool

        Reference: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/fine-grained-tool-streaming

        Args:

View on GitHub (pinned to 6c2dcb801b)

Solutions

  1. Reproduce with streaming disabled (stream=False) to see the raw body; if it is HTML/text, the problem is the endpoint or an intermediary, not your code.
  2. Inspect the {chunk} text in the message to identify what actually arrived (HTML error page, truncated JSON, empty string).
  3. If using a custom api_base or proxy, bypass it and call api.anthropic.com directly to confirm the proxy is mangling SSE frames.
  4. Ensure SSE is not buffered/compressed by intermediaries (disable gzip on the proxy, set 'Cache-Control: no-cache', flush per event).
  5. Upgrade LiteLLM — the accumulated-JSON flush logic in this iterator has had fixes for partial-JSON handling at stream end.

Example fix

# before
for chunk in completion(stream=True):
    process(chunk)  # ValueError kills the loop mid-stream

# after
for chunk in completion(stream=True):
    try:
        process(chunk)
    except ValueError as e:
        if "Failed to decode JSON from chunk" in str(e):
            log.warning("malformed SSE frame, skipping: %s", e)
            continue
        raise
Defensive patterns

Strategy: try-catch

Try / catch

try:
    for chunk in stream:
        consume(chunk)
except ValueError as e:
    if "Failed to decode JSON" in str(e):
        log.warning("malformed SSE frame: %s", e)
        # keep partial content, do not retry the whole stream blindly
    else:
        raise

Prevention

When it happens

Trigger: Iterating an anthropic/ streaming completion where a 'data:' payload contains malformed JSON: a proxy or gateway (e.g. nginx, Cloudflare, a custom LiteLLM proxy hop) that truncates or rewrites SSE frames; an upstream returning an HTML/text error page mid-stream; or accumulated partial JSON that was flushed without its remainder when the stream ended.

Common situations: Self-hosted LiteLLM proxy behind a buffering/rewriting proxy; Anthropic API base URL pointed at a third-party gateway that emits non-Anthropic SSE; truncation from aggressive read timeouts; occasionally a transient upstream 5xx body leaked into the stream.

Understand the failure class

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/951b6211e33540e9. Report an issue: GitHub.