BerriAI/litellm · error · OCIError

Chunk cannot be parsed as JSON: {e}

Error message

Chunk cannot be parsed as JSON: {e}

What it means

After stripping the 'data:' prefix, chunk_creator json.loads the remainder; if it is not valid JSON the handler raises OCIError(500) with the JSONDecodeError detail. This points to a truncated or corrupted SSE payload rather than a request problem — the connection delivered bytes that do not form a complete JSON document.

Source

Thrown at litellm/llms/oci/chat/transformation.py:754

        # tool calls. The Cohere handler uses this to decide whether the
        # terminal consolidation chunk's tool calls are duplicates (suppress)
        # or the only copy of the tool calls (pass through).
        self._cohere_tool_calls_emitted = False
        # Analogous flag for text content. Lets the Cohere handler distinguish
        # the common case (prior deltas already streamed the text, so the
        # terminal chunk's text is a duplicate to suppress) from the degenerate
        # single-event case (terminal chunk carries the only copy of the text).
        self._cohere_text_emitted = False

    def chunk_creator(self, chunk: Any) -> ModelResponseStream:
        if not isinstance(chunk, str):
            raise ValueError(f"Chunk is not a string: {chunk}")
        if not chunk.startswith("data:"):
            raise ValueError(f"Chunk does not start with 'data:': {chunk}")
        try:
            dict_chunk: Final = json.loads(chunk[5:])
        except json.JSONDecodeError as e:
            raise OCIError(
                status_code=500,
                message=f"Chunk cannot be parsed as JSON: {e}",
            )

        if dict_chunk.get("apiFormat") == "COHERE":
            result: Final = handle_cohere_stream_chunk(
                dict_chunk,
                prior_tool_calls_emitted=self._cohere_tool_calls_emitted,
                prior_text_emitted=self._cohere_text_emitted,
            )
            if not self._cohere_tool_calls_emitted:
                for choice in result.choices:
                    if getattr(choice.delta, "tool_calls", None) is not None:
                        self._cohere_tool_calls_emitted = True
                        break
            if not self._cohere_text_emitted:
                for choice in result.choices:
                    if getattr(choice.delta, "content", None):

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Bypass proxies/LBs or configure them for unbuffered SSE passthrough (disable gzip, set flush interval to 0).
2. Retry the request — truncation from transient network drops is not deterministic.
3. If reproducible with curl -N, capture the raw stream and inspect the malformed event to identify which hop corrupts it.
4. Reduce max_tokens or tool-call size if corruption correlates with very large single events.

Example fix

# before
stream = litellm.completion(model="oci/...", messages=m, stream=True)
for ev in stream:
    pass  # mid-stream OCIError(500) on truncated JSON

# after — treat parse failure as retryable stream error
from litellm.llms.oci.common_utils import OCIError
try:
    for ev in litellm.completion(model="oci/...", messages=m, stream=True):
        pass
except OCIError as e:
    if "parsed as JSON" in str(e):
        retry_with_backoff()

import json

def is_parseable_sse(chunk: str) -> bool:
    try:
        json.loads(chunk[5:])
        return True
    except json.JSONDecodeError:
        return False

from litellm.llms.oci.common_utils import OCIError
try:
    for ev in stream:
        handle(ev)
except OCIError as e:
    if "parsed as JSON" in str(e):
        restart_stream_from_last_checkpoint()  # retry, partial output already consumed
    raise

When it happens

Trigger: SSE line split mid-JSON by a buffering intermediary, a network drop producing a partial final event, or an upstream OCI gateway emitting a malformed event. Chunk strings like 'data:{"apiFormat": ' (cut off) trigger it.

Common situations: Load balancers/proxies with aggressive flush intervals truncating event frames; connections killed by idle timeouts leaving half-delivered events; very large tool-call deltas exceeding an intermediary's line-buffer limit. The status_code=500 here is synthetic — it represents a local parse failure, not an OCI HTTP status.

Related errors

• Error parsing chunk: {e}, Received chunk: {chunk}
• Chunk does not start with 'data:': {chunk}
• Failed to decode JSON from chunk: {chunk}
• error_message or raw_response.text
• Chunk cannot be parsed as CohereStreamChunk: {e}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/629b3ee9f352a2de. Report an issue: GitHub.