BerriAI/litellm · error · ValueError

Failed to decode JSON from chunk: {chunk}

Error message

Failed to decode JSON from chunk: {chunk}

What it means

TritonResponseIterator (used for triton streaming) expects every streamed chunk to be a JSON object with generated text; when chunk parsing hits json.JSONDecodeError it re-raises as ValueError naming the chunk. This means the byte stream contained something that isn't a standalone JSON doc — e.g. Triton error text, an incomplete frame, or NDJSON multi-object lines — so the stream cannot be decoded as expected.

Source

Thrown at litellm/llms/triton/completion/transformation.py:332

            provider_specific_fields: Final = None
            index: Final = int(chunk.get("index", 0))

            # set values
            text = chunk.get("text_output", "")
            finish_reason = chunk.get("stop_reason", "")
            is_finished = chunk.get("is_finished", False)

            return GenericStreamingChunk(
                text=text,
                tool_use=tool_use,
                is_finished=is_finished,
                finish_reason=finish_reason,
                usage=usage,
                index=index,
                provider_specific_fields=provider_specific_fields,
            )
        except json.JSONDecodeError:
            raise ValueError(f"Failed to decode JSON from chunk: {chunk}")

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Reproduce with stream=False to confirm the non-streaming path works and the model itself is healthy.
2. Verify the streaming URL: for generate-type models LiteLLM appends '_stream' to the endpoint — ensure your Triton deployment exposes it (e.g. .../generate_stream).
3. Capture the failing chunk from the message and check what Triton actually sent (error text vs concatenated objects).
4. If frames are concatenated JSON, use the non-streaming path or upgrade litellm where the iterator handles NDJSON.

Example fix

# before
for part in litellm.completion(
    model="triton/my-llm",
    messages=msgs,
    api_base="http://triton:8000/v2/models/my-llm/generate",
    stream=True,
):
    ...
# ValueError: Failed to decode JSON from chunk: ...

# after — ensure the *_stream endpoint exists on Triton and point api_base at it
for part in litellm.completion(
    model="triton/my-llm",
    messages=msgs,
    api_base="http://triton:8000/v2/models/my-llm/generate_stream",
    stream=True,
):
    print(part.choices[0].delta.content or "", end="")

import json

def first_stream_chunk_ok(resp_raw: bytes) -> bool:
    """Peek at the first bytes: streaming expects JSON-object chunks."""
    try:
        json.loads(resp_raw.decode("utf-8", errors="strict"))
        return True
    except (json.JSONDecodeError, UnicodeDecodeError):
        return False

try:
    for part in litellm.completion(
        model="triton/my-llm", messages=msgs,
        api_base="http://triton:8000/v2/models/my-llm/generate_stream",
        stream=True,
    ):
        handle(part)
except ValueError as e:
    if "Failed to decode JSON from chunk" in str(e):
        # fall back to non-streaming: same model, stream=False
        resp = litellm.completion(
            model="triton/my-llm", messages=msgs,
            api_base="http://triton:8000/v2/models/my-llm/generate",
        )
        handle(resp)
    else:
        raise

When it happens

Trigger: Streaming a triton model with stream=True where api_base doesn't get the '_stream' variant correctly, Triton flushes error text mid-stream (model error after 200), or chunks are actually newline-delimited JSON objects ('{...}{...}') that json.loads rejects; also proxies injecting keep-alive/comment frames.

Common situations: Behind corporate proxies that rewrite streamed bodies; Triton versions emitting concatenated JSON per flush; models that abort mid-generation (OOM) and emit an error string; mismatch between generate-stream and infer endpoints.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• raw_response.text
• raw_response.text
• Error in response object format
• LiteLLM: provider returned a response with no 'choices'. Raw
• AgentCore: Failed to read/parse JSON response body: {e}

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/50608747336df6a0. Report an issue: GitHub.