{"record":{"id":"50608747336df6a0","repo":"BerriAI/litellm","slug":"failed-to-decode-json-from-chunk-chunk-506087","errorCode":null,"errorMessage":"Failed to decode JSON from chunk: {chunk}","messagePattern":"Failed to decode JSON from chunk: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/llms/triton/completion/transformation.py","lineNumber":332,"sourceCode":"            provider_specific_fields: Final = None\n            index: Final = int(chunk.get(\"index\", 0))\n\n            # set values\n            text = chunk.get(\"text_output\", \"\")\n            finish_reason = chunk.get(\"stop_reason\", \"\")\n            is_finished = chunk.get(\"is_finished\", False)\n\n            return GenericStreamingChunk(\n                text=text,\n                tool_use=tool_use,\n                is_finished=is_finished,\n                finish_reason=finish_reason,\n                usage=usage,\n                index=index,\n                provider_specific_fields=provider_specific_fields,\n            )\n        except json.JSONDecodeError:\n            raise ValueError(f\"Failed to decode JSON from chunk: {chunk}\")\n","sourceCodeStart":314,"sourceCodeEnd":333,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/llms/triton/completion/transformation.py#L314-L333","documentation":"TritonResponseIterator (used for triton streaming) expects every streamed chunk to be a JSON object with generated text; when chunk parsing hits json.JSONDecodeError it re-raises as ValueError naming the chunk. This means the byte stream contained something that isn't a standalone JSON doc — e.g. Triton error text, an incomplete frame, or NDJSON multi-object lines — so the stream cannot be decoded as expected.","triggerScenarios":"Streaming a triton model with stream=True where api_base doesn't get the '_stream' variant correctly, Triton flushes error text mid-stream (model error after 200), or chunks are actually newline-delimited JSON objects ('{...}{...}') that json.loads rejects; also proxies injecting keep-alive/comment frames.","commonSituations":"Behind corporate proxies that rewrite streamed bodies; Triton versions emitting concatenated JSON per flush; models that abort mid-generation (OOM) and emit an error string; mismatch between generate-stream and infer endpoints.","solutions":["Reproduce with stream=False to confirm the non-streaming path works and the model itself is healthy.","Verify the streaming URL: for generate-type models LiteLLM appends '_stream' to the endpoint — ensure your Triton deployment exposes it (e.g. .../generate_stream).","Capture the failing chunk from the message and check what Triton actually sent (error text vs concatenated objects).","If frames are concatenated JSON, use the non-streaming path or upgrade litellm where the iterator handles NDJSON."],"exampleFix":"# before\nfor part in litellm.completion(\n    model=\"triton/my-llm\",\n    messages=msgs,\n    api_base=\"http://triton:8000/v2/models/my-llm/generate\",\n    stream=True,\n):\n    ...\n# ValueError: Failed to decode JSON from chunk: ...\n\n# after — ensure the *_stream endpoint exists on Triton and point api_base at it\nfor part in litellm.completion(\n    model=\"triton/my-llm\",\n    messages=msgs,\n    api_base=\"http://triton:8000/v2/models/my-llm/generate_stream\",\n    stream=True,\n):\n    print(part.choices[0].delta.content or \"\", end=\"\")","handlingStrategy":"try-catch","validationCode":"import json\n\ndef first_stream_chunk_ok(resp_raw: bytes) -> bool:\n    \"\"\"Peek at the first bytes: streaming expects JSON-object chunks.\"\"\"\n    try:\n        json.loads(resp_raw.decode(\"utf-8\", errors=\"strict\"))\n        return True\n    except (json.JSONDecodeError, UnicodeDecodeError):\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    for part in litellm.completion(\n        model=\"triton/my-llm\", messages=msgs,\n        api_base=\"http://triton:8000/v2/models/my-llm/generate_stream\",\n        stream=True,\n    ):\n        handle(part)\nexcept ValueError as e:\n    if \"Failed to decode JSON from chunk\" in str(e):\n        # fall back to non-streaming: same model, stream=False\n        resp = litellm.completion(\n            model=\"triton/my-llm\", messages=msgs,\n            api_base=\"http://triton:8000/v2/models/my-llm/generate\",\n        )\n        handle(resp)\n    else:\n        raise","preventionTips":["Confirm the _stream endpoint exists on your Triton deployment before enabling stream=True.","Smoke-test streaming after any Triton or proxy upgrade — chunk framing is the first thing to break.","Keep a non-streaming fallback path for streaming decode failures."],"tags":["triton","streaming","json-decode","response-parsing","litellm"],"backgroundTag":"invalid-json-response","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}