666ghj/MiroFish · error · LLMResponseError

LLM JSON output was truncated at the token limit

Error message

LLM JSON output was truncated at the token limit

What it means

LLMResponseError raised when choices[0].finish_reason == 'length': the JSON generation hit the max output token cap and was cut off mid-JSON, so the content cannot be trusted or parsed. The client deliberately does not repair truncated JSON (it never invents data), so it fails with the finish_reason attached.

Source

Thrown at backend/app/utils/llm_client.py:243

                    "retrying content generation%s",
                    error.finish_reason or "unknown",
                    " without an output token cap" if had_token_cap else "",
                )

        if last_error is not None:  # pragma: no cover - defensive loop guard
            raise last_error
        raise LLMResponseError("LLM did not produce a JSON response")

    @staticmethod
    def _parse_json_response(response: Any) -> Dict[str, Any]:
        choices = getattr(response, "choices", None) or []
        if not choices:
            raise LLMResponseError("LLM returned no choices")

        choice = choices[0]
        finish_reason = getattr(choice, "finish_reason", None)
        if finish_reason == "length":
            raise LLMResponseError(
                "LLM JSON output was truncated at the token limit",
                finish_reason=finish_reason,
            )
        if finish_reason not in {None, "stop"}:
            raise LLMResponseError(
                f"LLM JSON generation stopped unexpectedly ({finish_reason})",
                finish_reason=finish_reason,
            )

        content = _clean_chat_text(extract_chat_completion_text(response))
        if not content:
            raise LLMResponseError(
                "LLM returned empty JSON content",
                finish_reason=finish_reason,
            )

        try:
            value = json.loads(content)

View on GitHub (pinned to b5b53acc57)

Solutions

1. Increase max_tokens (or remove the output cap) for the affected call
2. Shrink the requested output: fewer items, shorter fields, or split the request into multiple smaller JSON calls
3. Switch to a model with a larger output window if the schema genuinely needs it
4. Do not attempt to 'fix up' the truncated JSON string — the code intentionally rejects repaired JSON

Example fix

# before
result = client.generate_json(messages, max_tokens=512)  # truncated -> LLMResponseError

# after
result = client.generate_json(messages, max_tokens=4096)

try:
    result = client.generate_json(messages, max_tokens=512)
except LLMResponseError as e:
    if e.finish_reason == "length":
        result = client.generate_json(messages, max_tokens=4096)  # retry with larger cap
    else:
        raise

When it happens

Trigger: Requesting structured JSON output (e.g. via _create_completion with max_tokens) where the model's JSON answer needs more tokens than the cap allows — large schemas, long arrays, or a max_tokens set too small for the requested structure.

Common situations: Tight max_tokens defaults to control cost, models that produce verbose JSON (long sub-question lists, big entity summaries), or prompts asking for many items in one response.

Related errors

• Ontology result must be an object
• LLM JSON generation stopped unexpectedly ({finish_reason})
• LLM returned empty JSON content
• LLM returned invalid JSON (line {strict_error.lineno}, colum
• LLM returned multiple JSON values

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/c7209e50b5d95b8d. Report an issue: GitHub.