BerriAI/litellm · error · ValueError

judge response is not a JSON object

Error message

judge response is not a JSON object

What it means

parse_json_verdict() in litellm_core_utils/llm_judge.py parses an LLM judge's textual verdict: it unwraps ```json fences, or falls back to slicing from the first '{' to the last '}'. If the parsed payload is not a JSON object - an array, bare string/number/boolean/null - it raises ValueError('judge response is not a JSON object') because evaluation code (llm_as_a_judge guardrail, shadow evals) needs a dict such as {'score': ...}.

Solutions

1. Tighten the judge prompt: require exactly {"score": <number>, "reason": <string>} and explicitly forbid arrays or prose
2. Catch ValueError and retry the judge call with corrective feedback (append the parse error to the prompt)
3. If an array is the legitimate shape for your metric, convert it in a custom evaluator before handing it to litellm's parser
4. Validate judge output in a dry-run batch before enabling the evaluation in production

Example fix

# before
judge_prompt = "Rate the answer 0-1."
verdict = parse_json_verdict(judge_llm(judge_prompt))  # model returns [0.5] -> ValueError

# after
judge_prompt = (
    "Rate the answer. Respond with ONLY a JSON object, no arrays, no prose: "
    '{"score": <float 0..1>, "reason": "<one sentence>"}'
)
verdict = parse_json_verdict(judge_llm(judge_prompt))

import json

def looks_like_json_object(text: str) -> bool:
    try:
        return isinstance(json.loads(text.strip().removeprefix("```json").removesuffix("```")), dict)
    except json.JSONDecodeError:
        return False

for attempt in range(3):
    raw = judge_llm(prompt)
    try:
        verdict = parse_json_verdict(raw)
        break
    except ValueError:
        prompt = corrective_prompt(raw)  # 'Return ONLY a JSON object like {"score": ...}'
else:
    raise RuntimeError("judge failed to produce a JSON object after retries")

When it happens

Trigger: A judge model answers an evaluation prompt with a JSON array of results, a bare scalar, or fenced JSON containing a list, e.g. [{"score": 0.5}] instead of {"score": 0.5}.

Common situations: Switching judge models that follow output-format instructions differently; prompts whose few-shot examples show arrays; judge returning null when it refuses; long verdicts where the model wraps prose around a list.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

• Error parsing OpenRouter response
• Error parsing OpenRouter response
• Error parsing response
• raw_response.text
• raw_response.text

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/e42c573e203bb108. Report an issue: GitHub.

Appendix: source

Thrown at litellm/litellm_core_utils/llm_judge.py:44

def parse_json_verdict(raw: str) -> dict[str, object]:  # mutable-ok: plain parsed-JSON payload
    """Parse a judge's JSON verdict, tolerating markdown fences and surrounding prose."""
    text = raw.strip()  # rebind-ok: progressively narrowed to the JSON payload
    fenced: Final = JSON_FENCE_RE.search(text)
    if fenced is not None:
        text = fenced.group(1).strip()  # rebind-ok: progressively narrowed to the JSON payload
    parsed: object
    try:
        parsed = json.loads(text)
    except json.JSONDecodeError:
        start: Final = text.find("{")
        end: Final = text.rfind("}")
        if start == -1 or end <= start:
            raise
        parsed = json.loads(text[start : end + 1])
    if not isinstance(parsed, dict):
        raise ValueError("judge response is not a JSON object")
    return {str(k): v for k, v in parsed.items()}  # mutable-ok: plain parsed-JSON payload


def extract_text_from_content(content: object) -> str:
    """Return plain text from a message content field (str or multimodal list)."""
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        return " ".join(
            str(part.get("text", "")) for part in content if isinstance(part, dict) and part.get("type") == "text"
        )
    return ""


def router_resolves_model(router: Router | None, model: str) -> bool:
    """Whether the model name resolves through the proxy's router (configured deployment
    or model-group alias), the same check the judge dispatch itself makes, so start-time
    validation cannot accept a name the call path then fails on."""

View on GitHub (pinned to 77b7c6c40c)