BerriAI/litellm · error · ValueError

judge response is not a JSON object

Error message

judge response is not a JSON object

What it means

litellm's LLM-judge utilities (guardrails / shadow eval) parse the judge model's output with parse_json_verdict, which tolerates markdown fences and surrounding prose, then requires the top-level JSON value to be an object (dict). If the model emits a JSON array, bare string, number, or null, parsing succeeds but the isinstance(parsed, dict) check fails and ValueError('judge response is not a JSON object') is raised.

Source

Thrown at litellm/litellm_core_utils/llm_judge.py:44

def parse_json_verdict(raw: str) -> dict[str, object]:  # mutable-ok: plain parsed-JSON payload
    """Parse a judge's JSON verdict, tolerating markdown fences and surrounding prose."""
    text = raw.strip()  # rebind-ok: progressively narrowed to the JSON payload
    fenced: Final = JSON_FENCE_RE.search(text)
    if fenced is not None:
        text = fenced.group(1).strip()  # rebind-ok: progressively narrowed to the JSON payload
    parsed: object
    try:
        parsed = json.loads(text)
    except json.JSONDecodeError:
        start: Final = text.find("{")
        end: Final = text.rfind("}")
        if start == -1 or end <= start:
            raise
        parsed = json.loads(text[start : end + 1])
    if not isinstance(parsed, dict):
        raise ValueError("judge response is not a JSON object")
    return {str(k): v for k, v in parsed.items()}  # mutable-ok: plain parsed-JSON payload


def extract_text_from_content(content: object) -> str:
    """Return plain text from a message content field (str or multimodal list)."""
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        return " ".join(
            str(part.get("text", "")) for part in content if isinstance(part, dict) and part.get("type") == "text"
        )
    return ""


def router_resolves_model(router: Router | None, model: str) -> bool:
    """Whether the model name resolves through the proxy's router (configured deployment
    or model-group alias), the same check the judge dispatch itself makes, so start-time
    validation cannot accept a name the call path then fails on."""

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Rewrite the judge prompt to demand a single top-level JSON object, e.g. {\"verdict\": ..., \"score\": ...}, and show an object-shaped example.
2. Force structured output: pass response_format={'type':'json_object'} (or a json_schema) on the judge model call.
3. Use a judge model that reliably follows JSON instructions (gpt-4o class) instead of a loose instruct model.
4. If you call parse_json_verdict yourself, pre-check the shape and re-ask the judge on non-dict output.

Example fix

// before (judge prompt)
Return the scores as JSON.

# after
Return ONLY a JSON object like {"pass": true, "reason": "..."}. No arrays, no bare values.

import json

def verdict_is_json_object(text: str) -> bool:
    try:
        return isinstance(json.loads(text), dict)
    except json.JSONDecodeError:
        return False

from litellm.litellm_core_utils.llm_judge import parse_json_verdict
try:
    verdict = parse_json_verdict(raw)
except (ValueError, json.JSONDecodeError):
    verdict = parse_json_verdict(retry_judge_call())  # one re-ask with a stricter prompt

When it happens

Trigger: Configuring an llm_as_a_judge guardrail or shadow-eval whose judge model returns a top-level JSON array or plain text instead of a JSON object; judge prompts that ask for 'a list of scores' (json.loads yields list); models that reply with 'true'/'0.8' (valid JSON scalars, not dicts).

Common situations: Judge prompt instructs the wrong output shape (list instead of object); swapping judge models where the new model ignores JSON-mode instructions; omitting response_format={'type':'json_object'}; few-shot examples in the prompt showing arrays.

Related errors

• Setting tag based guardrail modes is only available in litel
• No guardrail translation mapping found for call_type: {call_
• Error parsing response: {raw_response.text}, error: {e}
• Invalid guardrailConfig={raw_guardrail_config}. Expected for
• guardrailConfig={raw_guardrail_config} is missing 'guardrail

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/e42c573e203bb108. Report an issue: GitHub.