{"record":{"id":"e42c573e203bb108","repo":"BerriAI/litellm","slug":"judge-response-is-not-a-json-object","errorCode":null,"errorMessage":"judge response is not a JSON object","messagePattern":"judge response is not a JSON object","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/litellm_core_utils/llm_judge.py","lineNumber":44,"sourceCode":"\n\ndef parse_json_verdict(raw: str) -> dict[str, object]:  # mutable-ok: plain parsed-JSON payload\n    \"\"\"Parse a judge's JSON verdict, tolerating markdown fences and surrounding prose.\"\"\"\n    text = raw.strip()  # rebind-ok: progressively narrowed to the JSON payload\n    fenced: Final = JSON_FENCE_RE.search(text)\n    if fenced is not None:\n        text = fenced.group(1).strip()  # rebind-ok: progressively narrowed to the JSON payload\n    parsed: object\n    try:\n        parsed = json.loads(text)\n    except json.JSONDecodeError:\n        start: Final = text.find(\"{\")\n        end: Final = text.rfind(\"}\")\n        if start == -1 or end <= start:\n            raise\n        parsed = json.loads(text[start : end + 1])\n    if not isinstance(parsed, dict):\n        raise ValueError(\"judge response is not a JSON object\")\n    return {str(k): v for k, v in parsed.items()}  # mutable-ok: plain parsed-JSON payload\n\n\ndef extract_text_from_content(content: object) -> str:\n    \"\"\"Return plain text from a message content field (str or multimodal list).\"\"\"\n    if isinstance(content, str):\n        return content\n    if isinstance(content, list):\n        return \" \".join(\n            str(part.get(\"text\", \"\")) for part in content if isinstance(part, dict) and part.get(\"type\") == \"text\"\n        )\n    return \"\"\n\n\ndef router_resolves_model(router: Router | None, model: str) -> bool:\n    \"\"\"Whether the model name resolves through the proxy's router (configured deployment\n    or model-group alias), the same check the judge dispatch itself makes, so start-time\n    validation cannot accept a name the call path then fails on.\"\"\"","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/litellm_core_utils/llm_judge.py#L26-L62","documentation":"parse_json_verdict() in litellm_core_utils/llm_judge.py parses an LLM judge's textual verdict: it unwraps ```json fences, or falls back to slicing from the first '{' to the last '}'. If the parsed payload is not a JSON object - an array, bare string/number/boolean/null - it raises ValueError('judge response is not a JSON object') because evaluation code (llm_as_a_judge guardrail, shadow evals) needs a dict such as {'score': ...}.","triggerScenarios":"A judge model answers an evaluation prompt with a JSON array of results, a bare scalar, or fenced JSON containing a list, e.g. [{\"score\": 0.5}] instead of {\"score\": 0.5}.","commonSituations":"Switching judge models that follow output-format instructions differently; prompts whose few-shot examples show arrays; judge returning null when it refuses; long verdicts where the model wraps prose around a list.","solutions":["Tighten the judge prompt: require exactly {\"score\": <number>, \"reason\": <string>} and explicitly forbid arrays or prose","Catch ValueError and retry the judge call with corrective feedback (append the parse error to the prompt)","If an array is the legitimate shape for your metric, convert it in a custom evaluator before handing it to litellm's parser","Validate judge output in a dry-run batch before enabling the evaluation in production"],"exampleFix":"# before\njudge_prompt = \"Rate the answer 0-1.\"\nverdict = parse_json_verdict(judge_llm(judge_prompt))  # model returns [0.5] -> ValueError\n\n# after\njudge_prompt = (\n    \"Rate the answer. Respond with ONLY a JSON object, no arrays, no prose: \"\n    '{\"score\": <float 0..1>, \"reason\": \"<one sentence>\"}'\n)\nverdict = parse_json_verdict(judge_llm(judge_prompt))","handlingStrategy":"retry","validationCode":"import json\n\ndef looks_like_json_object(text: str) -> bool:\n    try:\n        return isinstance(json.loads(text.strip().removeprefix(\"```json\").removesuffix(\"```\")), dict)\n    except json.JSONDecodeError:\n        return False","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    raw = judge_llm(prompt)\n    try:\n        verdict = parse_json_verdict(raw)\n        break\n    except ValueError:\n        prompt = corrective_prompt(raw)  # 'Return ONLY a JSON object like {\"score\": ...}'\nelse:\n    raise RuntimeError(\"judge failed to produce a JSON object after retries\")","preventionTips":["Pin the output format in the judge prompt with an exact example object","Add a retry-with-feedback loop around judge parsing in evaluation pipelines","Dry-run new judge models over a sample batch before enabling them in production"],"tags":["llm-judge","evaluation","json-parsing","litellm"],"backgroundTag":"invalid-json-response","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}