{"record":{"id":"e42c573e203bb108","repo":"BerriAI/litellm","slug":"judge-response-is-not-a-json-object","errorCode":null,"errorMessage":"judge response is not a JSON object","messagePattern":"judge response is not a JSON object","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/litellm_core_utils/llm_judge.py","lineNumber":44,"sourceCode":"\n\ndef parse_json_verdict(raw: str) -> dict[str, object]:  # mutable-ok: plain parsed-JSON payload\n    \"\"\"Parse a judge's JSON verdict, tolerating markdown fences and surrounding prose.\"\"\"\n    text = raw.strip()  # rebind-ok: progressively narrowed to the JSON payload\n    fenced: Final = JSON_FENCE_RE.search(text)\n    if fenced is not None:\n        text = fenced.group(1).strip()  # rebind-ok: progressively narrowed to the JSON payload\n    parsed: object\n    try:\n        parsed = json.loads(text)\n    except json.JSONDecodeError:\n        start: Final = text.find(\"{\")\n        end: Final = text.rfind(\"}\")\n        if start == -1 or end <= start:\n            raise\n        parsed = json.loads(text[start : end + 1])\n    if not isinstance(parsed, dict):\n        raise ValueError(\"judge response is not a JSON object\")\n    return {str(k): v for k, v in parsed.items()}  # mutable-ok: plain parsed-JSON payload\n\n\ndef extract_text_from_content(content: object) -> str:\n    \"\"\"Return plain text from a message content field (str or multimodal list).\"\"\"\n    if isinstance(content, str):\n        return content\n    if isinstance(content, list):\n        return \" \".join(\n            str(part.get(\"text\", \"\")) for part in content if isinstance(part, dict) and part.get(\"type\") == \"text\"\n        )\n    return \"\"\n\n\ndef router_resolves_model(router: Router | None, model: str) -> bool:\n    \"\"\"Whether the model name resolves through the proxy's router (configured deployment\n    or model-group alias), the same check the judge dispatch itself makes, so start-time\n    validation cannot accept a name the call path then fails on.\"\"\"","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/litellm_core_utils/llm_judge.py#L26-L62","documentation":"litellm's LLM-judge utilities (guardrails / shadow eval) parse the judge model's output with parse_json_verdict, which tolerates markdown fences and surrounding prose, then requires the top-level JSON value to be an object (dict). If the model emits a JSON array, bare string, number, or null, parsing succeeds but the isinstance(parsed, dict) check fails and ValueError('judge response is not a JSON object') is raised.","triggerScenarios":"Configuring an llm_as_a_judge guardrail or shadow-eval whose judge model returns a top-level JSON array or plain text instead of a JSON object; judge prompts that ask for 'a list of scores' (json.loads yields list); models that reply with 'true'/'0.8' (valid JSON scalars, not dicts).","commonSituations":"Judge prompt instructs the wrong output shape (list instead of object); swapping judge models where the new model ignores JSON-mode instructions; omitting response_format={'type':'json_object'}; few-shot examples in the prompt showing arrays.","solutions":["Rewrite the judge prompt to demand a single top-level JSON object, e.g. {\\\"verdict\\\": ..., \\\"score\\\": ...}, and show an object-shaped example.","Force structured output: pass response_format={'type':'json_object'} (or a json_schema) on the judge model call.","Use a judge model that reliably follows JSON instructions (gpt-4o class) instead of a loose instruct model.","If you call parse_json_verdict yourself, pre-check the shape and re-ask the judge on non-dict output."],"exampleFix":"// before (judge prompt)\nReturn the scores as JSON.\n\n# after\nReturn ONLY a JSON object like {\"pass\": true, \"reason\": \"...\"}. No arrays, no bare values.","handlingStrategy":"validation","validationCode":"import json\n\ndef verdict_is_json_object(text: str) -> bool:\n    try:\n        return isinstance(json.loads(text), dict)\n    except json.JSONDecodeError:\n        return False","typeGuard":null,"tryCatchPattern":"from litellm.litellm_core_utils.llm_judge import parse_json_verdict\ntry:\n    verdict = parse_json_verdict(raw)\nexcept (ValueError, json.JSONDecodeError):\n    verdict = parse_json_verdict(retry_judge_call())  # one re-ask with a stricter prompt","preventionTips":["Always request JSON mode (response_format={'type':'json_object'}) on judge calls.","Show an object-shaped example in the judge prompt and forbid arrays explicitly.","Wrap judge parsing with a bounded retry (1-2 re-asks) before failing the guardrail."],"tags":["llm-as-a-judge","json-parsing","guardrails","prompt-engineering"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}