{"record":{"id":"46b36eb41db5fc98","repo":"BerriAI/litellm","slug":"llm-judge-rejected-response-score-below-threshold","errorCode":null,"errorMessage":"LLM judge rejected response: score below threshold","messagePattern":"LLM judge rejected response: score below threshold","errorType":"http","errorClass":"HTTPException","httpStatus":422,"severity":"error","filePath":"litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py","lineNumber":204,"sourceCode":"                \"overall_score\": overall_score,\n                \"passed\": passed,\n                \"judge_model\": self.judge_model,\n                \"threshold\": self.overall_threshold,\n                \"verdicts\": judge_result.get(\"verdicts\", []),\n            }\n            _metadata: Final = request_data.setdefault(\"metadata\", {})\n            existing: Final = _metadata.get(\"eval_information\")\n            if isinstance(existing, list):\n                existing.append(eval_info)\n            elif existing is not None:\n                _metadata[\"eval_information\"] = [existing, eval_info]\n            else:\n                _metadata[\"eval_information\"] = eval_info\n\n            if not passed:\n                status = \"guardrail_intervened\"\n                if self.on_failure == \"block\":\n                    raise HTTPException(\n                        status_code=422,\n                        detail={\n                            \"error\": \"LLM judge rejected response: score below threshold\",\n                            \"overall_score\": overall_score,\n                            \"threshold\": self.overall_threshold,\n                            \"verdicts\": judge_result.get(\"verdicts\", []),\n                        },\n                    )\n\n            return inputs\n\n        except HTTPException:\n            raise\n        except Exception as e:\n            verbose_logger.warning(\"llm_as_a_judge guardrail unexpected error: %s\", e)\n            return inputs\n        finally:\n            self.add_standard_logging_guardrail_information_to_request_data(","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py#L186-L222","documentation":"The llm_as_a_judge guardrail scores each LLM response in a post_call hook against weighted criteria. When the weighted overall_score falls below overall_threshold (default 80) and on_failure is 'block' (the default), the proxy raises HTTPException 422 with the score, threshold, and per-criterion verdicts. With on_failure='log' the failure is recorded under metadata.eval_information and the response passes through.","triggerScenarios":"A completion routed through a deployment with the llm_as_a_judge guardrail where the judged response's weighted score lands under overall_threshold (e.g. 74 vs threshold 80) while on_failure is 'block'. The detail body contains overall_score, threshold, and verdicts showing which criteria failed.","commonSituations":"Quality gates fronting cheap/small models whose answers fail judge criteria; thresholds set optimistically at 80+; judge criteria phrased so a whole class of valid answers scores low; double latency and cost per request because every response is judged.","solutions":["Lower overall_threshold in litellm_params to a value the judged model can realistically meet","Set on_failure: 'log' to record judge failures in metadata.eval_information without blocking responses","Improve the underlying model or its system prompt so responses satisfy the judge criteria","Rebalance criterion weights so critical criteria dominate and minor ones cannot drag the overall score under the threshold"],"exampleFix":"# before - default threshold 80 blocks this model's typical scores\nlitellm_params:\n  judge_model: gpt-4o\n  criteria: [{name: grounded, weight: 100}]\n\n# after - realistic threshold, failures logged not blocked\nlitellm_params:\n  judge_model: gpt-4o\n  criteria: [{name: grounded, weight: 100}]\n  overall_threshold: 70\n  on_failure: 'log'","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"import openai  \n  \ntry:  \n    resp = client.chat.completions.create(model=judge_model_deployment, messages=msgs)  \nexcept openai.UnprocessableEntityError as e:  \n    body = e.body if isinstance(e.body, dict) else {}  \n    if body.get(\"error\", \"\").startswith(\"LLM judge rejected\"):  \n        log_judge_failure(body.get(\"overall_score\"), body.get(\"verdicts\"))  \n        return regenerate_with_stricter_prompt(msgs)  \n    raise","preventionTips":["Set overall_threshold from measured scores of your real model, not aspiration - run a benchmark set through the judge first","Use on_failure: 'log' in staging to gather score distributions before enabling 'block' in production","Watch judge latency and cost: every request is judged, so consider judging a sample instead of all traffic"],"tags":["llm-as-a-judge","guardrail","quality-gate","http-422"],"backgroundTag":"quality-score-below-threshold","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}