{"record":{"id":"9d193a4967a85c2c","repo":"BerriAI/litellm","slug":"llm-as-a-judge-guardrail-requires-at-least-one-cri","errorCode":null,"errorMessage":"llm_as_a_judge guardrail requires at least one criterion","messagePattern":"llm_as_a_judge guardrail requires at least one criterion","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py","lineNumber":247,"sourceCode":"                event_type=GuardrailEventHooks.post_call,\n            )\n\n\ndef initialize_guardrail(\n    litellm_params: \"LitellmParams\",\n    guardrail: \"Guardrail\",\n) -> LLMAsAJudgeGuardrail:\n    guardrail_name: Final = guardrail.get(\"guardrail_name\")\n    if not guardrail_name:\n        raise ValueError(\"llm_as_a_judge guardrail requires a guardrail_name\")\n\n    judge_model: Final = _get_litellm_param(litellm_params, guardrail, \"judge_model\")\n    if not judge_model:\n        raise ValueError(\"llm_as_a_judge guardrail requires judge_model in litellm_params\")\n\n    criteria: Final = _get_litellm_param(litellm_params, guardrail, \"criteria\") or []\n    if not criteria:\n        raise ValueError(\"llm_as_a_judge guardrail requires at least one criterion\")\n\n    weight_total: Final = sum(float(c.get(\"weight\", 0)) for c in criteria)\n    if abs(weight_total - 100) > 0.5:\n        raise ValueError(f\"llm_as_a_judge criterion weights must sum to 100 (got {weight_total})\")\n\n    on_failure: Final = _get_litellm_param(litellm_params, guardrail, \"on_failure\", \"block\")\n    if on_failure not in _VALID_ON_FAILURE:\n        raise ValueError(f\"llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'\")\n\n    overall_threshold: Final = float(_get_litellm_param(litellm_params, guardrail, \"overall_threshold\", 80.0))\n\n    mode: Final = _get_litellm_param(litellm_params, guardrail, \"mode\")\n    event_hook: GuardrailEventHooks | None = None\n    if isinstance(mode, str) and mode in {e.value for e in GuardrailEventHooks}:\n        event_hook = GuardrailEventHooks(mode)\n\n    instance: Final = LLMAsAJudgeGuardrail(\n        guardrail_name=guardrail_name,","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py#L229-L265","documentation":"initialize_guardrail() for llm_as_a_judge defaults criteria to [] and raises ValueError when the list is empty - a judge with nothing to evaluate is meaningless. Criteria are read from litellm_params (or the guardrail entry) and each criterion carries a weight used in scoring.","triggerScenarios":"A litellm_llm_as_a_judge guardrail entry where the criteria key is missing, set to an empty list, or typo'd (criterion, criterias) so the fallback yields [].","commonSituations":"Minimal starter configs that only set judge_model; refactors that move criteria elsewhere and leave the list empty; YAML indentation putting criteria under the wrong parent so it is never read.","solutions":["Add a non-empty criteria list to litellm_params, e.g. criteria: [{name: grounded, weight: 100}]","Check YAML indentation so criteria sits directly under litellm_params","Use the exact key 'criteria' (singular variants are silently ignored)"],"exampleFix":"# before\n litellm_params:\n   judge_model: gpt-4o\n\n# after\n litellm_params:\n   judge_model: gpt-4o\n   criteria:\n     - name: grounded\n       weight: 100","handlingStrategy":"validation","validationCode":"lp = guardrail_entry[\"litellm_params\"]  \ncriteria = lp.get(\"criteria\")  \nassert isinstance(criteria, list) and len(criteria) > 0, (  \n    \"llm_as_a_judge requires a non-empty 'criteria' list under litellm_params\"  \n)","typeGuard":"def has_nonempty_criteria(lp: object) -> bool:  \n    return isinstance(lp, dict) and isinstance(lp.get(\"criteria\"), list) and len(lp[\"criteria\"]) > 0","tryCatchPattern":null,"preventionTips":["Check YAML indentation: criteria must sit under litellm_params, not the guardrail root","Use the exact key 'criteria' - singular or pluralized variants are silently ignored"],"tags":["llm-as-a-judge","guardrail","config","startup"],"backgroundTag":"config-validation-failed","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}