BerriAI/litellm · error · ValueError

llm_as_a_judge guardrail requires judge_model in…

Error message

llm_as_a_judge guardrail requires judge_model in litellm_params

What it means

llm_as_a_judge needs a model to evaluate responses; initialize_guardrail() resolves judge_model via _get_litellm_param from litellm_params (falling back to the guardrail entry) and raises ValueError when it is absent or empty. It fires at config load time, before any traffic is judged.

Solutions

1. Add judge_model to litellm_params, e.g. judge_model: gpt-4o
2. Make sure the key is spelled exactly judge_model and the value is non-empty
3. Confirm the judge model is deployed/routable on the proxy so the guardrail can call it

Example fix

# before
 litellm_params:
   criteria: [{name: grounded, weight: 100}]

# after
 litellm_params:
   judge_model: gpt-4o
   criteria: [{name: grounded, weight: 100}]

REQUIRED_LLM_JUDGE_KEYS = {"judge_model", "criteria"}  
  
def validate_llm_judge_params(litellm_params: dict) -> None:  
    missing = REQUIRED_LLM_JUDGE_KEYS - set(litellm_params)  
    assert not missing, f"llm_as_a_judge litellm_params missing: {sorted(missing)}"  
    assert litellm_params["judge_model"], "judge_model must be non-empty"

def is_valid_judge_config(lp: object) -> bool:  
    return (  
        isinstance(lp, dict)  
        and isinstance(lp.get("judge_model"), str)  
        and bool(lp["judge_model"].strip())  
        and isinstance(lp.get("criteria"), list)  
        and len(lp["criteria"]) > 0  
    )

When it happens

Trigger: A litellm_llm_as_a_judge guardrail entry whose litellm_params block has criteria but no judge_model key (or judge_model set to an empty string).

Common situations: Configs that specify criteria and threshold but assume the judge reuses the deployment's model; typos like judge-model or judgemodel; empty value left from templating.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• llm_as_a_judge criterion weights must sum to 100
• llm_as_a_judge guardrail requires a guardrail_name
• llm_as_a_judge guardrail requires at least one criterion
• llm_as_a_judge on_failure must be 'block' or 'log', got
• api_base is required for Generic Guardrail API. Set…

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/f4cf2a0a3af7d157. Report an issue: GitHub.

Appendix: source

Thrown at litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py:243

                request_data=request_data,
                guardrail_status=status,
                start_time=start_time.timestamp(),
                end_time=datetime.now().timestamp(),
                event_type=GuardrailEventHooks.post_call,
            )


def initialize_guardrail(
    litellm_params: "LitellmParams",
    guardrail: "Guardrail",
) -> LLMAsAJudgeGuardrail:
    guardrail_name: Final = guardrail.get("guardrail_name")
    if not guardrail_name:
        raise ValueError("llm_as_a_judge guardrail requires a guardrail_name")

    judge_model: Final = _get_litellm_param(litellm_params, guardrail, "judge_model")
    if not judge_model:
        raise ValueError("llm_as_a_judge guardrail requires judge_model in litellm_params")

    criteria: Final = _get_litellm_param(litellm_params, guardrail, "criteria") or []
    if not criteria:
        raise ValueError("llm_as_a_judge guardrail requires at least one criterion")

    weight_total: Final = sum(float(c.get("weight", 0)) for c in criteria)
    if abs(weight_total - 100) > 0.5:
        raise ValueError(f"llm_as_a_judge criterion weights must sum to 100 (got {weight_total})")

    on_failure: Final = _get_litellm_param(litellm_params, guardrail, "on_failure", "block")
    if on_failure not in _VALID_ON_FAILURE:
        raise ValueError(f"llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'")

    overall_threshold: Final = float(_get_litellm_param(litellm_params, guardrail, "overall_threshold", 80.0))

    mode: Final = _get_litellm_param(litellm_params, guardrail, "mode")
    event_hook: GuardrailEventHooks | None = None
    if isinstance(mode, str) and mode in {e.value for e in GuardrailEventHooks}:

View on GitHub (pinned to 77b7c6c40c)