BerriAI/litellm · error · ValueError

llm_as_a_judge guardrail requires judge_model in litellm_par

Error message

llm_as_a_judge guardrail requires judge_model in litellm_params

What it means

llm_as_a_judge needs a model to evaluate responses; initialize_guardrail() resolves judge_model via _get_litellm_param from litellm_params (falling back to the guardrail entry) and raises ValueError when it is absent or empty. It fires at config load time, before any traffic is judged.

Source

Thrown at litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py:243

                request_data=request_data,
                guardrail_status=status,
                start_time=start_time.timestamp(),
                end_time=datetime.now().timestamp(),
                event_type=GuardrailEventHooks.post_call,
            )


def initialize_guardrail(
    litellm_params: "LitellmParams",
    guardrail: "Guardrail",
) -> LLMAsAJudgeGuardrail:
    guardrail_name: Final = guardrail.get("guardrail_name")
    if not guardrail_name:
        raise ValueError("llm_as_a_judge guardrail requires a guardrail_name")

    judge_model: Final = _get_litellm_param(litellm_params, guardrail, "judge_model")
    if not judge_model:
        raise ValueError("llm_as_a_judge guardrail requires judge_model in litellm_params")

    criteria: Final = _get_litellm_param(litellm_params, guardrail, "criteria") or []
    if not criteria:
        raise ValueError("llm_as_a_judge guardrail requires at least one criterion")

    weight_total: Final = sum(float(c.get("weight", 0)) for c in criteria)
    if abs(weight_total - 100) > 0.5:
        raise ValueError(f"llm_as_a_judge criterion weights must sum to 100 (got {weight_total})")

    on_failure: Final = _get_litellm_param(litellm_params, guardrail, "on_failure", "block")
    if on_failure not in _VALID_ON_FAILURE:
        raise ValueError(f"llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'")

    overall_threshold: Final = float(_get_litellm_param(litellm_params, guardrail, "overall_threshold", 80.0))

    mode: Final = _get_litellm_param(litellm_params, guardrail, "mode")
    event_hook: GuardrailEventHooks | None = None
    if isinstance(mode, str) and mode in {e.value for e in GuardrailEventHooks}:

View on GitHub (pinned to 77b7c6c40c)

Solutions

  1. Add judge_model to litellm_params, e.g. judge_model: gpt-4o
  2. Make sure the key is spelled exactly judge_model and the value is non-empty
  3. Confirm the judge model is deployed/routable on the proxy so the guardrail can call it

Example fix

# before
 litellm_params:
   criteria: [{name: grounded, weight: 100}]

# after
 litellm_params:
   judge_model: gpt-4o
   criteria: [{name: grounded, weight: 100}]
Defensive patterns

Strategy: validation

Validate before calling

REQUIRED_LLM_JUDGE_KEYS = {"judge_model", "criteria"}  
  
def validate_llm_judge_params(litellm_params: dict) -> None:  
    missing = REQUIRED_LLM_JUDGE_KEYS - set(litellm_params)  
    assert not missing, f"llm_as_a_judge litellm_params missing: {sorted(missing)}"  
    assert litellm_params["judge_model"], "judge_model must be non-empty"

Type guard

def is_valid_judge_config(lp: object) -> bool:  
    return (  
        isinstance(lp, dict)  
        and isinstance(lp.get("judge_model"), str)  
        and bool(lp["judge_model"].strip())  
        and isinstance(lp.get("criteria"), list)  
        and len(lp["criteria"]) > 0  
    )

Prevention

When it happens

Trigger: A litellm_llm_as_a_judge guardrail entry whose litellm_params block has criteria but no judge_model key (or judge_model set to an empty string).

Common situations: Configs that specify criteria and threshold but assume the judge reuses the deployment's model; typos like judge-model or judgemodel; empty value left from templating.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/f4cf2a0a3af7d157. Report an issue: GitHub.