BerriAI/litellm · error · ValueError

llm_as_a_judge guardrail requires at least one criterion

Error message

llm_as_a_judge guardrail requires at least one criterion

What it means

initialize_guardrail() for llm_as_a_judge defaults criteria to [] and raises ValueError when the list is empty - a judge with nothing to evaluate is meaningless. Criteria are read from litellm_params (or the guardrail entry) and each criterion carries a weight used in scoring.

Source

Thrown at litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py:247

                event_type=GuardrailEventHooks.post_call,
            )


def initialize_guardrail(
    litellm_params: "LitellmParams",
    guardrail: "Guardrail",
) -> LLMAsAJudgeGuardrail:
    guardrail_name: Final = guardrail.get("guardrail_name")
    if not guardrail_name:
        raise ValueError("llm_as_a_judge guardrail requires a guardrail_name")

    judge_model: Final = _get_litellm_param(litellm_params, guardrail, "judge_model")
    if not judge_model:
        raise ValueError("llm_as_a_judge guardrail requires judge_model in litellm_params")

    criteria: Final = _get_litellm_param(litellm_params, guardrail, "criteria") or []
    if not criteria:
        raise ValueError("llm_as_a_judge guardrail requires at least one criterion")

    weight_total: Final = sum(float(c.get("weight", 0)) for c in criteria)
    if abs(weight_total - 100) > 0.5:
        raise ValueError(f"llm_as_a_judge criterion weights must sum to 100 (got {weight_total})")

    on_failure: Final = _get_litellm_param(litellm_params, guardrail, "on_failure", "block")
    if on_failure not in _VALID_ON_FAILURE:
        raise ValueError(f"llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'")

    overall_threshold: Final = float(_get_litellm_param(litellm_params, guardrail, "overall_threshold", 80.0))

    mode: Final = _get_litellm_param(litellm_params, guardrail, "mode")
    event_hook: GuardrailEventHooks | None = None
    if isinstance(mode, str) and mode in {e.value for e in GuardrailEventHooks}:
        event_hook = GuardrailEventHooks(mode)

    instance: Final = LLMAsAJudgeGuardrail(
        guardrail_name=guardrail_name,

View on GitHub (pinned to 77b7c6c40c)

Solutions

  1. Add a non-empty criteria list to litellm_params, e.g. criteria: [{name: grounded, weight: 100}]
  2. Check YAML indentation so criteria sits directly under litellm_params
  3. Use the exact key 'criteria' (singular variants are silently ignored)

Example fix

# before
 litellm_params:
   judge_model: gpt-4o

# after
 litellm_params:
   judge_model: gpt-4o
   criteria:
     - name: grounded
       weight: 100
Defensive patterns

Strategy: validation

Validate before calling

lp = guardrail_entry["litellm_params"]  
criteria = lp.get("criteria")  
assert isinstance(criteria, list) and len(criteria) > 0, (  
    "llm_as_a_judge requires a non-empty 'criteria' list under litellm_params"  
)

Type guard

def has_nonempty_criteria(lp: object) -> bool:  
    return isinstance(lp, dict) and isinstance(lp.get("criteria"), list) and len(lp["criteria"]) > 0

Prevention

When it happens

Trigger: A litellm_llm_as_a_judge guardrail entry where the criteria key is missing, set to an empty list, or typo'd (criterion, criterias) so the fallback yields [].

Common situations: Minimal starter configs that only set judge_model; refactors that move criteria elsewhere and leave the list empty; YAML indentation putting criteria under the wrong parent so it is never read.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/9d193a4967a85c2c. Report an issue: GitHub.