BerriAI/litellm · error · ValueError

llm_as_a_judge on_failure must be 'block' or 'log', got '{on

Error message

llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'

What it means

initialize_guardrail() validates on_failure against the allowed set {'block', 'log'} and raises ValueError with the offending value otherwise. on_failure controls whether a below-threshold judge score blocks the response (HTTP 422) or is merely logged.

Source

Thrown at litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py:255

    guardrail_name: Final = guardrail.get("guardrail_name")
    if not guardrail_name:
        raise ValueError("llm_as_a_judge guardrail requires a guardrail_name")

    judge_model: Final = _get_litellm_param(litellm_params, guardrail, "judge_model")
    if not judge_model:
        raise ValueError("llm_as_a_judge guardrail requires judge_model in litellm_params")

    criteria: Final = _get_litellm_param(litellm_params, guardrail, "criteria") or []
    if not criteria:
        raise ValueError("llm_as_a_judge guardrail requires at least one criterion")

    weight_total: Final = sum(float(c.get("weight", 0)) for c in criteria)
    if abs(weight_total - 100) > 0.5:
        raise ValueError(f"llm_as_a_judge criterion weights must sum to 100 (got {weight_total})")

    on_failure: Final = _get_litellm_param(litellm_params, guardrail, "on_failure", "block")
    if on_failure not in _VALID_ON_FAILURE:
        raise ValueError(f"llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'")

    overall_threshold: Final = float(_get_litellm_param(litellm_params, guardrail, "overall_threshold", 80.0))

    mode: Final = _get_litellm_param(litellm_params, guardrail, "mode")
    event_hook: GuardrailEventHooks | None = None
    if isinstance(mode, str) and mode in {e.value for e in GuardrailEventHooks}:
        event_hook = GuardrailEventHooks(mode)

    instance: Final = LLMAsAJudgeGuardrail(
        guardrail_name=guardrail_name,
        judge_model=judge_model,
        criteria=criteria,
        overall_threshold=overall_threshold,
        on_failure=on_failure,
        event_hook=event_hook,
        default_on=bool(_get_litellm_param(litellm_params, guardrail, "default_on", False)),
    )
    litellm.logging_callback_manager.add_litellm_callback(instance)

View on GitHub (pinned to 77b7c6c40c)

Solutions

Set on_failure: 'log' if you want failures recorded without blocking
Set on_failure: 'block' (or omit it - block is the default) to keep the 422 behavior
Check casing and spelling: the value must be lowercase 'block' or 'log'

Example fix

# before
 litellm_params:
   on_failure: alert

# after
 litellm_params:
   on_failure: log

Defensive patterns

Strategy: validation

Validate before calling

VALID_ON_FAILURE = {"block", "log"}  
on_failure = litellm_params.get("on_failure", "block")  
assert on_failure in VALID_ON_FAILURE, (  
    f"on_failure must be one of {sorted(VALID_ON_FAILURE)}, got {on_failure!r}"  
)

Type guard

def is_valid_on_failure(v: object) -> bool:  
    return isinstance(v, str) and v in {"block", "log"}

Prevention

Remember the vocabulary is block/log (lowercase) - not other guardrails' 'alert'
Omit on_failure entirely if you want the default 'block' behavior

When it happens

Trigger: A litellm_llm_as_a_judge guardrail with on_failure set to anything except 'block' or 'log' - e.g. 'raise', 'BLOCK' (case-sensitive), 'ignore', or 'warn'.

Common situations: Naming borrowed from other guardrails' on_violation vocabularies ('alert'); uppercase values from env-var templating; typos.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/b56433fe78c63bfe. Report an issue: GitHub.