BerriAI/litellm · error · ValueError

llm_as_a_judge guardrail requires a guardrail_name

Error message

llm_as_a_judge guardrail requires a guardrail_name

What it means

initialize_guardrail() for llm_as_a_judge reads guardrail_name from the guardrail entry and refuses to construct the guardrail without it. The ValueError surfaces at proxy startup or config reload, when a guardrails entry for litellm_llm_as_a_judge omits the guardrail_name field.

Source

Thrown at litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py:239

        finally:
            self.add_standard_logging_guardrail_information_to_request_data(
                guardrail_provider="llm_as_a_judge",
                guardrail_json_response=judge_result,
                request_data=request_data,
                guardrail_status=status,
                start_time=start_time.timestamp(),
                end_time=datetime.now().timestamp(),
                event_type=GuardrailEventHooks.post_call,
            )


def initialize_guardrail(
    litellm_params: "LitellmParams",
    guardrail: "Guardrail",
) -> LLMAsAJudgeGuardrail:
    guardrail_name: Final = guardrail.get("guardrail_name")
    if not guardrail_name:
        raise ValueError("llm_as_a_judge guardrail requires a guardrail_name")

    judge_model: Final = _get_litellm_param(litellm_params, guardrail, "judge_model")
    if not judge_model:
        raise ValueError("llm_as_a_judge guardrail requires judge_model in litellm_params")

    criteria: Final = _get_litellm_param(litellm_params, guardrail, "criteria") or []
    if not criteria:
        raise ValueError("llm_as_a_judge guardrail requires at least one criterion")

    weight_total: Final = sum(float(c.get("weight", 0)) for c in criteria)
    if abs(weight_total - 100) > 0.5:
        raise ValueError(f"llm_as_a_judge criterion weights must sum to 100 (got {weight_total})")

    on_failure: Final = _get_litellm_param(litellm_params, guardrail, "on_failure", "block")
    if on_failure not in _VALID_ON_FAILURE:
        raise ValueError(f"llm_as_a_judge on_failure must be 'block' or 'log', got '{on_failure}'")

    overall_threshold: Final = float(_get_litellm_param(litellm_params, guardrail, "overall_threshold", 80.0))

View on GitHub (pinned to 77b7c6c40c)

Solutions

  1. Add guardrail_name: <unique-name> to the guardrail entry in config.yaml
  2. If generating config programmatically, assert the key exists before writing the guardrail entry

Example fix

# before
 guardrails:
   - guardrail: litellm_llm_as_a_judge
     litellm_params:
       judge_model: gpt-4o

# after
 guardrails:
   - guardrail: litellm_llm_as_a_judge
     guardrail_name: my-judge
     litellm_params:
       judge_model: gpt-4o
Defensive patterns

Strategy: validation

Validate before calling

def validate_guardrail_config(guardrails: list[dict]) -> None:  
    for g in guardrails:  
        assert g.get("guardrail_name"), f"guardrail {g.get('guardrail')!r} is missing guardrail_name"  
  
validate_guardrail_config(config["guardrails"])  # run before proxy start / in CI

Type guard

def has_guardrail_name(entry: object) -> bool:  
    return isinstance(entry, dict) and isinstance(entry.get("guardrail_name"), str) and bool(entry["guardrail_name"].strip())

Prevention

When it happens

Trigger: A guardrails config entry with guardrail: litellm_llm_as_a_judge but no guardrail_name key at the entry's top level.

Common situations: Copy-pasted guardrail YAML where the name line got dropped; configs migrated from another format that treated guardrail_name as optional; entries built programmatically without the name.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/66be645c3d73d843. Report an issue: GitHub.