BerriAI/litellm · error · ValueError

baseline_model is required when direction is 'reverse'

Error message

baseline_model is required when direction is 'reverse'

What it means

Cross-field model validator on StartShadowEvalRequest: when direction == 'reverse', a baseline_model must be supplied to compare against, and its absence makes the shadow-eval job unrunnable, so validation fails at request parse time.

Source

Thrown at litellm/types/management_endpoints/auto_router_endpoints.py:217

    max_turns: int = Field(
        default=200,
        ge=1,
        le=2000,
        description=(
            "Sample budget: the job judges at most this many turns, then completes. This is also the spend "
            "bound; expected judge cost is roughly max_turns times one judge call"
        ),
    )

    @field_validator("shadow_percentage")
    @classmethod
    def _round_percentage(cls, value: float) -> float:
        return round(value, 2)

    @model_validator(mode="after")
    def _baseline_model_matches_direction(self) -> "StartShadowEvalRequest":
        if self.direction == "reverse" and self.baseline_model is None:
            raise ValueError("baseline_model is required when direction is 'reverse'")
        if self.direction == "forward" and self.baseline_model is not None:
            raise ValueError("baseline_model is only meaningful when direction is 'reverse'")
        return self


class ShadowEvalSlice(BaseModel):
    """Judge outcomes for one slice of a job's verdicts (a router tier, or one of the
    models that served the real arm)."""

    group: str
    turn_count: int
    real_win_rate_pct: float = Field(
        description=(
            "Share of judged turns the real arm won, meaning the response the caller actually received: "
            "the key's own model in forward mode, the router's pick in reverse"
        )
    )
    shadow_win_rate_pct: float = Field(

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Set baseline_model when using direction='reverse' so the comparison has a reference model.
2. Or use direction='forward' if no baseline comparison is wanted.

When it happens

Trigger: Thrown at litellm/types/management_endpoints/auto_router_endpoints.py:217 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/9234c8a691e40355. Report an issue: GitHub.