BerriAI/litellm · error · ValueError

baseline_model is only meaningful when direction is 'reverse

Error message

baseline_model is only meaningful when direction is 'reverse'

What it means

Pydantic model_validator on StartShadowEvalRequest: baseline_model was supplied while direction is not 'reverse'; in forward mode the baseline is meaningless, so the combination is rejected as likely misconfiguration.

Source

Thrown at litellm/types/management_endpoints/auto_router_endpoints.py:219

        ge=1,
        le=2000,
        description=(
            "Sample budget: the job judges at most this many turns, then completes. This is also the spend "
            "bound; expected judge cost is roughly max_turns times one judge call"
        ),
    )

    @field_validator("shadow_percentage")
    @classmethod
    def _round_percentage(cls, value: float) -> float:
        return round(value, 2)

    @model_validator(mode="after")
    def _baseline_model_matches_direction(self) -> "StartShadowEvalRequest":
        if self.direction == "reverse" and self.baseline_model is None:
            raise ValueError("baseline_model is required when direction is 'reverse'")
        if self.direction == "forward" and self.baseline_model is not None:
            raise ValueError("baseline_model is only meaningful when direction is 'reverse'")
        return self


class ShadowEvalSlice(BaseModel):
    """Judge outcomes for one slice of a job's verdicts (a router tier, or one of the
    models that served the real arm)."""

    group: str
    turn_count: int
    real_win_rate_pct: float = Field(
        description=(
            "Share of judged turns the real arm won, meaning the response the caller actually received: "
            "the key's own model in forward mode, the router's pick in reverse"
        )
    )
    shadow_win_rate_pct: float = Field(
        description=(
            "Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: "

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Remove baseline_model when direction is not 'reverse'.
2. Or set direction='reverse' if a baseline comparison is intended.

When it happens

Trigger: Thrown at litellm/types/management_endpoints/auto_router_endpoints.py:219 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/5d344b5e6c86bb01. Report an issue: GitHub.