BerriAI/litellm · error · ValueError

baseline_model is set for exactly the reverse jobs

Error message

baseline_model is set for exactly the reverse jobs

What it means

ActiveShadowEvalJob is a Pydantic model with a model_validator enforcing that baseline_model is set if and only if direction == 'reverse'. The ValueError fires in both mismatch directions: a forward job (direction='forward'/'pre-call') that carries a baseline_model, or a reverse job that omits it. Reverse jobs duplicate traffic onto a fixed baseline model; forward jobs duplicate onto the router itself, so a baseline is meaningless there.

Source

Thrown at litellm/integrations/shadow_eval_logger.py:243

    id: str
    router_name: str
    direction: ShadowEvalDirection = "forward"
    baseline_model: str | None = None
    shadow_percentage: float
    judge_model: str
    max_turns: int
    ends_at: datetime
    attempts: int = 0

    @field_validator("ends_at")
    @classmethod
    def _as_utc(cls, value: datetime) -> datetime:
        return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value

    @model_validator(mode="after")
    def _baseline_model_matches_direction(self) -> "ActiveShadowEvalJob":
        if (self.baseline_model is not None) != (self.direction == "reverse"):
            raise ValueError("baseline_model is set for exactly the reverse jobs")
        return self

    @property
    def shadow_target(self) -> str:
        """The model the duplicated arm calls: the router itself for a forward job, the
        fixed baseline for a reverse one. Total because the validator above pins
        baseline_model to reverse jobs and only those."""
        return self.baseline_model or self.router_name


def _as_active_job(record: object, attempts: int) -> ActiveShadowEvalJob | None:
    """The sampling path's view of one job row, or None for a row it cannot sample: an
    unknown direction, or a reverse job with no baseline model to duplicate against.
    Failing closed here is what keeps the dispatch path total."""
    try:
        job: Final = ActiveShadowEvalJob.model_validate(record)
    except ValidationError as e:
        verbose_logger.debug("shadow_eval: skipping unsamplable job row: %s", e)

View on GitHub (pinned to 6c2dcb801b)

Solutions

If direction is 'reverse', set baseline_model to the model the duplicated arm should call (e.g. 'gpt-4o-mini')
If direction is 'forward', remove baseline_model entirely
Validate before submission: (baseline_model is not None) == (direction == 'reverse')

Example fix

# before
job = {
    "direction": "reverse",
    "router_name": "prod-router",
    # baseline_model missing -> ValueError
}

# after
job = {
    "direction": "reverse",
    "router_name": "prod-router",
    "baseline_model": "gpt-4o-mini",
}

Defensive patterns

Strategy: validation

Validate before calling

def is_valid_shadow_job(payload: dict) -> bool:
    has_baseline = payload.get("baseline_model") is not None
    is_reverse = payload.get("direction") == "reverse"
    return has_baseline == is_reverse

assert is_valid_shadow_job(job_payload), "baseline_model must be set exactly for reverse jobs"

Type guard

from typing import Any

def baseline_matches_direction(payload: dict[str, Any]) -> bool:
    return (payload.get("baseline_model") is not None) == (payload.get("direction") == "reverse")

Try / catch

from pydantic import ValidationError

try:
    job = ActiveShadowEvalJob(**job_payload)
except ValidationError as e:
    if "baseline_model is set for exactly the reverse jobs" in str(e):
        # autofix or reject with a targeted message
        raise ValueError("Set baseline_model for reverse jobs, remove it for forward jobs") from e
    raise

Prevention

Validate job payloads client-side before submitting to the shadow-eval API
When flipping a job's direction, re-check the baseline_model field in the same edit
Document the invariant next to the job form: reverse duplicates to a fixed baseline, forward duplicates to the router

When it happens

Trigger: Creating a shadow-eval job via the proxy UI/API with direction='reverse' but no baseline_model; or pasting a job config that includes baseline_model while leaving direction as the default forward; programmatic job creation that always populates baseline_model regardless of direction.

Common situations: Copy-pasting job JSON between environments; changing direction after the fact without clearing/adding baseline_model; older job records replayed against the newer validator.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/3011dcf79e87395f. Report an issue: GitHub.