{"record":{"id":"3011dcf79e87395f","repo":"BerriAI/litellm","slug":"baseline-model-is-set-for-exactly-the-reverse-jobs","errorCode":null,"errorMessage":"baseline_model is set for exactly the reverse jobs","messagePattern":"baseline_model is set for exactly the reverse jobs","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/integrations/shadow_eval_logger.py","lineNumber":243,"sourceCode":"    id: str\n    router_name: str\n    direction: ShadowEvalDirection = \"forward\"\n    baseline_model: str | None = None\n    shadow_percentage: float\n    judge_model: str\n    max_turns: int\n    ends_at: datetime\n    attempts: int = 0\n\n    @field_validator(\"ends_at\")\n    @classmethod\n    def _as_utc(cls, value: datetime) -> datetime:\n        return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value\n\n    @model_validator(mode=\"after\")\n    def _baseline_model_matches_direction(self) -> \"ActiveShadowEvalJob\":\n        if (self.baseline_model is not None) != (self.direction == \"reverse\"):\n            raise ValueError(\"baseline_model is set for exactly the reverse jobs\")\n        return self\n\n    @property\n    def shadow_target(self) -> str:\n        \"\"\"The model the duplicated arm calls: the router itself for a forward job, the\n        fixed baseline for a reverse one. Total because the validator above pins\n        baseline_model to reverse jobs and only those.\"\"\"\n        return self.baseline_model or self.router_name\n\n\ndef _as_active_job(record: object, attempts: int) -> ActiveShadowEvalJob | None:\n    \"\"\"The sampling path's view of one job row, or None for a row it cannot sample: an\n    unknown direction, or a reverse job with no baseline model to duplicate against.\n    Failing closed here is what keeps the dispatch path total.\"\"\"\n    try:\n        job: Final = ActiveShadowEvalJob.model_validate(record)\n    except ValidationError as e:\n        verbose_logger.debug(\"shadow_eval: skipping unsamplable job row: %s\", e)","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/integrations/shadow_eval_logger.py#L225-L261","documentation":"ActiveShadowEvalJob is a Pydantic model with a model_validator enforcing that baseline_model is set if and only if direction == 'reverse'. The ValueError fires in both mismatch directions: a forward job (direction='forward'/'pre-call') that carries a baseline_model, or a reverse job that omits it. Reverse jobs duplicate traffic onto a fixed baseline model; forward jobs duplicate onto the router itself, so a baseline is meaningless there.","triggerScenarios":"Creating a shadow-eval job via the proxy UI/API with direction='reverse' but no baseline_model; or pasting a job config that includes baseline_model while leaving direction as the default forward; programmatic job creation that always populates baseline_model regardless of direction.","commonSituations":"Copy-pasting job JSON between environments; changing direction after the fact without clearing/adding baseline_model; older job records replayed against the newer validator.","solutions":["If direction is 'reverse', set baseline_model to the model the duplicated arm should call (e.g. 'gpt-4o-mini')","If direction is 'forward', remove baseline_model entirely","Validate before submission: (baseline_model is not None) == (direction == 'reverse')"],"exampleFix":"# before\njob = {\n    \"direction\": \"reverse\",\n    \"router_name\": \"prod-router\",\n    # baseline_model missing -> ValueError\n}\n\n# after\njob = {\n    \"direction\": \"reverse\",\n    \"router_name\": \"prod-router\",\n    \"baseline_model\": \"gpt-4o-mini\",\n}","handlingStrategy":"validation","validationCode":"def is_valid_shadow_job(payload: dict) -> bool:\n    has_baseline = payload.get(\"baseline_model\") is not None\n    is_reverse = payload.get(\"direction\") == \"reverse\"\n    return has_baseline == is_reverse\n\nassert is_valid_shadow_job(job_payload), \"baseline_model must be set exactly for reverse jobs\"","typeGuard":"from typing import Any\n\ndef baseline_matches_direction(payload: dict[str, Any]) -> bool:\n    return (payload.get(\"baseline_model\") is not None) == (payload.get(\"direction\") == \"reverse\")","tryCatchPattern":"from pydantic import ValidationError\n\ntry:\n    job = ActiveShadowEvalJob(**job_payload)\nexcept ValidationError as e:\n    if \"baseline_model is set for exactly the reverse jobs\" in str(e):\n        # autofix or reject with a targeted message\n        raise ValueError(\"Set baseline_model for reverse jobs, remove it for forward jobs\") from e\n    raise","preventionTips":["Validate job payloads client-side before submitting to the shadow-eval API","When flipping a job's direction, re-check the baseline_model field in the same edit","Document the invariant next to the job form: reverse duplicates to a fixed baseline, forward duplicates to the router"],"tags":["shadow-eval","pydantic","validation","configuration"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}