{"record":{"id":"fde258ed92592022","repo":"abhigyanpatwari/GitNexus","slug":"flag-must-not-use-a-mutable-auto-latest-model-al","errorCode":null,"errorMessage":"{flag} must not use a mutable auto/latest model alias: {model!r}","messagePattern":"(.+?) must not use a mutable auto/latest model alias: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"eval/workflow_bench/runner_tasks.py","lineNumber":21,"sourceCode":"from __future__ import annotations\n\nimport hashlib\nimport re\nfrom collections.abc import Mapping\nfrom pathlib import Path\nfrom typing import Any\n\nfrom .oracle_assets import TaskOracleSnapshot, capture_task_oracles, validate_oracle_declaration\nfrom .process_control import run_checked, run_managed\nfrom .task_assets import TaskAssetCache, capture_task_dependency_binding\n\n\ndef normalized_model_identifier(value: str | None, *, flag: str = \"--model\") -> str:\n    model = (value or \"\").strip()\n    if not model:\n        raise ValueError(f\"{flag} must name a nonblank, versioned model\")\n    if re.search(r\"(?:^|[-/@:])(?:auto|latest)$\", model.casefold()):\n        raise ValueError(f\"{flag} must not use a mutable auto/latest model alias: {model!r}\")\n    return model\n\n\ndef select_tasks(tasks: list[Any], *, include_expensive: bool) -> tuple[list[dict[str, Any]], list[str]]:\n    \"\"\"Validate task metadata and filter opt-in expensive scenarios.\"\"\"\n\n    selected: list[dict[str, Any]] = []\n    skipped: list[str] = []\n    seen: set[str] = set()\n    required_strings = (\"id\", \"class\", \"repo\", \"prompt\", \"verify\")\n    optional_strings = (\"ref\", \"setup\")\n    for index, raw_task in enumerate(tasks):\n        if not isinstance(raw_task, Mapping):\n            raise ValueError(f\"task {index} must be a mapping\")\n        task = dict(raw_task)\n        for field in required_strings:\n            if not isinstance(task.get(field), str) or not task[field].strip():\n                raise ValueError(f\"task {index} requires a nonblank string {field}\")","sourceCodeStart":3,"sourceCodeEnd":39,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/runner_tasks.py#L3-L39","documentation":"Thrown by normalized_model_identifier when the model id ends with a mutable alias segment — 'auto' or 'latest' — separated by -, /, @, or :, or is exactly 'auto'/'latest'. Such aliases point at shifting targets over time, which destroys the run-to-run reproducibility the benchmark exists to measure, so they are rejected even though they are syntactically valid model names.","triggerScenarios":"re.search(r'(?:^|[-/@:])(?:auto|latest)$', model.casefold()) matches. E.g. 'claude-latest', 'gpt/auto', 'provider/model:latest', 'auto', 'anthropic@latest'.","commonSituations":"A user-friendly alias was copied from a docs page ('use claude-latest'); a CI config pins 'auto' for convenience; a model router default leaked in; the alias resolved fine in dev but drifts in CI over time.","solutions":["Pin the concrete versioned model id, e.g. 'claude-sonnet-4-5-20250929' instead of 'claude-latest'.","Resolve the alias to a versioned id once (via the provider's API) and store that id in the benchmark config.","Add a preflight check that rejects 'auto'/'latest' in your own config loader.","Treat 'latest' in a benchmark config as a bug, not a convenience."],"exampleFix":"// before\nnormalized_model_identifier('claude-sonnet-latest')  # -> raises\n\n// after\nnormalized_model_identifier('claude-sonnet-4-5-20250929')","handlingStrategy":"validation","validationCode":"import re\n\ndef is_pinned_model(value: str) -> bool:\n    return isinstance(value, str) and not re.search(r'(?:^|[-/@:])(?:auto|latest)$', value.casefold())","typeGuard":"import re\n\ndef is_pinned_model(value: str) -> bool:\n    return isinstance(value, str) and bool(value.strip()) and not re.search(r'(?:^|[-/@:])(?:auto|latest)$', value.casefold())","tryCatchPattern":"try:\n    model = normalized_model_identifier(value)\nexcept ValueError as e:\n    if 'mutable auto/latest' in str(e):\n        # resolve the alias to a concrete versioned id and retry\n        raise\n    raise","preventionTips":["Pin concrete versioned model ids in benchmark configs (e.g. claude-sonnet-4-5-20250929).","Resolve 'auto'/'latest' aliases once via the provider API and store the resolved id.","Add is_pinned_model checks in your config loader.","Treat 'latest' in a reproducibility-critical config as a defect."],"tags":["validation","model-config","reproducibility","workflow-bench"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}