abhigyanpatwari/GitNexus · error · ValueError

{flag} must not use a mutable auto/latest model alias: {mode

Error message

{flag} must not use a mutable auto/latest model alias: {model!r}

What it means

Thrown by normalized_model_identifier when the model id ends with a mutable alias segment — 'auto' or 'latest' — separated by -, /, @, or :, or is exactly 'auto'/'latest'. Such aliases point at shifting targets over time, which destroys the run-to-run reproducibility the benchmark exists to measure, so they are rejected even though they are syntactically valid model names.

Source

Thrown at eval/workflow_bench/runner_tasks.py:21

from __future__ import annotations

import hashlib
import re
from collections.abc import Mapping
from pathlib import Path
from typing import Any

from .oracle_assets import TaskOracleSnapshot, capture_task_oracles, validate_oracle_declaration
from .process_control import run_checked, run_managed
from .task_assets import TaskAssetCache, capture_task_dependency_binding


def normalized_model_identifier(value: str | None, *, flag: str = "--model") -> str:
    model = (value or "").strip()
    if not model:
        raise ValueError(f"{flag} must name a nonblank, versioned model")
    if re.search(r"(?:^|[-/@:])(?:auto|latest)$", model.casefold()):
        raise ValueError(f"{flag} must not use a mutable auto/latest model alias: {model!r}")
    return model


def select_tasks(tasks: list[Any], *, include_expensive: bool) -> tuple[list[dict[str, Any]], list[str]]:
    """Validate task metadata and filter opt-in expensive scenarios."""

    selected: list[dict[str, Any]] = []
    skipped: list[str] = []
    seen: set[str] = set()
    required_strings = ("id", "class", "repo", "prompt", "verify")
    optional_strings = ("ref", "setup")
    for index, raw_task in enumerate(tasks):
        if not isinstance(raw_task, Mapping):
            raise ValueError(f"task {index} must be a mapping")
        task = dict(raw_task)
        for field in required_strings:
            if not isinstance(task.get(field), str) or not task[field].strip():
                raise ValueError(f"task {index} requires a nonblank string {field}")

View on GitHub (pinned to d540b00184)

Solutions

1. Pin the concrete versioned model id, e.g. 'claude-sonnet-4-5-20250929' instead of 'claude-latest'.
2. Resolve the alias to a versioned id once (via the provider's API) and store that id in the benchmark config.
3. Add a preflight check that rejects 'auto'/'latest' in your own config loader.
4. Treat 'latest' in a benchmark config as a bug, not a convenience.

Example fix

// before
normalized_model_identifier('claude-sonnet-latest')  # -> raises

// after
normalized_model_identifier('claude-sonnet-4-5-20250929')

import re

def is_pinned_model(value: str) -> bool:
    return isinstance(value, str) and not re.search(r'(?:^|[-/@:])(?:auto|latest)$', value.casefold())

import re

def is_pinned_model(value: str) -> bool:
    return isinstance(value, str) and bool(value.strip()) and not re.search(r'(?:^|[-/@:])(?:auto|latest)$', value.casefold())

try:
    model = normalized_model_identifier(value)
except ValueError as e:
    if 'mutable auto/latest' in str(e):
        # resolve the alias to a concrete versioned id and retry
        raise
    raise

When it happens

Trigger: re.search(r'(?:^|[-/@:])(?:auto|latest)$', model.casefold()) matches. E.g. 'claude-latest', 'gpt/auto', 'provider/model:latest', 'auto', 'anthropic@latest'.

Common situations: A user-friendly alias was copied from a docs page ('use claude-latest'); a CI config pins 'auto' for convenience; a model router default leaked in; the alias resolved fine in dev but drifts in CI over time.

Related errors

• {flag} must name a nonblank, versioned model
• {label} is unavailable: {path}: {exc}
• {label} must be a real non-symlink directory: {path}
• {label} contains an unsafe path component: {relative}
• {label} is unreadable: {path}: {exc}

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/fde258ed92592022. Report an issue: GitHub.