abi/screenshot-to-code · warning · HTTPException

Invalid eval set name: {set_name!r}

Error message

Invalid eval set name: {set_name!r}

What it means

400 raised inside _resolve_set_run in backend/routes/evals.py:126 when request.set_name fails eval_sets.get_set() with InvalidSetNameError. That error comes from the eval-set store when the name does not match the allowed naming scheme, so the set name is malformed before any on-disk lookup happens.

Source

Thrown at backend/routes/evals.py:126

    models: List[str]
    stack: Stack
    files: List[str] = []  # Optional list of specific file paths to run evals on
    diff_mode: bool = False
    # When set, inputs come from {EVALS_DIR}/sets/{set_name}/inputs and runs
    # attach to the active eval session (auto-created when none exists).
    set_name: Optional[str] = None


def _resolve_set_run(
    request: RunEvalsRequest,
) -> tuple[Optional[str], Optional[eval_sessions.EvalSession], Dict[str, set[str]]]:
    """Validate the requested set and resolve the session + per-model skips."""
    if not request.set_name:
        return None, None, {}
    try:
        set_info = eval_sets.get_set(request.set_name)
    except eval_sets.InvalidSetNameError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except eval_sets.EvalSetNotFoundError:
        raise HTTPException(
            status_code=404, detail=f"Eval set not found: {request.set_name}"
        )
    if set_info.image_count == 0:
        raise HTTPException(
            status_code=400, detail=f"Eval set {request.set_name!r} has no images"
        )
    try:
        session = eval_sessions.resolve_session_for_run(request.set_name)
    except eval_sessions.SessionSetMismatchError as e:
        raise HTTPException(
            status_code=400,
            detail=(
                f"Active session {e.active_session.name!r} is pinned to set "
                f"{e.active_session.eval_set!r}. Start a new session for set "
                f"{request.set_name!r} first (POST /eval-sessions)."
            ),

View on GitHub (pinned to d026163f58)

Solutions

1. Use a set name created via the eval-set management routes (POST /eval-sets) — those names are known-valid.
2. Sanitize the name to the allowed pattern (typically lowercase letters, digits, dashes/underscores) before sending.
3. Read str(e) in the response detail: it states the expected naming rule.

Example fix

// before
body: JSON.stringify({ set_name: rawUserInput, ... })

// after
const setName = rawUserInput.trim().toLowerCase().replace(/[^a-z0-9-_]/g, '-');
if (!/^[a-z0-9][a-z0-9-_]*$/.test(setName)) throw new Error('Invalid set name');
body: JSON.stringify({ set_name: setName, ... })

const SET_NAME_RE = /^[a-z0-9][a-z0-9-_]*$/;
if (!SET_NAME_RE.test(setName)) throw new Error(`Invalid set name: ${setName}`);

function isValidSetName(name: string): boolean {
  return /^[a-z0-9][a-z0-9-_]*$/.test(name);
}

const res = await fetch('/run_evals_stream', ...);
if (res.status === 400) { /* detail explains the naming rule; show it */ }

When it happens

Trigger: POST /run_evals_stream (or /run_evals) with set_name containing characters outside the allowed set-name pattern, e.g. '../escape', spaces, or empty-ish strings that still parse as invalid.

Common situations: Frontend passes a user-typed set name without sanitizing; scripts constructing set names from file paths; traversal-style names rejected by the store's validation regex.

Related errors

• Invalid eval set name: {set_name!r}
• Not a set image: {filename!r}
• Eval set {request.set_name!r} has no images
• Invalid run id
• max_age_days must be >= 1

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/37e3c42008f52e5d. Report an issue: GitHub.