abi/screenshot-to-code · warning · HTTPException

Eval set not found: {request.set_name}

Error message

Eval set not found: {request.set_name}

What it means

404 raised inside _resolve_set_run in backend/routes/evals.py:128 when request.set_name passes name validation but eval_sets.get_set() raises EvalSetNotFoundError — the named set does not exist in the store (no such set directory/metadata on disk).

Source

Thrown at backend/routes/evals.py:128

    files: List[str] = []  # Optional list of specific file paths to run evals on
    diff_mode: bool = False
    # When set, inputs come from {EVALS_DIR}/sets/{set_name}/inputs and runs
    # attach to the active eval session (auto-created when none exists).
    set_name: Optional[str] = None


def _resolve_set_run(
    request: RunEvalsRequest,
) -> tuple[Optional[str], Optional[eval_sessions.EvalSession], Dict[str, set[str]]]:
    """Validate the requested set and resolve the session + per-model skips."""
    if not request.set_name:
        return None, None, {}
    try:
        set_info = eval_sets.get_set(request.set_name)
    except eval_sets.InvalidSetNameError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except eval_sets.EvalSetNotFoundError:
        raise HTTPException(
            status_code=404, detail=f"Eval set not found: {request.set_name}"
        )
    if set_info.image_count == 0:
        raise HTTPException(
            status_code=400, detail=f"Eval set {request.set_name!r} has no images"
        )
    try:
        session = eval_sessions.resolve_session_for_run(request.set_name)
    except eval_sessions.SessionSetMismatchError as e:
        raise HTTPException(
            status_code=400,
            detail=(
                f"Active session {e.active_session.name!r} is pinned to set "
                f"{e.active_session.eval_set!r}. Start a new session for set "
                f"{request.set_name!r} first (POST /eval-sessions)."
            ),
        )
    skip_per_model: Dict[str, set[str]] = {}

View on GitHub (pinned to d026163f58)

Solutions

1. List existing sets via the eval-sets listing endpoint and pick the exact name.
2. Create the set (and upload its images) before referencing it in a run.
3. If the set should exist, check the backend's EVALS_DIR/sets location and working directory.

const sets = await (await fetch('/eval-sets')).json();
if (!sets.some(s => s.name === setName)) throw new Error(`Unknown set: ${setName}`);

if (res.status === 404) { /* re-list sets; offer to create it */ }

When it happens

Trigger: POST /run_evals_stream with set_name of a set that was never created, was deleted, or is misspelled (but well-formed).

Common situations: Set was pruned between selection and run; different backend working directory so the sets root resolves elsewhere; typo in the set name.

Related errors

• Eval set not found: {set_name}
• Image not found
• Eval set not found: {request.eval_set}
• No runs recorded
• Run not found

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/ee28474b3db7b2cd. Report an issue: GitHub.