abi/screenshot-to-code · error · HTTPException

Eval set not found: {set_name}

Error message

Eval set not found: {set_name}

What it means

Raised by GET /eval-sets/{set_name} (404) when the set name is valid but no such set exists — the underlying evals module raises EvalSetNotFoundError because the set directory (or its briefs.json / inputs) is absent under the evals sets directory. Distinguish it from the 400 invalid-name error: 400 means bad format, 404 means well-formed but missing.

Source

Thrown at backend/routes/eval_sets.py:155

    try:
        info = eval_sets.get_set(set_name)
        if info.kind == "text":
            briefs = eval_sets.list_set_briefs(set_name)
            return EvalSetDetailModel(
                **_set_info_to_model(info).model_dump(),
                images=[],
                briefs=[
                    EvalSetBriefModel(
                        id=b.id, title=b.title, brief=b.brief, tests=b.tests
                    )
                    for b in briefs
                ],
            )
        images = eval_sets.list_set_images(set_name)
    except eval_sets.InvalidSetNameError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except eval_sets.EvalSetNotFoundError:
        raise HTTPException(status_code=404, detail=f"Eval set not found: {set_name}")
    return EvalSetDetailModel(
        **_set_info_to_model(info).model_dump(),
        images=[
            EvalSetImageModel(
                filename=image.filename,
                sha256=image.sha256,
                size_bytes=image.size_bytes,
                tags=image.tags,
            )
            for image in images
        ],
    )


@router.get("/eval-sets/{set_name}/images/{filename}")
async def get_eval_set_image(set_name: str, filename: str) -> FileResponse:
    try:
        path = eval_sets.resolve_set_image_path(set_name, filename)

View on GitHub (pinned to d026163f58)

Solutions

1. GET /eval-sets to list existing names and use one of them.
2. Verify the set directory exists under the configured EVALS_DIR/sets.
3. If the set should exist, check the evals data dir configuration and re-create the set.

Example fix

# before
GET /eval-sets/jun-21-evalss  # 404 (typo)

# after
names = [s["name"] for s in requests.get(url + "/eval-sets").json()]
GET /eval-sets/{names[0]}

names = [s["name"] for s in requests.get(url + "/eval-sets").json()]
if set_name not in names:
    raise LookupError(f"eval set {set_name!r} not in {names}")

def eval_set_exists(name: str, listing: list[dict]) -> bool:
    return any(s.get("name") == name for s in listing)

resp = requests.get(url + f"/eval-sets/{set_name}")
if resp.status_code == 404:
    listing = requests.get(url + "/eval-sets").json()
    raise LookupError(f"set gone; available: {[s['name'] for s in listing]}")
resp.raise_for_status()

When it happens

Trigger: GET /eval-sets/{set_name} for a set that was never created, was deleted, or whose directory was renamed/moved outside the tool; also when get_set_kind finds neither inputs/ nor briefs.json.

Common situations: Typo in a valid-format name ('jul-21-evals' vs 'jun-21-evals'); evals data dir not synced to this machine; set created in a different environment.

Related errors

• Eval set not found: {request.eval_set}
• Run not found
• Run has no captured output
• Image not found
• Session not found

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/1c5765877033a499. Report an issue: GitHub.