abi/screenshot-to-code · error · HTTPException

Invalid eval set name: {set_name!r}

Error message

Invalid eval set name: {set_name!r}

What it means

Surfaces from evals.sets._validate_set_name() via GET /eval-sets/{set_name} as a 400. Set names must match ^[A-Za-z0-9][A-Za-z0-9._ -]*$ (first char alphanumeric, then alphanumerics, dot, underscore, space, hyphen) because the name is joined into filesystem paths — the pattern blocks traversal. The same exception type also carries messages about invalid brief entries, but this route's variant is the name check.

Source

Thrown at backend/routes/eval_sets.py:153

@router.get("/eval-sets/{set_name}", response_model=EvalSetDetailModel)
async def get_eval_set(set_name: str) -> EvalSetDetailModel:
    try:
        info = eval_sets.get_set(set_name)
        if info.kind == "text":
            briefs = eval_sets.list_set_briefs(set_name)
            return EvalSetDetailModel(
                **_set_info_to_model(info).model_dump(),
                images=[],
                briefs=[
                    EvalSetBriefModel(
                        id=b.id, title=b.title, brief=b.brief, tests=b.tests
                    )
                    for b in briefs
                ],
            )
        images = eval_sets.list_set_images(set_name)
    except eval_sets.InvalidSetNameError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except eval_sets.EvalSetNotFoundError:
        raise HTTPException(status_code=404, detail=f"Eval set not found: {set_name}")
    return EvalSetDetailModel(
        **_set_info_to_model(info).model_dump(),
        images=[
            EvalSetImageModel(
                filename=image.filename,
                sha256=image.sha256,
                size_bytes=image.size_bytes,
                tags=image.tags,
            )
            for image in images
        ],
    )


@router.get("/eval-sets/{set_name}/images/{filename}")
async def get_eval_set_image(set_name: str, filename: str) -> FileResponse:

View on GitHub (pinned to d026163f58)

Solutions

1. Use a name matching ^[A-Za-z0-9][A-Za-z0-9._ -]*$ such as 'jun-21-evals' or 'Landing Pages v2'.
2. List valid names first via GET /eval-sets and pick from the response.
3. URL-encode the path segment properly (spaces allowed but must be encoded in URLs).
4. Never send leading dots/hyphens or path separators.

Example fix

# before
GET /eval-sets/../other-dir  # 400 Invalid eval set name

# after
import re
assert re.match(r"^[A-Za-z0-9][A-Za-z0-9._ -]*$", set_name), "bad set name"
GET /eval-sets/Landing%20Pages%20v2

import re
SET_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._ -]*$")

def is_valid_set_name(name: str) -> bool:
    return bool(SET_NAME_RE.match(name))

if not is_valid_set_name(set_name):
    raise ValueError(f"invalid eval set name: {set_name!r}")

def is_valid_set_name(name: object) -> bool:
    """First char alphanumeric; then alnum . _ space - only."""
    return (
        isinstance(name, str)
        and bool(re.match(r"^[A-Za-z0-9][A-Za-z0-9._ -]*$", name))
    )

resp = requests.get(url + f"/eval-sets/{set_name}")
if resp.status_code == 400:
    raise ValueError(f"set name rejected: {set_name!r}")
resp.raise_for_status()

When it happens

Trigger: GET /eval-sets/{set_name} where set_name starts with '.', '-', or '/', contains slashes or '..', or is empty — e.g. '../secrets', '-evals', '.hidden'.

Common situations: URL-encoding issues turning spaces into %20 mishandled; names taken from directory listings of non-set dirs; attempted traversal probes; names with slashes from nested paths.

Related errors

• Invalid run id
• Not a set image: {filename!r}
• Invalid eval set name: {set_name!r}
• Eval set {request.set_name!r} has no images
• Invalid report filename

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/1960c74dff2feb5d. Report an issue: GitHub.