abi/screenshot-to-code · error · EvalSetNotFoundError

Eval set not found: {set_name}

Error message

Eval set not found: {set_name}

What it means

list_set_images raises EvalSetNotFoundError when the sets/{name}/inputs directory does not exist. Because sets are created only by dropping PNGs into that folder (no creation API), a missing inputs dir effectively means the set name is unknown. The name is also validated against a strict pattern first, so traversal-style names fail earlier with InvalidSetNameError.

Source

Thrown at backend/evals/sets.py:175

    except OSError as exc:
        # The manifest is a cache/metadata sidecar; failing to persist it
        # must not fail set operations.
        print(f"[EVAL SETS] Failed to write manifest for {set_name}: {exc}")


def _sha256_of_file(path: str) -> str:
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            digest.update(chunk)
    return digest.hexdigest()


def list_set_images(set_name: str) -> list[EvalSetImage]:
    """PNGs of the set, hashes served from the manifest cache when fresh."""
    inputs_dir = get_set_inputs_dir(set_name)
    if not os.path.isdir(inputs_dir):
        raise EvalSetNotFoundError(f"Eval set not found: {set_name}")

    manifest = _load_manifest(set_name)
    raw_images = manifest.get("images")
    cached_images: dict[str, Any] = (
        cast(dict[str, Any], raw_images) if isinstance(raw_images, dict) else {}
    )
    manifest_changed = not manifest

    filenames = sorted(
        entry
        for entry in os.listdir(inputs_dir)
        if entry.lower().endswith(".png")
        and os.path.isfile(os.path.join(inputs_dir, entry))
    )

    images: list[EvalSetImage] = []
    fresh_entries: dict[str, Any] = {}
    for filename in filenames:

View on GitHub (pinned to d026163f58)

Solutions

1. Create the set by placing PNGs in {EVALS_DIR}/sets/{name}/inputs/.
2. Check get_set_kind(name): if it returns "text", use list_set_briefs instead of list_set_images.
3. Verify EVALS_DIR matches the directory that actually contains the set.
4. List available sets via list_sets() and compare the exact name spelling.

from evals.sets import get_set_inputs_dir, list_sets
import os

def set_has_images(set_name: str) -> bool:
    return os.path.isdir(get_set_inputs_dir(set_name)) and any(
        f.lower().endswith(".png") for f in os.listdir(get_set_inputs_dir(set_name))
    )

try:
    images = list_set_images(set_name)
except EvalSetNotFoundError:
    return JSONResponse({"error": "set not found"}, status_code=404)

When it happens

Trigger: Calling list_set_images(name) where {EVALS_DIR}/sets/{name}/inputs is absent: set never created, set is a text set (briefs.json only, no inputs dir), typo in the name, or EVALS_DIR env var points somewhere else than where the set lives.

Common situations: Requesting an image set by the name of a text/briefs set; EVALS_DIR configured differently between processes; renaming a folder on disk while the UI still lists the old name; trailing whitespace or different casing in the requested name.

Related errors

• Text eval set not found: {set_name}
• Unreadable briefs.json for {set_name}: {exc}
• Error reading input files: {str(e)}
• Folder not found: {folder}
• Error processing evals: {str(e)}

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/9a9cc6e7b0cfbcf4. Report an issue: GitHub.