abi/screenshot-to-code · error · EvalSetNotFoundError

Text eval set not found: {set_name}

Error message

Text eval set not found: {set_name}

What it means

EvalSetNotFoundError raised by list_set_briefs when the set's briefs.json does not exist at the expected path. A set is 'text' kind only when briefs.json is present (get_set_kind probes the same file); requesting briefs for an image-kind set or a set that doesn't exist at all both land here.

Source

Thrown at backend/evals/sets.py:101

def _briefs_path(set_name: str) -> str:
    return os.path.join(_get_set_dir(set_name), "briefs.json")


_BRIEF_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]*$")


def get_set_kind(set_name: str) -> str:
    """"text" when the set is a briefs.json collection, else "image"."""
    if os.path.isfile(_briefs_path(set_name)):
        return "text"
    return "image"


def list_set_briefs(set_name: str) -> list[EvalSetBrief]:
    path = _briefs_path(set_name)
    if not os.path.isfile(path):
        raise EvalSetNotFoundError(f"Text eval set not found: {set_name}")
    try:
        with open(path, "r", encoding="utf-8") as f:
            loaded = cast(dict[str, Any], json.load(f))
    except (OSError, json.JSONDecodeError) as exc:
        raise EvalSetNotFoundError(f"Unreadable briefs.json for {set_name}: {exc}")
    briefs: list[EvalSetBrief] = []
    raw_briefs = loaded.get("briefs")
    entries = (
        cast(list[object], raw_briefs) if isinstance(raw_briefs, list) else []
    )
    for entry in entries:
        if not isinstance(entry, dict):
            continue
        record = cast(dict[str, Any], entry)
        brief_id = str(record.get("id") or "")
        brief = str(record.get("brief") or "")
        if not _BRIEF_ID_PATTERN.match(brief_id) or not brief:
            raise InvalidSetNameError(

View on GitHub (pinned to d026163f58)

Solutions

1. Check get_set_kind(set_name) == 'text' before listing briefs.
2. Verify the set name against list of existing sets before use.
3. If the set should be text-kind, create a valid briefs.json with a {"briefs": [...]} structure.

Example fix

# before
briefs = list_set_briefs(set_name)

# after
if get_set_kind(set_name) != "text":
    raise ValueError(f"Set {set_name!r} has no briefs (image set)")
briefs = list_set_briefs(set_name)

from evals.sets import get_set_kind
if get_set_kind(set_name) != "text":
    raise ValueError(f"{set_name!r} is not a text/briefs set")

def is_text_set(set_name: str) -> bool:
    return os.path.isfile(_briefs_path(set_name))

try:
    briefs = list_set_briefs(set_name)
except EvalSetNotFoundError:
    return HTTPException(status_code=404, detail=f"Eval set {set_name} not found")

When it happens

Trigger: Calling list_set_briefs on a set directory with only an inputs/ folder (image set), a misspelled/deleted set name, or a set created without writing briefs.json.

Common situations: UI letting users pick any set for a text-brief flow, set folder renamed manually on disk, or a race where the set was deleted between listing and reading.

Related errors

• Eval set not found: {set_name}
• Unreadable briefs.json for {set_name}: {exc}
• Error reading input files: {str(e)}
• Folder not found: {folder}
• Error processing evals: {str(e)}

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/39a8ed64714286d0. Report an issue: GitHub.