abi/screenshot-to-code · error · InvalidSetNameError

Invalid brief entry in

Error message

Invalid brief entry in {set_name}: id={brief_id!r}

What it means

Raised while parsing briefs.json for a text eval set: an entry's `id` fails the pattern ^[a-z0-9][a-z0-9-]*$ or its `brief` field is empty. The loader (list_set_briefs) iterates `briefs` entries and hard-fails on the first invalid one, refusing to serve a partially malformed set. It surfaces as InvalidSetNameError even though the real problem is entry content, not the set name.

Solutions

1. Fix the offending entry id to lowercase alphanumerics and hyphens only, starting with a-z or 0-9 (e.g. "landing-page-1").
2. Add or fill in a non-empty `brief` string for the entry named in the message.
3. Validate briefs.json with the same regex before dropping it into sets/{name}/.
4. If the entry is junk, remove it entirely (non-dict entries are ignored, but a dict with bad fields is fatal).

Example fix

// before (briefs.json)
{"briefs": [{"id": "Brief_1", "title": "Landing", "brief": "..."}]}

// after
{"briefs": [{"id": "brief-1", "title": "Landing", "brief": "..."}]}

import json, re

BRIEF_ID = re.compile(r"^[a-z0-9][a-z0-9-]*$")

def validate_briefs_file(path: str) -> list[str]:
    problems = []
    data = json.loads(open(path, encoding="utf-8").read())
    for i, entry in enumerate(data.get("briefs", [])):
        if not isinstance(entry, dict):
            continue
        if not BRIEF_ID.match(str(entry.get("id") or "")):
            problems.append(f"entry {i}: bad id {entry.get('id')!r}")
        if not str(entry.get("brief") or "").strip():
            problems.append(f"entry {i}: empty brief")
    return problems

try:
    briefs = list_set_briefs(set_name)
except InvalidSetNameError as e:
    # message names the offending id; fix briefs.json
    log.error("briefs.json invalid for %s: %s", set_name, e)
    return []

When it happens

Trigger: Calling list_set_briefs(set_name) when sets/{name}/briefs.json contains an entry with id like "My Brief", "brief_1" (underscore), leading hyphen, or an empty/missing id, or an entry whose `brief` value is ""/missing. Non-dict entries are skipped silently; only dict entries with bad ids/empty briefs trigger it.

Common situations: Hand-editing briefs.json with capitalized or underscore ids; copy-pasting ids from filenames; an entry that has a title but the author forgot the `brief` body; upstream tooling that generates ids with UUID format (dashes ok but letters uppercase).

Related errors

• Prompt file must contain a JSON array.
• Prompt item is missing a prompt string.
• Prompt item must be a string or object.
• Active session is pinned to set , not
• At least one model is required

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/f70e430f14bf5b76. Report an issue: GitHub.

Appendix: source

Thrown at backend/evals/sets.py:119

        raise EvalSetNotFoundError(f"Text eval set not found: {set_name}")
    try:
        with open(path, "r", encoding="utf-8") as f:
            loaded = cast(dict[str, Any], json.load(f))
    except (OSError, json.JSONDecodeError) as exc:
        raise EvalSetNotFoundError(f"Unreadable briefs.json for {set_name}: {exc}")
    briefs: list[EvalSetBrief] = []
    raw_briefs = loaded.get("briefs")
    entries = (
        cast(list[object], raw_briefs) if isinstance(raw_briefs, list) else []
    )
    for entry in entries:
        if not isinstance(entry, dict):
            continue
        record = cast(dict[str, Any], entry)
        brief_id = str(record.get("id") or "")
        brief = str(record.get("brief") or "")
        if not _BRIEF_ID_PATTERN.match(brief_id) or not brief:
            raise InvalidSetNameError(
                f"Invalid brief entry in {set_name}: id={brief_id!r}"
            )
        briefs.append(
            EvalSetBrief(
                id=brief_id,
                title=str(record.get("title") or brief_id),
                brief=brief,
                tests=str(record.get("tests") or ""),
            )
        )
    return briefs


def _load_manifest(set_name: str) -> dict[str, Any]:
    try:
        with open(_manifest_path(set_name), "r", encoding="utf-8") as f:
            loaded = cast(Any, json.load(f))
            if isinstance(loaded, dict):

View on GitHub (pinned to d026163f58)