abi/screenshot-to-code · error · InvalidSetNameError

Invalid brief entry in {set_name}: id={brief_id!r}

Error message

Invalid brief entry in {set_name}: id={brief_id!r}

What it means

Raised while parsing briefs.json for a text eval set: an entry's `id` fails the pattern ^[a-z0-9][a-z0-9-]*$ or its `brief` field is empty. The loader (list_set_briefs) iterates `briefs` entries and hard-fails on the first invalid one, refusing to serve a partially malformed set. It surfaces as InvalidSetNameError even though the real problem is entry content, not the set name.

Source

Thrown at backend/evals/sets.py:119

        raise EvalSetNotFoundError(f"Text eval set not found: {set_name}")
    try:
        with open(path, "r", encoding="utf-8") as f:
            loaded = cast(dict[str, Any], json.load(f))
    except (OSError, json.JSONDecodeError) as exc:
        raise EvalSetNotFoundError(f"Unreadable briefs.json for {set_name}: {exc}")
    briefs: list[EvalSetBrief] = []
    raw_briefs = loaded.get("briefs")
    entries = (
        cast(list[object], raw_briefs) if isinstance(raw_briefs, list) else []
    )
    for entry in entries:
        if not isinstance(entry, dict):
            continue
        record = cast(dict[str, Any], entry)
        brief_id = str(record.get("id") or "")
        brief = str(record.get("brief") or "")
        if not _BRIEF_ID_PATTERN.match(brief_id) or not brief:
            raise InvalidSetNameError(
                f"Invalid brief entry in {set_name}: id={brief_id!r}"
            )
        briefs.append(
            EvalSetBrief(
                id=brief_id,
                title=str(record.get("title") or brief_id),
                brief=brief,
                tests=str(record.get("tests") or ""),
            )
        )
    return briefs


def _load_manifest(set_name: str) -> dict[str, Any]:
    try:
        with open(_manifest_path(set_name), "r", encoding="utf-8") as f:
            loaded = cast(Any, json.load(f))
            if isinstance(loaded, dict):

View on GitHub (pinned to d026163f58)

Solutions

1. Fix the offending entry id to lowercase alphanumerics and hyphens only, starting with a-z or 0-9 (e.g. "landing-page-1").
2. Add or fill in a non-empty `brief` string for the entry named in the message.
3. Validate briefs.json with the same regex before dropping it into sets/{name}/.
4. If the entry is junk, remove it entirely (non-dict entries are ignored, but a dict with bad fields is fatal).

Example fix

// before (briefs.json)
{"briefs": [{"id": "Brief_1", "title": "Landing", "brief": "..."}]}

// after
{"briefs": [{"id": "brief-1", "title": "Landing", "brief": "..."}]}

import json, re

BRIEF_ID = re.compile(r"^[a-z0-9][a-z0-9-]*$")

def validate_briefs_file(path: str) -> list[str]:
    problems = []
    data = json.loads(open(path, encoding="utf-8").read())
    for i, entry in enumerate(data.get("briefs", [])):
        if not isinstance(entry, dict):
            continue
        if not BRIEF_ID.match(str(entry.get("id") or "")):
            problems.append(f"entry {i}: bad id {entry.get('id')!r}")
        if not str(entry.get("brief") or "").strip():
            problems.append(f"entry {i}: empty brief")
    return problems

try:
    briefs = list_set_briefs(set_name)
except InvalidSetNameError as e:
    # message names the offending id; fix briefs.json
    log.error("briefs.json invalid for %s: %s", set_name, e)
    return []

When it happens

Trigger: Calling list_set_briefs(set_name) when sets/{name}/briefs.json contains an entry with id like "My Brief", "brief_1" (underscore), leading hyphen, or an empty/missing id, or an entry whose `brief` value is ""/missing. Non-dict entries are skipped silently; only dict entries with bad ids/empty briefs trigger it.

Common situations: Hand-editing briefs.json with capitalized or underscore ids; copy-pasting ids from filenames; an entry that has a title but the author forgot the `brief` body; upstream tooling that generates ids with UUID format (dashes ok but letters uppercase).

Related errors

• Prompt file must contain a JSON array.
• Prompt item {index} must be a string or object.
• Prompt item {index} is missing a prompt string.
• No stack was provided
• No model was provided

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/f70e430f14bf5b76. Report an issue: GitHub.