abi/screenshot-to-code · error · InvalidSetNameError

Invalid eval set name: {set_name!r}

Error message

Invalid eval set name: {set_name!r}

What it means

InvalidSetNameError raised by _validate_set_name when the set name fails ^[A-Za-z0-9][A-Za-z0-9._ -]*$ — it must start with an alphanumeric and may then contain letters, digits, dot, underscore, space, and hyphen. The validator fronts _get_set_dir, so any path-building operation on a set (inputs dir, manifest) is protected from path traversal and odd filesystem characters.

Source

Thrown at backend/evals/sets.py:68

@dataclass
class EvalSetInfo:
    name: str
    display_name: str
    created_at: Optional[str]
    notes: str
    image_count: int
    kind: str = "image"  # "image" | "text"


def get_sets_dir() -> str:
    return os.path.join(evals_config.EVALS_DIR, "sets")


def _validate_set_name(set_name: str) -> str:
    if not _SET_NAME_PATTERN.match(set_name):
        raise InvalidSetNameError(f"Invalid eval set name: {set_name!r}")
    return set_name


def _get_set_dir(set_name: str) -> str:
    return os.path.join(get_sets_dir(), _validate_set_name(set_name))


def get_set_inputs_dir(set_name: str) -> str:
    return os.path.join(_get_set_dir(set_name), "inputs")


def _manifest_path(set_name: str) -> str:
    return os.path.join(_get_set_dir(set_name), "manifest.json")


def _briefs_path(set_name: str) -> str:
    return os.path.join(_get_set_dir(set_name), "briefs.json")

View on GitHub (pinned to d026163f58)

Solutions

1. Rename the set to match the pattern: start with a letter/digit, use only [A-Za-z0-9._ -].
2. Sanitize user input before calling set APIs: strip, replace disallowed characters with '-' or '_'.
3. Validate client-side in the eval sessions form so the user gets immediate feedback.

Example fix

// before
const name = userInput; // "my set#2"

// after
const name = userInput.trim().replace(/[^A-Za-z0-9._ -]/g, "-").replace(/^[. _-]+/, "") || "untitled";

import re
SET_NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._ -]*$")
def valid_set_name(name: str) -> bool:
    return isinstance(name, str) and bool(SET_NAME.match(name))

function isValidSetName(name: unknown): name is string {
  return typeof name === "string" && /^[A-Za-z0-9][A-Za-z0-9._ -]*$/.test(name);
}

try:
    _get_set_dir(set_name)
except InvalidSetNameError:
    return HTTPException(status_code=422, detail="Set name must start with a letter/digit")

When it happens

Trigger: Creating or reading a set whose name starts with '.', '/', '-', or a space, contains slashes or '..', or uses non-ASCII characters — e.g. POSTing a name like '../secrets' or 'my set#2'.

Common situations: UI allowing free-text set names, API consumers passing URL-decoded paths, or attempts at path traversal (the pattern blocks them).

Related errors

• No stack was provided
• No model was provided
• Active session {active_session.name!r} is pinned to set {act
• Invalid brief entry in {set_name}: id={brief_id!r}
• Not a set image: {filename!r}

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/f2b57cca1788be62. Report an issue: GitHub.