{"record":{"id":"9a9cc6e7b0cfbcf4","repo":"abi/screenshot-to-code","slug":"eval-set-not-found-set-name","errorCode":null,"errorMessage":"Eval set not found: {set_name}","messagePattern":"Eval set not found: (.+?)","errorType":"exception","errorClass":"EvalSetNotFoundError","httpStatus":null,"severity":"error","filePath":"backend/evals/sets.py","lineNumber":175,"sourceCode":"    except OSError as exc:\n        # The manifest is a cache/metadata sidecar; failing to persist it\n        # must not fail set operations.\n        print(f\"[EVAL SETS] Failed to write manifest for {set_name}: {exc}\")\n\n\ndef _sha256_of_file(path: str) -> str:\n    digest = hashlib.sha256()\n    with open(path, \"rb\") as f:\n        for chunk in iter(lambda: f.read(1 << 20), b\"\"):\n            digest.update(chunk)\n    return digest.hexdigest()\n\n\ndef list_set_images(set_name: str) -> list[EvalSetImage]:\n    \"\"\"PNGs of the set, hashes served from the manifest cache when fresh.\"\"\"\n    inputs_dir = get_set_inputs_dir(set_name)\n    if not os.path.isdir(inputs_dir):\n        raise EvalSetNotFoundError(f\"Eval set not found: {set_name}\")\n\n    manifest = _load_manifest(set_name)\n    raw_images = manifest.get(\"images\")\n    cached_images: dict[str, Any] = (\n        cast(dict[str, Any], raw_images) if isinstance(raw_images, dict) else {}\n    )\n    manifest_changed = not manifest\n\n    filenames = sorted(\n        entry\n        for entry in os.listdir(inputs_dir)\n        if entry.lower().endswith(\".png\")\n        and os.path.isfile(os.path.join(inputs_dir, entry))\n    )\n\n    images: list[EvalSetImage] = []\n    fresh_entries: dict[str, Any] = {}\n    for filename in filenames:","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/abi/screenshot-to-code/blob/d026163f586dfa8c5c10d28c36edd59a9d3b0e88/backend/evals/sets.py#L157-L193","documentation":"list_set_images raises EvalSetNotFoundError when the sets/{name}/inputs directory does not exist. Because sets are created only by dropping PNGs into that folder (no creation API), a missing inputs dir effectively means the set name is unknown. The name is also validated against a strict pattern first, so traversal-style names fail earlier with InvalidSetNameError.","triggerScenarios":"Calling list_set_images(name) where {EVALS_DIR}/sets/{name}/inputs is absent: set never created, set is a text set (briefs.json only, no inputs dir), typo in the name, or EVALS_DIR env var points somewhere else than where the set lives.","commonSituations":"Requesting an image set by the name of a text/briefs set; EVALS_DIR configured differently between processes; renaming a folder on disk while the UI still lists the old name; trailing whitespace or different casing in the requested name.","solutions":["Create the set by placing PNGs in {EVALS_DIR}/sets/{name}/inputs/.","Check get_set_kind(name): if it returns \"text\", use list_set_briefs instead of list_set_images.","Verify EVALS_DIR matches the directory that actually contains the set.","List available sets via list_sets() and compare the exact name spelling."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from evals.sets import get_set_inputs_dir, list_sets\nimport os\n\ndef set_has_images(set_name: str) -> bool:\n    return os.path.isdir(get_set_inputs_dir(set_name)) and any(\n        f.lower().endswith(\".png\") for f in os.listdir(get_set_inputs_dir(set_name))\n    )","typeGuard":null,"tryCatchPattern":"try:\n    images = list_set_images(set_name)\nexcept EvalSetNotFoundError:\n    return JSONResponse({\"error\": \"set not found\"}, status_code=404)","preventionTips":["Check get_set_kind(name) first: text sets have no inputs dir","Derive set names in UI code from list_sets(), never hand-type them","Keep EVALS_DIR consistent across all processes that touch evals"],"tags":["evals","filesystem","not-found"],"backgroundTag":null,"analyzedSha":"d026163f586dfa8c5c10d28c36edd59a9d3b0e88","analyzedAt":"2026-08-14T22:02:06.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}