abi/screenshot-to-code · error · HTTPException

Eval set not found: {request.eval_set}

Error message

Eval set not found: {request.eval_set}

What it means

Raised by POST /eval-sessions (404) when the eval_set name is well-formed but eval_sets.get_set() cannot find the set — same missing-set condition as GET /eval-sets/{name}, checked here so sessions always reference an existing set.

Source

Thrown at backend/routes/eval_sets.py:206

        sessions=[_session_to_model(s) for s in sessions],
        active_session_id=active.session_id if active else None,
    )


@router.get("/eval-sessions/active", response_model=Optional[EvalSessionModel])
async def get_active_eval_session() -> Optional[EvalSessionModel]:
    active = eval_sessions.get_active_session()
    return _session_to_model(active) if active else None


@router.post("/eval-sessions", response_model=EvalSessionModel)
async def create_eval_session(request: CreateEvalSessionRequest) -> EvalSessionModel:
    try:
        eval_sets.get_set(request.eval_set)
    except eval_sets.InvalidSetNameError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except eval_sets.EvalSetNotFoundError:
        raise HTTPException(
            status_code=404, detail=f"Eval set not found: {request.eval_set}"
        )
    session = eval_sessions.create_session(request.eval_set, request.name)
    return _session_to_model(session)


@router.post(
    "/eval-sessions/{session_id}/activate", response_model=EvalSessionModel
)
async def activate_eval_session(session_id: str) -> EvalSessionModel:
    if eval_sessions.get_session(session_id) is None:
        raise HTTPException(status_code=404, detail="Session not found")
    session = eval_sessions.activate_session(session_id)
    assert session is not None
    return _session_to_model(session)


def _is_stale_running(status: str, created_at: str) -> bool:

View on GitHub (pinned to d026163f58)

Solutions

1. GET /eval-sets first and POST with a name from that list.
2. Create the set before creating a session for it.
3. Verify EVALS_DIR configuration if the set should exist.

Example fix

# before
requests.post(url + "/eval-sessions", json={"eval_set": "jun-21-evals"})  # 404

# after
names = [s["name"] for s in requests.get(url + "/eval-sets").json()]
assert "jun-21-evals" in names, "create the set first"
requests.post(url + "/eval-sessions", json={"eval_set": "jun-21-evals"})

names = [s["name"] for s in requests.get(url + "/eval-sets").json()]
if payload["eval_set"] not in names:
    raise LookupError(f"eval set {payload['eval_set']!r} missing; available: {names}")
requests.post(url + "/eval-sessions", json=payload)

def eval_set_exists(name: str, listing: list[dict]) -> bool:
    return any(s.get("name") == name for s in listing)

resp = requests.post(url + "/eval-sessions", json=payload)
if resp.status_code == 404:
    raise LookupError("create the eval set before starting a session for it")
resp.raise_for_status()

When it happens

Trigger: POST /eval-sessions with a valid-format name that has no set directory: typo, deleted set, or a different evals data dir on this machine.

Common situations: Environment drift between machines (set exists locally, not on server); set renamed after the client cached the name; race with set deletion.

Related errors

• Eval set not found: {set_name}
• Session not found
• Run not found
• Run has no captured output
• Image not found

AI-assisted analysis of abi/screenshot-to-code@d026163f58 (2026-08-14). Data as JSON: /api/errors/a2136544581de8ed. Report an issue: GitHub.