rohitg00/ai-engineering-from-scratch · error · ValueError

adapter.model_id values must be unique within a single eval

Error message

adapter.model_id values must be unique within a single eval run

What it means

Error "adapter.model_id values must be unique within a single eval run" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/75-end-to-end-eval-runner/code/main.py:147

        correct=bool(correct),
        confidence=float(min(1.0, max(0.0, gen.confidence))),
        generation=gen.text,
        detail=detail,
        wall_seconds=float(elapsed),
    )


def run_eval(
    adapters: Sequence[ModelAdapter],
    tasks: Sequence[TaskSpec],
    parallel: bool = True,
    max_workers: int = 8,
    code_exec_timeout_s: float = 3.0,
) -> tuple[list[TaskResult], dict[str, list[tuple[float, float, int]]]]:
    results: list[TaskResult] = []
    model_ids = [a.model_id for a in adapters]
    if len(set(model_ids)) != len(model_ids):
        raise ValueError("adapter.model_id values must be unique within a single eval run")
    calibration_buf: dict[str, list[tuple[float, float, int]]] = {mid: [] for mid in model_ids}

    if not tasks or not adapters:
        return results, calibration_buf

    work: list[tuple[ModelAdapter, TaskSpec]] = []
    for adapter in adapters:
        for task in tasks:
            work.append((adapter, task))

    if parallel and len(work) > 1:
        workers = min(max_workers, len(work))
        with ThreadPoolExecutor(max_workers=workers) as ex:
            futures = {
                ex.submit(_score_one, adapter, task, code_exec_timeout_s): (adapter, task)
                for adapter, task in work
            }
            for fut in as_completed(futures):

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/75-end-to-end-eval-runner/code/main.py:147 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/0782a0e669d5fd45. Report an issue: GitHub.