rohitg00/ai-engineering-from-scratch · error · RuntimeError

fixture validation failed: {errors}

Error message

fixture validation failed: {errors}

What it means

Error "fixture validation failed: {errors}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/75-end-to-end-eval-runner/code/main.py:345

            if task.post_process == "extract_letter":
                text = "Answer: B"
            if task.metric_name == "code_exec":
                text = "```python\ndef add(a, b):\n    return None\n```"
            confidence = 0.4
            nll_per_token = 3.0
        token_count = max(1, len(text.split()))
        nll = token_count * nll_per_token
        self.token_stats.append((nll, token_count))
        return Generation(text=text, confidence=confidence, token_nll=nll, token_count=token_count)


def _load_fixture_tasks() -> list:
    import tempfile
    out_dir = tempfile.mkdtemp(prefix="aie_l75_")
    good, _bad = spec_mod.load_fixtures(out_dir)
    tasks, errors = spec_mod.validate_file(good)
    if errors:
        raise RuntimeError(f"fixture validation failed: {errors}")
    return tasks


def demo() -> int:
    tasks = _load_fixture_tasks()
    adapters = [RuleBasedAdapter(), NoisyAdapter(seed=1), BiasedAdapter(good_category="arithmetic")]
    t0 = time.time()
    results, calibration_buf = run_eval(adapters, tasks, parallel=True, max_workers=6, code_exec_timeout_s=2.0)
    wall = time.time() - t0

    adapter_token_stats = {
        a.model_id: list(getattr(a, "token_stats", [])) for a in adapters
    }
    report = render_report(adapters, tasks, results, calibration_buf, adapter_token_stats, wall)

    print(render_markdown_block(report))
    print()
    print(f"summary: {json.dumps(report.summary, indent=2)}")

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/75-end-to-end-eval-runner/code/main.py:345 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/59cd552cdb409814. Report an issue: GitHub.