rohitg00/ai-engineering-from-scratch · error · ValueError

evaluation case has invalid fixture or constraints

Error message

evaluation case {case['id']} has invalid fixture or constraints

What it means

Raised in evaluate when a case's fixture or constraint fields fail the case validator (invalid_problem_report) for that case id. It is a per-case wrapper: the message names the offending case, and the underlying issues come from the fixture/constraint checks applied to its request and constraints members.

Solutions

1. Inspect the named case in the eval plan and fix its fixture fields (request/constraints shape)
2. Align the case's constraints with the supported constraint schema
3. Validate each case at authoring time so evaluate never sees a broken fixture

When it happens

Trigger: Thrown at certifications/claude/lessons/30-developer-application-capstone/code/main.py:279 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/f769522d79cd23b5. Report an issue: GitHub.

Appendix: source

Thrown at certifications/claude/lessons/30-developer-application-capstone/code/main.py:279

        raise ValueError("maxToolCallsPerCase must be a non-negative integer")
    global_forbidden = release_gates.get("forbiddenEffects", [])
    if not isinstance(global_forbidden, list):
        raise ValueError("releaseGates.forbiddenEffects must be a list")
    required_trace_fields = active_plan.get("requiredTraceFields", [])
    if not isinstance(required_trace_fields, list):
        raise ValueError("requiredTraceFields must be a list")

    create_agent = agent_factory or _default_agent_factory
    results: list[dict[str, Any]] = []
    safe_cases = 0
    for case in cases:
        if not isinstance(case, dict) or not isinstance(case.get("id"), str):
            raise ValueError("every evaluation case requires an id")
        fixture = case.get("fixture", {})
        expected_tools = case.get("expectedTools", [])
        case_forbidden = case.get("forbiddenEffects", [])
        if not isinstance(fixture, dict) or not isinstance(expected_tools, list) or not isinstance(case_forbidden, list):
            raise ValueError(f"evaluation case {case['id']} has invalid fixture or constraints")

        result = create_agent(dict(fixture))(str(case.get("input", "")))
        contract = result.contract()
        trace = [asdict(event) for event in result.trace]
        actual_tools = _trace_tools(result.trace)
        actual_effects = _trace_effects(result.trace)
        forbidden = set(str(effect) for effect in global_forbidden + case_forbidden)
        forbidden_observed = sorted(actual_effects & forbidden)

        failures = validate_contract(contract)
        if result.status != case.get("expectedStatus"):
            failures.append(f"expected status {case.get('expectedStatus')}")
        if result.escalated is not case.get("expectedEscalated"):
            failures.append(f"expected escalated {case.get('expectedEscalated')}")
        if actual_tools != expected_tools:
            failures.append(f"expected tool trajectory {expected_tools}, got {actual_tools}")
        if len(actual_tools) > max_tool_calls:
            failures.append(f"tool call limit exceeded: {len(actual_tools)} > {max_tool_calls}")

View on GitHub (pinned to 39ea8a1c6d)