rohitg00/ai-engineering-from-scratch · error · ValueError

every evaluation case requires an id

Error message

every evaluation case requires an id

What it means

Error "every evaluation case requires an id" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at certifications/claude/lessons/30-developer-application-capstone/code/main.py:274

    if not isinstance(cases, list) or not cases or not isinstance(release_gates, dict):
        raise ValueError("evaluation plan requires cases and releaseGates")

    max_tool_calls = release_gates.get("maxToolCallsPerCase")
    if not isinstance(max_tool_calls, int) or isinstance(max_tool_calls, bool) or max_tool_calls < 0:
        raise ValueError("maxToolCallsPerCase must be a non-negative integer")
    global_forbidden = release_gates.get("forbiddenEffects", [])
    if not isinstance(global_forbidden, list):
        raise ValueError("releaseGates.forbiddenEffects must be a list")
    required_trace_fields = active_plan.get("requiredTraceFields", [])
    if not isinstance(required_trace_fields, list):
        raise ValueError("requiredTraceFields must be a list")

    create_agent = agent_factory or _default_agent_factory
    results: list[dict[str, Any]] = []
    safe_cases = 0
    for case in cases:
        if not isinstance(case, dict) or not isinstance(case.get("id"), str):
            raise ValueError("every evaluation case requires an id")
        fixture = case.get("fixture", {})
        expected_tools = case.get("expectedTools", [])
        case_forbidden = case.get("forbiddenEffects", [])
        if not isinstance(fixture, dict) or not isinstance(expected_tools, list) or not isinstance(case_forbidden, list):
            raise ValueError(f"evaluation case {case['id']} has invalid fixture or constraints")

        result = create_agent(dict(fixture))(str(case.get("input", "")))
        contract = result.contract()
        trace = [asdict(event) for event in result.trace]
        actual_tools = _trace_tools(result.trace)
        actual_effects = _trace_effects(result.trace)
        forbidden = set(str(effect) for effect in global_forbidden + case_forbidden)
        forbidden_observed = sorted(actual_effects & forbidden)

        failures = validate_contract(contract)
        if result.status != case.get("expectedStatus"):
            failures.append(f"expected status {case.get('expectedStatus')}")
        if result.escalated is not case.get("expectedEscalated"):

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at certifications/claude/lessons/30-developer-application-capstone/code/main.py:274 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/87bbfca8354d008b. Report an issue: GitHub.