{"record":{"id":"f769522d79cd23b5","repo":"rohitg00/ai-engineering-from-scratch","slug":"evaluation-case-case-id-has-invalid-fixture-o","errorCode":null,"errorMessage":"evaluation case {case['id']} has invalid fixture or constraints","messagePattern":"evaluation case (.+?) has invalid fixture or constraints","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"certifications/claude/lessons/30-developer-application-capstone/code/main.py","lineNumber":279,"sourceCode":"        raise ValueError(\"maxToolCallsPerCase must be a non-negative integer\")\n    global_forbidden = release_gates.get(\"forbiddenEffects\", [])\n    if not isinstance(global_forbidden, list):\n        raise ValueError(\"releaseGates.forbiddenEffects must be a list\")\n    required_trace_fields = active_plan.get(\"requiredTraceFields\", [])\n    if not isinstance(required_trace_fields, list):\n        raise ValueError(\"requiredTraceFields must be a list\")\n\n    create_agent = agent_factory or _default_agent_factory\n    results: list[dict[str, Any]] = []\n    safe_cases = 0\n    for case in cases:\n        if not isinstance(case, dict) or not isinstance(case.get(\"id\"), str):\n            raise ValueError(\"every evaluation case requires an id\")\n        fixture = case.get(\"fixture\", {})\n        expected_tools = case.get(\"expectedTools\", [])\n        case_forbidden = case.get(\"forbiddenEffects\", [])\n        if not isinstance(fixture, dict) or not isinstance(expected_tools, list) or not isinstance(case_forbidden, list):\n            raise ValueError(f\"evaluation case {case['id']} has invalid fixture or constraints\")\n\n        result = create_agent(dict(fixture))(str(case.get(\"input\", \"\")))\n        contract = result.contract()\n        trace = [asdict(event) for event in result.trace]\n        actual_tools = _trace_tools(result.trace)\n        actual_effects = _trace_effects(result.trace)\n        forbidden = set(str(effect) for effect in global_forbidden + case_forbidden)\n        forbidden_observed = sorted(actual_effects & forbidden)\n\n        failures = validate_contract(contract)\n        if result.status != case.get(\"expectedStatus\"):\n            failures.append(f\"expected status {case.get('expectedStatus')}\")\n        if result.escalated is not case.get(\"expectedEscalated\"):\n            failures.append(f\"expected escalated {case.get('expectedEscalated')}\")\n        if actual_tools != expected_tools:\n            failures.append(f\"expected tool trajectory {expected_tools}, got {actual_tools}\")\n        if len(actual_tools) > max_tool_calls:\n            failures.append(f\"tool call limit exceeded: {len(actual_tools)} > {max_tool_calls}\")","sourceCodeStart":261,"sourceCodeEnd":297,"githubUrl":"https://github.com/rohitg00/ai-engineering-from-scratch/blob/39ea8a1c6d0b61f071226eff7ede4d4105fed820/certifications/claude/lessons/30-developer-application-capstone/code/main.py#L261-L297","documentation":"Raised in evaluate when a case's fixture or constraint fields fail the case validator (invalid_problem_report) for that case id. It is a per-case wrapper: the message names the offending case, and the underlying issues come from the fixture/constraint checks applied to its request and constraints members.","triggerScenarios":"Thrown at certifications/claude/lessons/30-developer-application-capstone/code/main.py:279 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Inspect the named case in the eval plan and fix its fixture fields (request/constraints shape)","Align the case's constraints with the supported constraint schema","Validate each case at authoring time so evaluate never sees a broken fixture"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"39ea8a1c6d0b61f071226eff7ede4d4105fed820","analyzedAt":"2026-08-26T03:13:46.626Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}