{"record":{"id":"2487bb3eb994116a","repo":"langchain-ai/deepagents","slug":"graderresponse-result-needs-revision-but-every","errorCode":null,"errorMessage":"GraderResponse: result='needs_revision' but every criterion has passed=True.","messagePattern":"GraderResponse: result='needs_revision' but every criterion has passed=True\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"libs/deepagents/deepagents/middleware/rubric.py","lineNumber":348,"sourceCode":"    )\n\n    @model_validator(mode=\"after\")\n    def _check_result_consistency(self) -> GraderResponse:\n        \"\"\"Reject grader output where `result` contradicts the per-criterion verdicts.\n\n        The grader is an LLM and can hallucinate self-inconsistent\n        responses (e.g. claiming `satisfied` while flagging a failing\n        criterion). The discriminated union on `CriterionEval` enforces\n        the per-criterion `gap` invariant; this validator catches the\n        cross-field one.\n        \"\"\"\n        has_fail = any(not c[\"passed\"] for c in self.criteria)\n        if self.result == \"satisfied\" and has_fail:\n            msg = \"GraderResponse: result='satisfied' but at least one criterion has passed=False.\"\n            raise ValueError(msg)\n        if self.result == \"needs_revision\" and self.criteria and not has_fail:\n            msg = \"GraderResponse: result='needs_revision' but every criterion has passed=True.\"\n            raise ValueError(msg)\n        return self\n\n\n_StructuredOutputStrategy = Literal[\"ProviderStrategy\", \"ToolStrategy\"]\n\"\"\"Structured-output strategies LangChain can select for the grader.\"\"\"\n\n\ndef _model_identifier(model: object) -> str | None:\n    \"\"\"Return the model identifier exposed by supported chat integrations.\n\n    LangChain integrations do not share one identifier attribute: common\n    implementations expose `model_name`, `model`, or `model_id`. Checking them\n    in LangChain's precedence order keeps diagnostic labels and strategy\n    inference consistent.\n    \"\"\"\n    for attr in (\"model_name\", \"model\", \"model_id\"):\n        value = getattr(model, attr, None)\n        if isinstance(value, str) and value:","sourceCodeStart":330,"sourceCodeEnd":366,"githubUrl":"https://github.com/langchain-ai/deepagents/blob/a1af029e6e73cb17c36bff823d227747b28e91e1/libs/deepagents/deepagents/middleware/rubric.py#L330-L366","documentation":"The same _check_result_consistency validator also rejects the opposite inconsistency: result='needs_revision' with non-empty criteria where every criterion has passed=True. If revision is needed, at least one criterion must have failed.","triggerScenarios":"Building GraderResponse with result=\"needs_revision\" and criteria=[{...\"passed\": True...}] for all criteria (or criteria all defaulting to passed=True).","commonSituations":"Grader LLM emits needs_revision without marking any criterion failed; mocked/fixture responses written carelessly; empty explanatory text mapped to passed=True while overall result stayed needs_revision.","solutions":["Set result=\"satisfied\" when all criteria passed","Or mark at least one criterion passed=False if revision truly is needed","Ensure an empty criteria list doesn't accompany needs_revision (that combination is allowed but suspicious) — align criteria with the verdict","Improve grader prompt so verdict and per-criterion flags stay consistent"],"exampleFix":"// before\nGraderResponse(result=\"needs_revision\", criteria=[{\"passed\": True}, {\"passed\": True}])\n// after\nGraderResponse(result=\"satisfied\", criteria=[{\"passed\": True}, {\"passed\": True}])","handlingStrategy":"validation","validationCode":"def is_consistent_grader_response(data: dict) -> bool:\n    if data.get(\"result\") == \"needs_revision\" and data.get(\"criteria\"):\n        return any(not c.get(\"passed\", True) for c in data[\"criteria\"])\n    return True","typeGuard":"def grader_response_consistent(resp) -> bool:\n    has_fail = any(not c[\"passed\"] for c in resp.criteria)\n    return not (resp.result == \"needs_revision\" and resp.criteria and not has_fail)","tryCatchPattern":"try:\n    resp = GraderResponse.model_validate(llm_output)\nexcept ValueError as e:\n    logger.warning(\"Grader verdict/criteria mismatch: %s — re-grading\", e)\n    resp = regrade_with_strict_prompt()","preventionTips":["Derive result programmatically from criteria rather than asking the LLM for both","Validate fixtures/mocks against the model in tests","Retry grading on validation failure and log raw output for debugging"],"tags":["python","validation","pydantic","rubric","llm"],"backgroundTag":"inconsistent-grader-result","analyzedSha":"a1af029e6e73cb17c36bff823d227747b28e91e1","analyzedAt":"2026-08-29T11:43:24.718Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}