{"record":{"id":"2bd13c0f3fdf0be1","repo":"langchain-ai/deepagents","slug":"graderresponse-result-satisfied-but-at-least-on","errorCode":null,"errorMessage":"GraderResponse: result='satisfied' but at least one criterion has passed=False.","messagePattern":"GraderResponse: result='satisfied' but at least one criterion has passed=False\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"libs/deepagents/deepagents/middleware/rubric.py","lineNumber":345,"sourceCode":"            \"never omit criteria or collapse several into one. Each entry carries `passed` \"\n            \"True/False, plus a `gap` string when failing.\"\n        ),\n    )\n\n    @model_validator(mode=\"after\")\n    def _check_result_consistency(self) -> GraderResponse:\n        \"\"\"Reject grader output where `result` contradicts the per-criterion verdicts.\n\n        The grader is an LLM and can hallucinate self-inconsistent\n        responses (e.g. claiming `satisfied` while flagging a failing\n        criterion). The discriminated union on `CriterionEval` enforces\n        the per-criterion `gap` invariant; this validator catches the\n        cross-field one.\n        \"\"\"\n        has_fail = any(not c[\"passed\"] for c in self.criteria)\n        if self.result == \"satisfied\" and has_fail:\n            msg = \"GraderResponse: result='satisfied' but at least one criterion has passed=False.\"\n            raise ValueError(msg)\n        if self.result == \"needs_revision\" and self.criteria and not has_fail:\n            msg = \"GraderResponse: result='needs_revision' but every criterion has passed=True.\"\n            raise ValueError(msg)\n        return self\n\n\n_StructuredOutputStrategy = Literal[\"ProviderStrategy\", \"ToolStrategy\"]\n\"\"\"Structured-output strategies LangChain can select for the grader.\"\"\"\n\n\ndef _model_identifier(model: object) -> str | None:\n    \"\"\"Return the model identifier exposed by supported chat integrations.\n\n    LangChain integrations do not share one identifier attribute: common\n    implementations expose `model_name`, `model`, or `model_id`. Checking them\n    in LangChain's precedence order keeps diagnostic labels and strategy\n    inference consistent.\n    \"\"\"","sourceCodeStart":327,"sourceCodeEnd":363,"githubUrl":"https://github.com/langchain-ai/deepagents/blob/a1af029e6e73cb17c36bff823d227747b28e91e1/libs/deepagents/deepagents/middleware/rubric.py#L327-L363","documentation":"GraderResponse (a pydantic model in rubric.py) has a model_validator, _check_result_consistency, enforcing cross-field invariants. result='satisfied' is incompatible with any criterion having passed=False; this ValueError indicates the grader LLM produced internally inconsistent output.","triggerScenarios":"Constructing or parsing GraderResponse with result=\"satisfied\" while criteria contains at least one entry with passed=False (e.g. from a malformed or hallucinating grader model response).","commonSituations":"LLM grader outputs inconsistent structured JSON; hand-written test fixtures or mocked grader responses that violate the invariant.","solutions":["Fix the data: set result=\"needs_revision\" when any criterion passed=False","Or correct the failing criterion's passed to True if it truly passed","Harden the grader prompt/schema so the model produces consistent result/criteria pairs","If mocking, update fixtures to satisfy the invariant"],"exampleFix":"// before\nGraderResponse(result=\"satisfied\", criteria=[{\"passed\": False, \"gap\": \"...\"}])\n// after\nGraderResponse(result=\"needs_revision\", criteria=[{\"passed\": False, \"gap\": \"...\"}])","handlingStrategy":"validation","validationCode":"def is_consistent_grader_response(data: dict) -> bool:\n    if data.get(\"result\") == \"satisfied\":\n        return all(c.get(\"passed\", True) for c in data.get(\"criteria\", []))\n    return True\n# check before constructing GraderResponse / after parsing LLM JSON","typeGuard":"def grader_response_consistent(resp) -> bool:\n    has_fail = any(not c[\"passed\"] for c in resp.criteria)\n    return not (resp.result == \"satisfied\" and has_fail)","tryCatchPattern":"try:\n    resp = GraderResponse.model_validate(llm_output)\nexcept ValueError as e:\n    logger.warning(\"Inconsistent grader output: %s — retrying with stricter prompt\", e)\n    resp = regrade_with_strict_prompt()","preventionTips":["Constrain the grader LLM schema so result is derived from criteria, not free-form","Retry grading once when validation fails (transient LLM inconsistency)","Validate mocked fixtures with the same pydantic model used in production"],"tags":["python","validation","pydantic","rubric","llm"],"backgroundTag":"inconsistent-grader-result","analyzedSha":"a1af029e6e73cb17c36bff823d227747b28e91e1","analyzedAt":"2026-08-29T11:43:24.718Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}