langchain-ai/deepagents · error · ValueError

GraderResponse: result='satisfied' but at least one criterio

Error message

GraderResponse: result='satisfied' but at least one criterion has passed=False.

What it means

GraderResponse (a pydantic model in rubric.py) has a model_validator, _check_result_consistency, enforcing cross-field invariants. result='satisfied' is incompatible with any criterion having passed=False; this ValueError indicates the grader LLM produced internally inconsistent output.

Source

Thrown at libs/deepagents/deepagents/middleware/rubric.py:345

            "never omit criteria or collapse several into one. Each entry carries `passed` "
            "True/False, plus a `gap` string when failing."
        ),
    )

    @model_validator(mode="after")
    def _check_result_consistency(self) -> GraderResponse:
        """Reject grader output where `result` contradicts the per-criterion verdicts.

        The grader is an LLM and can hallucinate self-inconsistent
        responses (e.g. claiming `satisfied` while flagging a failing
        criterion). The discriminated union on `CriterionEval` enforces
        the per-criterion `gap` invariant; this validator catches the
        cross-field one.
        """
        has_fail = any(not c["passed"] for c in self.criteria)
        if self.result == "satisfied" and has_fail:
            msg = "GraderResponse: result='satisfied' but at least one criterion has passed=False."
            raise ValueError(msg)
        if self.result == "needs_revision" and self.criteria and not has_fail:
            msg = "GraderResponse: result='needs_revision' but every criterion has passed=True."
            raise ValueError(msg)
        return self


_StructuredOutputStrategy = Literal["ProviderStrategy", "ToolStrategy"]
"""Structured-output strategies LangChain can select for the grader."""


def _model_identifier(model: object) -> str | None:
    """Return the model identifier exposed by supported chat integrations.

    LangChain integrations do not share one identifier attribute: common
    implementations expose `model_name`, `model`, or `model_id`. Checking them
    in LangChain's precedence order keeps diagnostic labels and strategy
    inference consistent.
    """

View on GitHub (pinned to a1af029e6e)

Solutions

Fix the data: set result="needs_revision" when any criterion passed=False
Or correct the failing criterion's passed to True if it truly passed
Harden the grader prompt/schema so the model produces consistent result/criteria pairs
If mocking, update fixtures to satisfy the invariant

Example fix

// before
GraderResponse(result="satisfied", criteria=[{"passed": False, "gap": "..."}])
// after
GraderResponse(result="needs_revision", criteria=[{"passed": False, "gap": "..."}])

Defensive patterns

Strategy: validation

Validate before calling

def is_consistent_grader_response(data: dict) -> bool:
    if data.get("result") == "satisfied":
        return all(c.get("passed", True) for c in data.get("criteria", []))
    return True
# check before constructing GraderResponse / after parsing LLM JSON

Type guard

def grader_response_consistent(resp) -> bool:
    has_fail = any(not c["passed"] for c in resp.criteria)
    return not (resp.result == "satisfied" and has_fail)

Try / catch

try:
    resp = GraderResponse.model_validate(llm_output)
except ValueError as e:
    logger.warning("Inconsistent grader output: %s — retrying with stricter prompt", e)
    resp = regrade_with_strict_prompt()

Prevention

Constrain the grader LLM schema so result is derived from criteria, not free-form
Retry grading once when validation fails (transient LLM inconsistency)
Validate mocked fixtures with the same pydantic model used in production

When it happens

Trigger: Constructing or parsing GraderResponse with result="satisfied" while criteria contains at least one entry with passed=False (e.g. from a malformed or hallucinating grader model response).

Common situations: LLM grader outputs inconsistent structured JSON; hand-written test fixtures or mocked grader responses that violate the invariant.

Related errors

AI-assisted analysis of langchain-ai/deepagents@a1af029e6e (2026-08-29). Data as JSON: /api/errors/2bd13c0f3fdf0be1. Report an issue: GitHub.