langchain-ai/deepagents · error · ValueError

GraderResponse: result='needs_revision' but every criterion

Error message

GraderResponse: result='needs_revision' but every criterion has passed=True.

What it means

The same _check_result_consistency validator also rejects the opposite inconsistency: result='needs_revision' with non-empty criteria where every criterion has passed=True. If revision is needed, at least one criterion must have failed.

Source

Thrown at libs/deepagents/deepagents/middleware/rubric.py:348

    )

    @model_validator(mode="after")
    def _check_result_consistency(self) -> GraderResponse:
        """Reject grader output where `result` contradicts the per-criterion verdicts.

        The grader is an LLM and can hallucinate self-inconsistent
        responses (e.g. claiming `satisfied` while flagging a failing
        criterion). The discriminated union on `CriterionEval` enforces
        the per-criterion `gap` invariant; this validator catches the
        cross-field one.
        """
        has_fail = any(not c["passed"] for c in self.criteria)
        if self.result == "satisfied" and has_fail:
            msg = "GraderResponse: result='satisfied' but at least one criterion has passed=False."
            raise ValueError(msg)
        if self.result == "needs_revision" and self.criteria and not has_fail:
            msg = "GraderResponse: result='needs_revision' but every criterion has passed=True."
            raise ValueError(msg)
        return self


_StructuredOutputStrategy = Literal["ProviderStrategy", "ToolStrategy"]
"""Structured-output strategies LangChain can select for the grader."""


def _model_identifier(model: object) -> str | None:
    """Return the model identifier exposed by supported chat integrations.

    LangChain integrations do not share one identifier attribute: common
    implementations expose `model_name`, `model`, or `model_id`. Checking them
    in LangChain's precedence order keeps diagnostic labels and strategy
    inference consistent.
    """
    for attr in ("model_name", "model", "model_id"):
        value = getattr(model, attr, None)
        if isinstance(value, str) and value:

View on GitHub (pinned to a1af029e6e)

Solutions

1. Set result="satisfied" when all criteria passed
2. Or mark at least one criterion passed=False if revision truly is needed
3. Ensure an empty criteria list doesn't accompany needs_revision (that combination is allowed but suspicious) — align criteria with the verdict
4. Improve grader prompt so verdict and per-criterion flags stay consistent

Example fix

// before
GraderResponse(result="needs_revision", criteria=[{"passed": True}, {"passed": True}])
// after
GraderResponse(result="satisfied", criteria=[{"passed": True}, {"passed": True}])

def is_consistent_grader_response(data: dict) -> bool:
    if data.get("result") == "needs_revision" and data.get("criteria"):
        return any(not c.get("passed", True) for c in data["criteria"])
    return True

def grader_response_consistent(resp) -> bool:
    has_fail = any(not c["passed"] for c in resp.criteria)
    return not (resp.result == "needs_revision" and resp.criteria and not has_fail)

try:
    resp = GraderResponse.model_validate(llm_output)
except ValueError as e:
    logger.warning("Grader verdict/criteria mismatch: %s — re-grading", e)
    resp = regrade_with_strict_prompt()

When it happens

Trigger: Building GraderResponse with result="needs_revision" and criteria=[{..."passed": True...}] for all criteria (or criteria all defaulting to passed=True).

Common situations: Grader LLM emits needs_revision without marking any criterion failed; mocked/fixture responses written carelessly; empty explanatory text mapped to passed=True while overall result stayed needs_revision.

Related errors

• GraderResponse: result='satisfied' but at least one criterio
• RubricMiddleware: `model` is required.
• modes can only be provided when agent is a factory
• models can only be provided when agent is a factory
• -32602

AI-assisted analysis of langchain-ai/deepagents@a1af029e6e (2026-08-29). Data as JSON: /api/errors/2487bb3eb994116a. Report an issue: GitHub.