crewAIInc/crewAI · error · Exception

Failed to evaluate model input and output. Status code: {res

Error message

Failed to evaluate model input and output. Status code: {response.status_code}. Reason: {response.text}

What it means

Raised by PatronusPredefinedCriteriaEvaluationTool when its POST to the Patronus evaluate endpoint returns a non-200 status. Unlike the custom-criteria variant, this tool sends a named predefined evaluator (e.g. 'lynx-v1'); any auth failure, unknown predefined evaluator name, malformed payload, or server error produces this Exception with the status code and API's reason text.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_predefined_criteria_eval_tool.py:102

                evaluated_model_gold_answer
                if isinstance(evaluated_model_gold_answer, str)
                else evaluated_model_gold_answer.get("description")  # type: ignore[union-attr]
            ),
            "evaluators": (
                evaluators
                if isinstance(evaluators, list)
                else evaluators.get("description")
            ),
        }

        response = requests.post(
            self.evaluate_url,
            headers=headers,
            data=json.dumps(data),
            timeout=30,
        )
        if response.status_code != 200:
            raise Exception(
                f"Failed to evaluate model input and output. Status code: {response.status_code}. Reason: {response.text}"
            )

        return response.json()

View on GitHub (pinned to 754d7323be)

Solutions

1. Read response.text in the message — it identifies auth vs evaluator-name vs validation failure
2. Verify PATRONUS_API_KEY (or the constructor arg) is the correct, active key
3. Confirm the predefined evaluator name exists in the current Patronus evaluator catalog
4. Add retry with exponential backoff for 5xx/429 responses during batch runs
5. Ensure evaluated_model_input and evaluated_model_output kwargs are non-empty strings

Example fix

# before
tool = PatronusPredefinedCriteriaEvaluationTool(criteria="lynx-v99")  # bad name -> 4xx

# after
tool = PatronusPredefinedCriteriaEvaluationTool(criteria="lynx-v1")
result = tool.run(evaluated_model_input=..., evaluated_model_output=...)

import os

def patronus_predefined_ready() -> bool:
    return bool(os.environ.get("PATRONUS_API_KEY"))

for attempt in range(3):
    try:
        result = tool.run(evaluated_model_input=i, evaluated_model_output=o)
        break
    except Exception as e:
        s = str(e)
        if "status code: 5" in s or "status code: 429" in s:
            time.sleep(2 ** attempt); continue
        raise

When it happens

Trigger: Calling _run/evaluation with an invalid API key (401/403), a predefined evaluator name that does not exist or isn't enabled for the account (4xx), missing evaluated_model_input/output fields (422), or a Patronus-side 5xx.

Common situations: Typo in the predefined criteria/evaluator name, key from a different environment (staging vs prod), API contract changes after a Patronus update, or rate limiting during batch evaluation runs.

Related errors

• Failed to evaluate model input and output. Response status c
• Project name '{name}' would generate invalid Python class na
• Failed to retrieve organization list: {e!s}
• Error. A valid pyproject.toml file is required. Check that a
• Error: {e}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/8e29b25422d7f66d. Report an issue: GitHub.