crewAIInc/crewAI · error · Exception

Failed to evaluate model input and output. Response status c

Error message

Failed to evaluate model input and output. Response status code: {response.status_code}. Reason: {response.text}

What it means

Raised by PatronusEvaluationTool when the HTTP call to Patronus's /evaluate endpoint returns a non-200 status. The tool posts a JSON payload (evaluator config plus model input/output) with the API key header and a 30s timeout; any response other than 200 — bad key, malformed request, wrong evaluator name, server error — surfaces as this generic Exception with the status code and response body embedded.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_eval_tool.py:152

            "evaluated_model_retrieved_context": evaluated_model_retrieved_context,
            "evaluators": evals,
        }

        api_key = os.getenv("PATRONUS_API_KEY", "")
        headers = {
            "X-API-KEY": api_key,
            "accept": "application/json",
            "content-type": "application/json",
        }

        response = requests.post(
            self.evaluate_url,
            headers=headers,
            data=json.dumps(data),
            timeout=30,
        )
        if response.status_code != 200:
            raise Exception(
                f"Failed to evaluate model input and output. Response status code: {response.status_code}. Reason: {response.text}"
            )

        return response.json()

View on GitHub (pinned to 754d7323be)

Solutions

1. Inspect response.text embedded in the message — it names the exact API-side reason (auth, validation, server error)
2. Verify the Patronus API key is valid and correctly passed (PATRONUS_API_KEY env var or constructor arg)
3. Check the evaluator specification (name/ID exists for your account, criteria fields are complete)
4. If the status is 5xx or rate-limit related, retry after a short backoff
5. Confirm evaluate_url matches the current Patronus API base URL documented for your account

Example fix

# before
tool = PatronusEvaluationTool()
result = tool.run(...)  # raises Exception: status 401

# after
import os
assert os.environ.get("PATRONUS_API_KEY"), "set PATRONUS_API_KEY"
result = tool.run(...)  # valid key -> 200

import os

def patronus_ready() -> bool:
    return bool(os.environ.get("PATRONUS_API_KEY"))

import time
for attempt in range(3):
    try:
        result = tool.run(...)
        break
    except Exception as e:
        msg = str(e)
        if "status code: 5" in msg or "status code: 429" in msg:
            time.sleep(2 ** attempt)
            continue
        raise  # 4xx auth/validation errors are not retryable

When it happens

Trigger: Calling the tool's evaluation with an invalid/expired Patronus API key (401/403), referencing a nonexistent evaluator id (4xx), sending a payload the API rejects (422), or Patronus returning a 5xx. Also triggered if the evaluate_url property points at a wrong endpoint.

Common situations: Expired API key, typo in the evaluator name, using a custom criteria payload with missing required fields, environment drift between staging and production API URLs, or transient 5xx/timeout issues from the Patronus service.

Related errors

• Failed to evaluate model input and output. Status code: {res
• Project name '{name}' would generate invalid Python class na
• Failed to retrieve organization list: {e!s}
• Error. A valid pyproject.toml file is required. Check that a
• Error: {e}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/a3a2e122a062e905. Report an issue: GitHub.