{"record":{"id":"8e29b25422d7f66d","repo":"crewAIInc/crewAI","slug":"failed-to-evaluate-model-input-and-output-status","errorCode":null,"errorMessage":"Failed to evaluate model input and output. Status code: {response.status_code}. Reason: {response.text}","messagePattern":"Failed to evaluate model input and output\\. Status code: (.+?)\\. Reason: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_predefined_criteria_eval_tool.py","lineNumber":102,"sourceCode":"                evaluated_model_gold_answer\n                if isinstance(evaluated_model_gold_answer, str)\n                else evaluated_model_gold_answer.get(\"description\")  # type: ignore[union-attr]\n            ),\n            \"evaluators\": (\n                evaluators\n                if isinstance(evaluators, list)\n                else evaluators.get(\"description\")\n            ),\n        }\n\n        response = requests.post(\n            self.evaluate_url,\n            headers=headers,\n            data=json.dumps(data),\n            timeout=30,\n        )\n        if response.status_code != 200:\n            raise Exception(\n                f\"Failed to evaluate model input and output. Status code: {response.status_code}. Reason: {response.text}\"\n            )\n\n        return response.json()\n","sourceCodeStart":84,"sourceCodeEnd":107,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_predefined_criteria_eval_tool.py#L84-L107","documentation":"Raised by PatronusPredefinedCriteriaEvaluationTool when its POST to the Patronus evaluate endpoint returns a non-200 status. Unlike the custom-criteria variant, this tool sends a named predefined evaluator (e.g. 'lynx-v1'); any auth failure, unknown predefined evaluator name, malformed payload, or server error produces this Exception with the status code and API's reason text.","triggerScenarios":"Calling _run/evaluation with an invalid API key (401/403), a predefined evaluator name that does not exist or isn't enabled for the account (4xx), missing evaluated_model_input/output fields (422), or a Patronus-side 5xx.","commonSituations":"Typo in the predefined criteria/evaluator name, key from a different environment (staging vs prod), API contract changes after a Patronus update, or rate limiting during batch evaluation runs.","solutions":["Read response.text in the message — it identifies auth vs evaluator-name vs validation failure","Verify PATRONUS_API_KEY (or the constructor arg) is the correct, active key","Confirm the predefined evaluator name exists in the current Patronus evaluator catalog","Add retry with exponential backoff for 5xx/429 responses during batch runs","Ensure evaluated_model_input and evaluated_model_output kwargs are non-empty strings"],"exampleFix":"# before\ntool = PatronusPredefinedCriteriaEvaluationTool(criteria=\"lynx-v99\")  # bad name -> 4xx\n\n# after\ntool = PatronusPredefinedCriteriaEvaluationTool(criteria=\"lynx-v1\")\nresult = tool.run(evaluated_model_input=..., evaluated_model_output=...)\n","handlingStrategy":"retry","validationCode":"import os\n\ndef patronus_predefined_ready() -> bool:\n    return bool(os.environ.get(\"PATRONUS_API_KEY\"))","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        result = tool.run(evaluated_model_input=i, evaluated_model_output=o)\n        break\n    except Exception as e:\n        s = str(e)\n        if \"status code: 5\" in s or \"status code: 429\" in s:\n            time.sleep(2 ** attempt); continue\n        raise","preventionTips":["Confirm the predefined evaluator name against the Patronus catalog before use","Supply non-empty evaluated_model_input/output strings every call","Cache evaluation results to avoid hitting rate limits in batch runs","Distinguish 4xx (fix request) from 5xx/429 (retry) in your handler"],"tags":["api","http","authentication","patronus","llm-evaluation"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}