{"record":{"id":"a6c470a15d62ca65","repo":"run-llama/llama_index","slug":"evaluation-result-must-contain-response-and-feedba","errorCode":null,"errorMessage":"Evaluation result must contain response and feedback.","messagePattern":"Evaluation result must contain response and feedback\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/indices/query/query_transform/feedback_transform.py","lineNumber":69,"sourceCode":"    def _get_prompts(self) -> PromptDictType:\n        \"\"\"Get prompts.\"\"\"\n        return {\"resynthesis_prompt\": self.resynthesis_prompt}\n\n    def _update_prompts(self, prompts: PromptDictType) -> None:\n        \"\"\"Update prompts.\"\"\"\n        if \"resynthesis_prompt\" in prompts:\n            self.resynthesis_prompt = prompts[\"resynthesis_prompt\"]\n\n    def _run(self, query_bundle: QueryBundle, metadata: Dict) -> QueryBundle:\n        orig_query_str = query_bundle.query_str\n        if metadata.get(\"evaluation\") and isinstance(\n            metadata.get(\"evaluation\"), Evaluation\n        ):\n            self.evaluation = metadata.get(\"evaluation\")\n        if self.evaluation is None or not isinstance(self.evaluation, Evaluation):\n            raise ValueError(\"Evaluation is not set.\")\n        if self.evaluation.response is None or self.evaluation.feedback is None:\n            raise ValueError(\"Evaluation result must contain response and feedback.\")\n\n        if self.evaluation.feedback == \"YES\" or self.evaluation.feedback == \"NO\":\n            new_query = (\n                orig_query_str\n                + \"\\n----------------\\n\"\n                + self._construct_feedback(response=self.evaluation.response)\n            )\n        else:\n            if self.should_resynthesize_query:\n                new_query_str = self._resynthesize_query(\n                    orig_query_str, self.evaluation.response, self.evaluation.feedback\n                )\n            else:\n                new_query_str = orig_query_str\n            new_query = (\n                self._construct_feedback(response=self.evaluation.response)\n                + \"\\n\"\n                + \"Here is some feedback from the evaluator about the response given.\\n\"","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/indices/query/query_transform/feedback_transform.py#L51-L87","documentation":"FeedbackQueryTransform._run additionally requires the Evaluation to carry both a non-None response and feedback. An Evaluation built manually, or one whose fields were never populated because the evaluator failed, fails this check.","triggerScenarios":"Constructing Evaluation(query, response=None) or Evaluation(query, response=..., feedback=None) manually and passing it in metadata; mutating/copying an Evaluation and dropping feedback; calling with a partially-initialized Evaluation after an evaluator exception was swallowed.","commonSituations":"Custom evaluators that return Evaluation objects without setting feedback (e.g. returning None on parse failure); retry loops that reuse a stale evaluation object across iterations where the first evaluation lacked a response.","solutions":["Always produce the Evaluation via BaseEvaluator.evaluate_response(query, response) so response and feedback are populated","If wrapping a custom judge, ensure your evaluator sets evaluation.response and evaluation.feedback (e.g. 'YES'/'NO' plus reasoning) before returning","Guard before transform.run: skip or raise early if evaluation.response is None or evaluation.feedback is None"],"exampleFix":"# before\nevaluation = Evaluation(query=query_str, response=None)  # manual, incomplete\ntransform.run(qb, metadata={'evaluation': evaluation})\n\n# after\nevaluation = await evaluator.aevaluate_response(query=query_str, response=response)\nif evaluation.response is None or evaluation.feedback is None:\n    raise RuntimeError('evaluator produced incomplete evaluation')\ntransform.run(qb, metadata={'evaluation': evaluation})","handlingStrategy":"validation","validationCode":"if self.evaluation is None or getattr(self.evaluation, 'response', None) is None or getattr(self.evaluation, 'feedback', None) is None:\n    raise ValueError('evaluation incomplete; ensure evaluate_response() populated response+feedback')","typeGuard":"def is_complete_evaluation(ev) -> bool:\n    from llama_index.core.evaluation import Evaluation\n    return (\n        isinstance(ev, Evaluation)\n        and ev.response is not None\n        and ev.feedback is not None\n    )","tryCatchPattern":"try:\n    new_qb = transform.run(qb, metadata={'evaluation': evaluation})\nexcept ValueError as e:\n    if 'response and feedback' in str(e):\n        evaluation = await evaluator.aevaluate_response(query, response)  # regenerate\n        new_qb = transform.run(qb, metadata={'evaluation': evaluation})\n    else:\n        raise","preventionTips":["Never hand-construct Evaluation objects for the feedback transform","Fail fast after evaluation: check response/feedback are non-None before reusing the evaluation"],"tags":["llama-index","query-transform","evaluation","retry-engine"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}