{"record":{"id":"d44ddc12dcb9156d","repo":"run-llama/llama_index","slug":"evaluation-is-not-set","errorCode":null,"errorMessage":"Evaluation is not set.","messagePattern":"Evaluation is not set\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/indices/query/query_transform/feedback_transform.py","lineNumber":67,"sourceCode":"        self.resynthesis_prompt = resynthesis_prompt or DEFAULT_RESYNTHESIS_PROMPT\n\n    def _get_prompts(self) -> PromptDictType:\n        \"\"\"Get prompts.\"\"\"\n        return {\"resynthesis_prompt\": self.resynthesis_prompt}\n\n    def _update_prompts(self, prompts: PromptDictType) -> None:\n        \"\"\"Update prompts.\"\"\"\n        if \"resynthesis_prompt\" in prompts:\n            self.resynthesis_prompt = prompts[\"resynthesis_prompt\"]\n\n    def _run(self, query_bundle: QueryBundle, metadata: Dict) -> QueryBundle:\n        orig_query_str = query_bundle.query_str\n        if metadata.get(\"evaluation\") and isinstance(\n            metadata.get(\"evaluation\"), Evaluation\n        ):\n            self.evaluation = metadata.get(\"evaluation\")\n        if self.evaluation is None or not isinstance(self.evaluation, Evaluation):\n            raise ValueError(\"Evaluation is not set.\")\n        if self.evaluation.response is None or self.evaluation.feedback is None:\n            raise ValueError(\"Evaluation result must contain response and feedback.\")\n\n        if self.evaluation.feedback == \"YES\" or self.evaluation.feedback == \"NO\":\n            new_query = (\n                orig_query_str\n                + \"\\n----------------\\n\"\n                + self._construct_feedback(response=self.evaluation.response)\n            )\n        else:\n            if self.should_resynthesize_query:\n                new_query_str = self._resynthesize_query(\n                    orig_query_str, self.evaluation.response, self.evaluation.feedback\n                )\n            else:\n                new_query_str = orig_query_str\n            new_query = (\n                self._construct_feedback(response=self.evaluation.response)","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/indices/query/query_transform/feedback_transform.py#L49-L85","documentation":"FeedbackQueryTransform (used by RetryQueryEngine / RetrySourceQueryEngine) resynthesizes a query based on an Evaluation of the previous response. _run requires self.evaluation (set at construction or passed via metadata['evaluation']) to be an Evaluation instance; otherwise it raises.","triggerScenarios":"Constructing FeedbackQueryTransform() with no evaluation kwarg and calling run/query without metadata={'evaluation': evaluator_result}; or passing a dict/other object instead of a BaseEvaluator-produced Evaluation in metadata.","commonSituations":"Wiring RetryQueryEngine manually instead of via its from_defaults helper and forgetting to thread the evaluator; passing the evaluator itself rather than its evaluate_response() output into metadata; building a custom retry loop around query_transform.run().","solutions":["Pass the evaluation in metadata: transform.run(query, metadata={'evaluation': evaluation}) where evaluation came from BaseEvaluator.evaluate_response(...)","Or construct with an initial evaluation: FeedbackQueryTransform(evaluation=evaluation)","Use RetryQueryEngine.from_defaults(...) which pipelines the evaluator feedback for you","Ensure metadata['evaluation'] is a llama_index.core.evaluation.Evaluation instance, not a dict"],"exampleFix":"# before\ntransform = FeedbackQueryTransform()\nnew_query = transform.run(query_bundle)  # raises\n\n# after\nresponse = await query_engine.aquery(query_str)\nevaluation = await evaluator.aevaluate_response(query=query_str, response=response)\nnew_query = transform.run(query_bundle, metadata={'evaluation': evaluation})","handlingStrategy":"validation","validationCode":"from llama_index.core.evaluation import Evaluation\n\nevaluation = metadata.get('evaluation')\nif not isinstance(evaluation, Evaluation):\n    raise ValueError('FeedbackQueryTransform requires metadata[\"evaluation\"] to be an Evaluation')\nnew_qb = transform.run(qb, metadata={'evaluation': evaluation})","typeGuard":"def is_evaluation(obj) -> bool:\n    from llama_index.core.evaluation import Evaluation\n    return isinstance(obj, Evaluation)","tryCatchPattern":"try:\n    new_qb = transform.run(qb, metadata=metadata)\nexcept ValueError as e:\n    if 'Evaluation is not set' in str(e):\n        raise RuntimeError('producer did not attach an Evaluation; fix the retry pipeline') from e\n    raise","preventionTips":["Use RetryQueryEngine.from_defaults to get a correctly wired feedback loop","Always thread evaluator output through metadata rather than relying on constructor state"],"tags":["llama-index","query-transform","retry-engine","evaluation"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}