run-llama/llama_index · error · ValueError

Evaluation is not set.

Error message

Evaluation is not set.

What it means

FeedbackQueryTransform (used by RetryQueryEngine / RetrySourceQueryEngine) resynthesizes a query based on an Evaluation of the previous response. _run requires self.evaluation (set at construction or passed via metadata['evaluation']) to be an Evaluation instance; otherwise it raises.

Source

Thrown at llama-index-core/llama_index/core/indices/query/query_transform/feedback_transform.py:67

        self.resynthesis_prompt = resynthesis_prompt or DEFAULT_RESYNTHESIS_PROMPT

    def _get_prompts(self) -> PromptDictType:
        """Get prompts."""
        return {"resynthesis_prompt": self.resynthesis_prompt}

    def _update_prompts(self, prompts: PromptDictType) -> None:
        """Update prompts."""
        if "resynthesis_prompt" in prompts:
            self.resynthesis_prompt = prompts["resynthesis_prompt"]

    def _run(self, query_bundle: QueryBundle, metadata: Dict) -> QueryBundle:
        orig_query_str = query_bundle.query_str
        if metadata.get("evaluation") and isinstance(
            metadata.get("evaluation"), Evaluation
        ):
            self.evaluation = metadata.get("evaluation")
        if self.evaluation is None or not isinstance(self.evaluation, Evaluation):
            raise ValueError("Evaluation is not set.")
        if self.evaluation.response is None or self.evaluation.feedback is None:
            raise ValueError("Evaluation result must contain response and feedback.")

        if self.evaluation.feedback == "YES" or self.evaluation.feedback == "NO":
            new_query = (
                orig_query_str
                + "\n----------------\n"
                + self._construct_feedback(response=self.evaluation.response)
            )
        else:
            if self.should_resynthesize_query:
                new_query_str = self._resynthesize_query(
                    orig_query_str, self.evaluation.response, self.evaluation.feedback
                )
            else:
                new_query_str = orig_query_str
            new_query = (
                self._construct_feedback(response=self.evaluation.response)

View on GitHub (pinned to afd0fef371)

Solutions

1. Pass the evaluation in metadata: transform.run(query, metadata={'evaluation': evaluation}) where evaluation came from BaseEvaluator.evaluate_response(...)
2. Or construct with an initial evaluation: FeedbackQueryTransform(evaluation=evaluation)
3. Use RetryQueryEngine.from_defaults(...) which pipelines the evaluator feedback for you
4. Ensure metadata['evaluation'] is a llama_index.core.evaluation.Evaluation instance, not a dict

Example fix

# before
transform = FeedbackQueryTransform()
new_query = transform.run(query_bundle)  # raises

# after
response = await query_engine.aquery(query_str)
evaluation = await evaluator.aevaluate_response(query=query_str, response=response)
new_query = transform.run(query_bundle, metadata={'evaluation': evaluation})

from llama_index.core.evaluation import Evaluation

evaluation = metadata.get('evaluation')
if not isinstance(evaluation, Evaluation):
    raise ValueError('FeedbackQueryTransform requires metadata["evaluation"] to be an Evaluation')
new_qb = transform.run(qb, metadata={'evaluation': evaluation})

def is_evaluation(obj) -> bool:
    from llama_index.core.evaluation import Evaluation
    return isinstance(obj, Evaluation)

try:
    new_qb = transform.run(qb, metadata=metadata)
except ValueError as e:
    if 'Evaluation is not set' in str(e):
        raise RuntimeError('producer did not attach an Evaluation; fix the retry pipeline') from e
    raise

When it happens

Trigger: Constructing FeedbackQueryTransform() with no evaluation kwarg and calling run/query without metadata={'evaluation': evaluator_result}; or passing a dict/other object instead of a BaseEvaluator-produced Evaluation in metadata.

Common situations: Wiring RetryQueryEngine manually instead of via its from_defaults helper and forgetting to thread the evaluator; passing the evaluator itself rather than its evaluate_response() output into metadata; building a custom retry loop around query_transform.run().

Related errors

• Evaluation result must contain response and feedback.
• First argument to Readability constructor should be a docume
• Aborting parsing document; {numTags} elements found
• Command failed: {command} {result.stderr}
• Git command failed: {result.stderr}

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/d44ddc12dcb9156d. Report an issue: GitHub.