run-llama/llama_index · error · ValueError

Evaluation result must contain response and feedback.

Error message

Evaluation result must contain response and feedback.

What it means

FeedbackQueryTransform._run additionally requires the Evaluation to carry both a non-None response and feedback. An Evaluation built manually, or one whose fields were never populated because the evaluator failed, fails this check.

Source

Thrown at llama-index-core/llama_index/core/indices/query/query_transform/feedback_transform.py:69

    def _get_prompts(self) -> PromptDictType:
        """Get prompts."""
        return {"resynthesis_prompt": self.resynthesis_prompt}

    def _update_prompts(self, prompts: PromptDictType) -> None:
        """Update prompts."""
        if "resynthesis_prompt" in prompts:
            self.resynthesis_prompt = prompts["resynthesis_prompt"]

    def _run(self, query_bundle: QueryBundle, metadata: Dict) -> QueryBundle:
        orig_query_str = query_bundle.query_str
        if metadata.get("evaluation") and isinstance(
            metadata.get("evaluation"), Evaluation
        ):
            self.evaluation = metadata.get("evaluation")
        if self.evaluation is None or not isinstance(self.evaluation, Evaluation):
            raise ValueError("Evaluation is not set.")
        if self.evaluation.response is None or self.evaluation.feedback is None:
            raise ValueError("Evaluation result must contain response and feedback.")

        if self.evaluation.feedback == "YES" or self.evaluation.feedback == "NO":
            new_query = (
                orig_query_str
                + "\n----------------\n"
                + self._construct_feedback(response=self.evaluation.response)
            )
        else:
            if self.should_resynthesize_query:
                new_query_str = self._resynthesize_query(
                    orig_query_str, self.evaluation.response, self.evaluation.feedback
                )
            else:
                new_query_str = orig_query_str
            new_query = (
                self._construct_feedback(response=self.evaluation.response)
                + "\n"
                + "Here is some feedback from the evaluator about the response given.\n"

View on GitHub (pinned to afd0fef371)

Solutions

1. Always produce the Evaluation via BaseEvaluator.evaluate_response(query, response) so response and feedback are populated
2. If wrapping a custom judge, ensure your evaluator sets evaluation.response and evaluation.feedback (e.g. 'YES'/'NO' plus reasoning) before returning
3. Guard before transform.run: skip or raise early if evaluation.response is None or evaluation.feedback is None

Example fix

# before
evaluation = Evaluation(query=query_str, response=None)  # manual, incomplete
transform.run(qb, metadata={'evaluation': evaluation})

# after
evaluation = await evaluator.aevaluate_response(query=query_str, response=response)
if evaluation.response is None or evaluation.feedback is None:
    raise RuntimeError('evaluator produced incomplete evaluation')
transform.run(qb, metadata={'evaluation': evaluation})

if self.evaluation is None or getattr(self.evaluation, 'response', None) is None or getattr(self.evaluation, 'feedback', None) is None:
    raise ValueError('evaluation incomplete; ensure evaluate_response() populated response+feedback')

def is_complete_evaluation(ev) -> bool:
    from llama_index.core.evaluation import Evaluation
    return (
        isinstance(ev, Evaluation)
        and ev.response is not None
        and ev.feedback is not None
    )

try:
    new_qb = transform.run(qb, metadata={'evaluation': evaluation})
except ValueError as e:
    if 'response and feedback' in str(e):
        evaluation = await evaluator.aevaluate_response(query, response)  # regenerate
        new_qb = transform.run(qb, metadata={'evaluation': evaluation})
    else:
        raise

When it happens

Trigger: Constructing Evaluation(query, response=None) or Evaluation(query, response=..., feedback=None) manually and passing it in metadata; mutating/copying an Evaluation and dropping feedback; calling with a partially-initialized Evaluation after an evaluator exception was swallowed.

Common situations: Custom evaluators that return Evaluation objects without setting feedback (e.g. returning None on parse failure); retry loops that reuse a stale evaluation object across iterations where the first evaluation lacked a response.

Related errors

• Evaluation is not set.
• First argument to Readability constructor should be a docume
• Aborting parsing document; {numTags} elements found
• Command failed: {command} {result.stderr}
• Git command failed: {result.stderr}

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/a6c470a15d62ca65. Report an issue: GitHub.