run-llama/llama_index · error · ValueError

query and response must be provided

Error message

query and response must be provided

What it means

AnswerRelevancyEvaluator.aevaluate() scores how well a response addresses a query and requires both pieces of text. It deliberately ignores contexts/kwargs, so passing only contexts (a common habit from faithfulness/relevancy evaluators) or omitting either string is rejected with this ValueError before any LLM call.

Source

Thrown at llama-index-core/llama_index/core/evaluation/answer_relevancy.py:117

        if "eval_template" in prompts:
            self._eval_template = prompts["eval_template"]
        if "refine_template" in prompts:
            self._refine_template = prompts["refine_template"]

    async def aevaluate(
        self,
        query: str | None = None,
        response: str | None = None,
        contexts: Sequence[str] | None = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Evaluate whether the response is relevant to the query."""
        del kwargs  # Unused
        del contexts  # Unused

        if query is None or response is None:
            raise ValueError("query and response must be provided")

        await asyncio.sleep(sleep_time_in_seconds)

        eval_response = await self._llm.apredict(
            prompt=self._eval_template,
            query=query,
            response=response,
        )

        score, reasoning = self.parser_function(eval_response)

        invalid_result, invalid_reason = False, None
        if score is None and reasoning is None:
            if self._raise_error:
                raise ValueError("The response is invalid")
            invalid_result = True
            invalid_reason = "Unable to parse the output string."

View on GitHub (pinned to afd0fef371)

Solutions

1. Pass both: await evaluator.aevaluate(query=..., response=...)
2. In shared eval loops, branch per evaluator type or use getattr on results to source query/response from a QueryResponse pair
3. Validate your eval dataset rows have non-null query and response fields before running

Example fix

// before
result = await evaluator.aevaluate(
    query="What is LlamaIndex?", contexts=response.source_nodes  # ValueError
)

// after
result = await evaluator.aevaluate(
    query="What is LlamaIndex?", response=response.response
)

if not query or not response:
    raise ValueError("AnswerRelevancyEvaluator needs non-empty query and response")

def is_evaluable_pair(query, response) -> bool:
    return bool(query) and bool(response)

try:
    result = await evaluator.aevaluate(query=q, response=r)
except ValueError as e:
    if "must be provided" in str(e):
        result = None  # skip incomplete eval rows
    else:
        raise

When it happens

Trigger: Calling evaluator.aevaluate(query=q) without response; aevaluate(response=r) without query; aevaluate(contexts=[...]) expecting contexts to suffice (they are deleted); positionally passing only one argument.

Common situations: Reusing one generic eval loop across multiple evaluator types (FaithfulnessEvaluator needs contexts, AnswerRelevancy needs query+response); refactors that drop the response field; batch-eval notebooks where one row has a missing value.

Related errors

• query, contexts, and response must be provided
• Metric key {metric_key} not in results_df
• names and results_arr must have same length.
• query, response, second_response, and reference must be prov
• query, contexts, and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/4ceb1905c61a90f6. Report an issue: GitHub.