run-llama/llama_index · error · ValueError

query, and response must be provided

Error message

query, and response must be provided

What it means

Thrown by CorrectnessEvaluator.aevaluate when query or response is None (contexts is discarded). The evaluator formats the eval prompt with query, generated_answer, and a reference_answer, so the query/response pair is mandatory; a missing reference degrades gracefully to a placeholder string instead.

Source

Thrown at llama-index-core/llama_index/core/evaluation/correctness.py:135

        if "eval_template" in prompts:
            self._eval_template = prompts["eval_template"]

    async def aevaluate(
        self,
        query: Optional[str] = None,
        response: Optional[str] = None,
        contexts: Optional[Sequence[str]] = None,
        reference: Optional[str] = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        del kwargs  # Unused
        del contexts  # Unused

        await asyncio.sleep(sleep_time_in_seconds)

        if query is None or response is None:
            raise ValueError("query, and response must be provided")

        eval_response = await self._llm.apredict(
            prompt=self._eval_template,
            query=query,
            generated_answer=response,
            reference_answer=reference or "(NO REFERENCE ANSWER SUPPLIED)",
        )

        # Use the parser function
        score, reasoning = self.parser_function(eval_response)

        return EvaluationResult(
            query=query,
            response=response,
            passing=score >= self._score_threshold if score is not None else None,
            score=score,
            feedback=reasoning,
        )

View on GitHub (pinned to afd0fef371)

Solutions

1. Ensure both query and response are non-None strings before calling aevaluate.
2. In batch pipelines, filter or placeholder-fill: skip items whose response is None, or substitute "(NO RESPONSE)" if you want a scored result.
3. Validate your QA dataset up front for null query/answer fields.

Example fix

# before
result = await evaluator.aevaluate(query=q, response=resp)  # resp may be None

# after
if q is None or resp is None:
    continue  # or skip/log this item
result = await evaluator.aevaluate(query=q, response=str(resp))

if not query or not response:
    return None  # skip incomplete pair
result = await evaluator.aevaluate(query=query, response=str(response))

def complete_qa_pair(q, r) -> bool:
    return q is not None and r is not None

When it happens

Trigger: Calling await evaluator.aevaluate(query=None, response=r) or aevaluate(query=q, response=None); via BatchRunner when responses for some items are None (e.g. a query engine returned nothing) and are passed straight through.

Common situations: Evaluating retrieved question/answer pairs where one side is missing; response objects stringified to None because the engine errored or returned an empty completion; dataset rows with null fields.

Related errors

• Both query and contexts must be provided
• contexts and response must be provided
• query and response must be provided
• code_execute_fn must be provided for CodeActAgent
• query and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/128554d8efdc0cef. Report an issue: GitHub.