run-llama/llama_index · error · ValueError

query and response must be provided

Error message

query and response must be provided

What it means

Thrown by GuidelineEvaluator.aevaluate when query or response is None (contexts is discarded). The evaluator formats the guideline prompt with the query/response pair, so both are required before the LLM call.

Source

Thrown at llama-index-core/llama_index/core/evaluation/guideline.py:101

    def _update_prompts(self, prompts: PromptDictType) -> None:
        """Update prompts."""
        if "eval_template" in prompts:
            self._eval_template = prompts["eval_template"]

    async def aevaluate(
        self,
        query: Optional[str] = None,
        response: Optional[str] = None,
        contexts: Optional[Sequence[str]] = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Evaluate whether the query and response pair passes the guidelines."""
        del contexts  # Unused
        del kwargs  # Unused
        if query is None or response is None:
            raise ValueError("query and response must be provided")

        logger.debug("prompt: %s", self._eval_template)
        logger.debug("query: %s", query)
        logger.debug("response: %s", response)
        logger.debug("guidelines: %s", self._guidelines)

        await asyncio.sleep(sleep_time_in_seconds)

        eval_response = await self._llm.apredict(
            self._eval_template,
            query=query,
            response=response,
            guidelines=self._guidelines,
        )
        eval_data = self._output_parser.parse(eval_response)
        eval_data = cast(EvaluationData, eval_data)

        return EvaluationResult(

View on GitHub (pinned to afd0fef371)

Solutions

1. Ensure both query and response are non-None strings at the call site.
2. Filter batch items first: skip pairs where either side is missing instead of evaluating them.
3. Log and count skipped items so silent data gaps become visible.

Example fix

# before
result = await evaluator.aevaluate(query=q, response=resp)  # resp None for failed gens

# after
if q and resp:
    result = await evaluator.aevaluate(query=q, response=str(resp))
else:
    skipped.append(q)

if not (query and response):
    return None
result = await evaluator.aevaluate(query=query, response=str(response))

def guideline_evaluable(query, response) -> bool:
    return query is not None and response is not None

When it happens

Trigger: Calling await evaluator.aevaluate(query=q, response=None) or with a missing query; via BatchRunner where per-item responses can be None (failed generations) and are forwarded as-is.

Common situations: Batch pipelines where some queries produced no answer; stringifying a None response object; datasets with empty answer fields passed without cleaning.

Related errors

• Both query and contexts must be provided
• query, and response must be provided
• contexts and response must be provided
• code_execute_fn must be provided for CodeActAgent
• query and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/6ecb5782bb9c8cad. Report an issue: GitHub.