run-llama/llama_index · error · ValueError

query, contexts, and response must be provided

Error message

query, contexts, and response must be provided

What it means

RelevancyEvaluator.aevaluate (text-only) requires query, contexts, and response: it joins contexts into Documents, builds a SummaryIndex, and queries it with the eval template. If any of the three is None it raises ValueError immediately — the Optional annotations exist only for keyword-based calling.

Source

Thrown at llama-index-core/llama_index/core/evaluation/relevancy.py:109

        """Update prompts."""
        if "eval_template" in prompts:
            self._eval_template = prompts["eval_template"]
        if "refine_template" in prompts:
            self._refine_template = prompts["refine_template"]

    async def aevaluate(
        self,
        query: str | None = None,
        response: str | None = None,
        contexts: Sequence[str] | None = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Evaluate whether the contexts and response are relevant to the query."""
        del kwargs  # Unused

        if query is None or contexts is None or response is None:
            raise ValueError("query, contexts, and response must be provided")

        docs = [Document(text=context) for context in contexts]
        index = SummaryIndex.from_documents(docs)

        query_response = f"Question: {query}\nResponse: {response}"

        await asyncio.sleep(sleep_time_in_seconds)

        query_engine = index.as_query_engine(
            llm=self._llm,
            text_qa_template=self._eval_template,
            refine_template=self._refine_template,
        )
        response_obj = await query_engine.aquery(query_response)

        raw_response_txt = str(response_obj)

        if "yes" in raw_response_txt.lower():

View on GitHub (pinned to afd0fef371)

Solutions

1. Ensure all three are provided and non-None before the call
2. Coalesce missing values intentionally (response or "", contexts or []) if you want degenerate evals instead of crashes
3. Pre-filter datasets in your eval harness: drop rows lacking query/contexts/response

Example fix

# before
result = await evaluator.aevaluate(query=q, contexts=ctx, response=None)

# after
if not (q and ctx and resp):
    continue
result = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)

def eval_sample_ready(sample) -> bool:
    return bool(sample.query and sample.contexts is not None and sample.response)

ready = [s for s in samples if eval_sample_ready(s)]
results = await asyncio.gather(*[
    evaluator.aevaluate(query=s.query, contexts=s.contexts, response=s.response)
    for s in ready
])

try:
    res = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)
except ValueError as e:
    if "must be provided" in str(e):
        res = None
    else:
        raise

When it happens

Trigger: Awaiting aevaluate(query=q, contexts=None, response=r), e.g. contexts list lost during serialization or an empty retrieval step returning None; omitting any of the three kwargs.

Common situations: Async eval batches where retrieval returned nothing for some queries; rag pipeline returning None response on error; data loading producing None fields that flow into the evaluator.

Related errors

• query, response, second_response, and reference must be prov
• query, contexts, and response must be provided
• names and results_arr must have same length.
• The response is invalid
• Retrieved ids and expected ids must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/3bc5dccf2b15e98d. Report an issue: GitHub.