What causes "Both query and contexts must be provided" in run-llama/llama_index?

Always pass both arguments: await evaluator.aevaluate(query=q, contexts=ctxs) with ctxs a list of strings.

run-llama/llama_index · error · ValueError

Both query and contexts must be provided

Error message

Both query and contexts must be provided

What it means

Thrown by ContextRelevancyEvaluator.aevaluate when either query or contexts is None (response is deliberately discarded). The evaluator builds a SummaryIndex over the context strings and queries it with the query string, so both are mandatory.

Source

Thrown at llama-index-core/llama_index/core/evaluation/context_relevancy.py:143

        if "eval_template" in prompts:
            self._eval_template = prompts["eval_template"]
        if "refine_template" in prompts:
            self._refine_template = prompts["refine_template"]

    async def aevaluate(
        self,
        query: str | None = None,
        response: str | None = None,
        contexts: Sequence[str] | None = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Evaluate whether the contexts is relevant to the query."""
        del kwargs  # Unused
        del response  # Unused

        if query is None or contexts is None:
            raise ValueError("Both query and contexts must be provided")

        docs = [Document(text=context) for context in contexts]
        index = SummaryIndex.from_documents(docs)

        await asyncio.sleep(sleep_time_in_seconds)

        query_engine = index.as_query_engine(
            llm=self._llm,
            text_qa_template=self._eval_template,
            refine_template=self._refine_template,
        )
        response_obj = await query_engine.aquery(query)
        raw_response_txt = str(response_obj)

        score, reasoning = self.parser_function(raw_response_txt)

        invalid_result, invalid_reason = False, None
        if score is None and reasoning is None:

View on GitHub (pinned to afd0fef371)

Solutions

Always pass both arguments: await evaluator.aevaluate(query=q, contexts=ctxs) with ctxs a list of strings.
When using BatchRunner, pass contexts=[...] (aligned with queries) so each item gets a real list; use None entries only where genuinely absent.
If contexts can be empty, pass [] or [''] rather than None so evaluation can proceed (or skip that item yourself).

Example fix

# before
result = await evaluator.aevaluate(query=q, contexts=None)

# after
contexts = [n.get_content() for n in response.source_nodes] or [""]
result = await evaluator.aevaluate(query=q, contexts=contexts)

Defensive patterns

Strategy: validation

Validate before calling

if query is None or contexts is None:
    raise ValueError("skipping context relevancy: missing query or contexts") from None

Type guard

def evaluable_contexts(query, contexts) -> bool:
    return bool(query) and contexts is not None

Prevention

Always derive contexts from response.source_nodes and pass them alongside the query.
In BatchRunner runs, include an aligned contexts=[...] kwarg list.

When it happens

Trigger: Calling await evaluator.aevaluate(query=None, contexts=[...]) or aevaluate(query=q, contexts=None); commonly via BatchRunner where contexts were not supplied per-item (all-None contexts list is fine item-wise, but a missing list propagates None).

Common situations: Evaluating from a response object where source_nodes were never retrieved (contexts=[] vs None confusion); integrating with a custom retriever that returns None on empty; forgetting that BatchRunner needs contexts passed as a kwargs list for this evaluator.

Related errors

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/91d5224c7a6b769e. Report an issue: GitHub.