run-llama/llama_index · error · ValueError

contexts and response must be provided

Error message

contexts and response must be provided

What it means

Thrown by FaithfulnessEvaluator.aevaluate when contexts or response is None. The evaluator builds a SummaryIndex over the context strings and queries it with the response text, so both must be present to judge faithfulness.

Source

Thrown at llama-index-core/llama_index/core/evaluation/faithfulness.py:173

            self._eval_template = prompts["eval_template"]
        if "refine_template" in prompts:
            self._refine_template = prompts["refine_template"]

    async def aevaluate(
        self,
        query: str | None = None,
        response: str | None = None,
        contexts: Sequence[str] | None = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Evaluate whether the response is faithful to the contexts."""
        del kwargs  # Unused

        await asyncio.sleep(sleep_time_in_seconds)

        if contexts is None or response is None:
            raise ValueError("contexts and response must be provided")

        docs = [Document(text=context) for context in contexts]
        index = SummaryIndex.from_documents(docs)

        query_engine = index.as_query_engine(
            llm=self._llm,
            text_qa_template=self._eval_template,
            refine_template=self._refine_template,
        )
        response_obj = await query_engine.aquery(response)

        raw_response_txt = str(response_obj)

        if "yes" in raw_response_txt.lower():
            passing = True
        else:
            passing = False
            if self._raise_error:

View on GitHub (pinned to afd0fef371)

Solutions

1. Pass both: await evaluator.aevaluate(response=r, contexts=[c1, c2, ...]).
2. With BatchRunner, include a contexts list aligned with queries/responses.
3. Guard upstream: skip evaluation when a response or its retrieved nodes are missing rather than forwarding None.

Example fix

# before
result = await evaluator.aevaluate(response=str(resp))

# after
contexts = [n.get_content() for n in resp.source_nodes]
if not contexts:
    raise/skip  # nothing to verify faithfulness against
result = await evaluator.aevaluate(response=str(resp), contexts=contexts)

contexts = [n.get_content() for n in response.source_nodes]
if response is None or not contexts:
    raise ValueError("cannot evaluate faithfulness without response and contexts")

def faithfulness_evaluable(response, contexts) -> bool:
    return response is not None and contexts is not None and len(contexts) > 0

When it happens

Trigger: Calling await evaluator.aevaluate(response=r, contexts=None) or aevaluate(response=None, contexts=ctxs); via BatchRunner when the contexts kwarg list was omitted; passing response.source_nodes of an empty/failed retrieval as None.

Common situations: Forgetting to pass the contexts kwarg to aevaluate/aevaluate_responses (query alone is not enough for faithfulness); a query engine exception path returning None response; converting source_nodes to texts with a helper that returns None on empty.

Related errors

• Both query and contexts must be provided
• query, and response must be provided
• The response is invalid
• query and response must be provided
• contexts and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/85729e8618dac17f. Report an issue: GitHub.