{"record":{"id":"85729e8618dac17f","repo":"run-llama/llama_index","slug":"contexts-and-response-must-be-provided","errorCode":null,"errorMessage":"contexts and response must be provided","messagePattern":"contexts and response must be provided","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/faithfulness.py","lineNumber":173,"sourceCode":"            self._eval_template = prompts[\"eval_template\"]\n        if \"refine_template\" in prompts:\n            self._refine_template = prompts[\"refine_template\"]\n\n    async def aevaluate(\n        self,\n        query: str | None = None,\n        response: str | None = None,\n        contexts: Sequence[str] | None = None,\n        sleep_time_in_seconds: int = 0,\n        **kwargs: Any,\n    ) -> EvaluationResult:\n        \"\"\"Evaluate whether the response is faithful to the contexts.\"\"\"\n        del kwargs  # Unused\n\n        await asyncio.sleep(sleep_time_in_seconds)\n\n        if contexts is None or response is None:\n            raise ValueError(\"contexts and response must be provided\")\n\n        docs = [Document(text=context) for context in contexts]\n        index = SummaryIndex.from_documents(docs)\n\n        query_engine = index.as_query_engine(\n            llm=self._llm,\n            text_qa_template=self._eval_template,\n            refine_template=self._refine_template,\n        )\n        response_obj = await query_engine.aquery(response)\n\n        raw_response_txt = str(response_obj)\n\n        if \"yes\" in raw_response_txt.lower():\n            passing = True\n        else:\n            passing = False\n            if self._raise_error:","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/faithfulness.py#L155-L191","documentation":"Thrown by FaithfulnessEvaluator.aevaluate when contexts or response is None. The evaluator builds a SummaryIndex over the context strings and queries it with the response text, so both must be present to judge faithfulness.","triggerScenarios":"Calling await evaluator.aevaluate(response=r, contexts=None) or aevaluate(response=None, contexts=ctxs); via BatchRunner when the contexts kwarg list was omitted; passing response.source_nodes of an empty/failed retrieval as None.","commonSituations":"Forgetting to pass the contexts kwarg to aevaluate/aevaluate_responses (query alone is not enough for faithfulness); a query engine exception path returning None response; converting source_nodes to texts with a helper that returns None on empty.","solutions":["Pass both: await evaluator.aevaluate(response=r, contexts=[c1, c2, ...]).","With BatchRunner, include a contexts list aligned with queries/responses.","Guard upstream: skip evaluation when a response or its retrieved nodes are missing rather than forwarding None."],"exampleFix":"# before\nresult = await evaluator.aevaluate(response=str(resp))\n\n# after\ncontexts = [n.get_content() for n in resp.source_nodes]\nif not contexts:\n    raise/skip  # nothing to verify faithfulness against\nresult = await evaluator.aevaluate(response=str(resp), contexts=contexts)","handlingStrategy":"validation","validationCode":"contexts = [n.get_content() for n in response.source_nodes]\nif response is None or not contexts:\n    raise ValueError(\"cannot evaluate faithfulness without response and contexts\")","typeGuard":"def faithfulness_evaluable(response, contexts) -> bool:\n    return response is not None and contexts is not None and len(contexts) > 0","tryCatchPattern":null,"preventionTips":["Always extract contexts from response.source_nodes and pass them with the response.","Pass a contexts kwarg list in BatchRunner runs — queries alone are not enough for faithfulness."],"tags":["evaluation","faithfulness","input-validation"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}