{"record":{"id":"3bc5dccf2b15e98d","repo":"run-llama/llama_index","slug":"query-contexts-and-response-must-be-provided-3bc5dc","errorCode":null,"errorMessage":"query, contexts, and response must be provided","messagePattern":"query, contexts, and response must be provided","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/relevancy.py","lineNumber":109,"sourceCode":"        \"\"\"Update prompts.\"\"\"\n        if \"eval_template\" in prompts:\n            self._eval_template = prompts[\"eval_template\"]\n        if \"refine_template\" in prompts:\n            self._refine_template = prompts[\"refine_template\"]\n\n    async def aevaluate(\n        self,\n        query: str | None = None,\n        response: str | None = None,\n        contexts: Sequence[str] | None = None,\n        sleep_time_in_seconds: int = 0,\n        **kwargs: Any,\n    ) -> EvaluationResult:\n        \"\"\"Evaluate whether the contexts and response are relevant to the query.\"\"\"\n        del kwargs  # Unused\n\n        if query is None or contexts is None or response is None:\n            raise ValueError(\"query, contexts, and response must be provided\")\n\n        docs = [Document(text=context) for context in contexts]\n        index = SummaryIndex.from_documents(docs)\n\n        query_response = f\"Question: {query}\\nResponse: {response}\"\n\n        await asyncio.sleep(sleep_time_in_seconds)\n\n        query_engine = index.as_query_engine(\n            llm=self._llm,\n            text_qa_template=self._eval_template,\n            refine_template=self._refine_template,\n        )\n        response_obj = await query_engine.aquery(query_response)\n\n        raw_response_txt = str(response_obj)\n\n        if \"yes\" in raw_response_txt.lower():","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/relevancy.py#L91-L127","documentation":"RelevancyEvaluator.aevaluate (text-only) requires query, contexts, and response: it joins contexts into Documents, builds a SummaryIndex, and queries it with the eval template. If any of the three is None it raises ValueError immediately — the Optional annotations exist only for keyword-based calling.","triggerScenarios":"Awaiting aevaluate(query=q, contexts=None, response=r), e.g. contexts list lost during serialization or an empty retrieval step returning None; omitting any of the three kwargs.","commonSituations":"Async eval batches where retrieval returned nothing for some queries; rag pipeline returning None response on error; data loading producing None fields that flow into the evaluator.","solutions":["Ensure all three are provided and non-None before the call","Coalesce missing values intentionally (response or \"\", contexts or []) if you want degenerate evals instead of crashes","Pre-filter datasets in your eval harness: drop rows lacking query/contexts/response"],"exampleFix":"# before\nresult = await evaluator.aevaluate(query=q, contexts=ctx, response=None)\n\n# after\nif not (q and ctx and resp):\n    continue\nresult = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)","handlingStrategy":"validation","validationCode":"def eval_sample_ready(sample) -> bool:\n    return bool(sample.query and sample.contexts is not None and sample.response)\n\nready = [s for s in samples if eval_sample_ready(s)]\nresults = await asyncio.gather(*[\n    evaluator.aevaluate(query=s.query, contexts=s.contexts, response=s.response)\n    for s in ready\n])","typeGuard":null,"tryCatchPattern":"try:\n    res = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)\nexcept ValueError as e:\n    if \"must be provided\" in str(e):\n        res = None\n    else:\n        raise","preventionTips":["Reject None fields at dataset load time, not at eval time","Keep one validation helper reused across all evaluators","Track skip counts to notice data-quality regressions"],"tags":["validation","evaluation","relevancy","async","required-args"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}