{"record":{"id":"f94edc4b415617ba","repo":"run-llama/llama_index","slug":"query-response-second-response-and-reference-mu","errorCode":null,"errorMessage":"query, response, second_response, and reference must be provided","messagePattern":"query, response, second_response, and reference must be provided","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/pairwise.py","lineNumber":250,"sourceCode":"                    feedback=\"\",\n                    pairwise_source=EvaluationSource.NEITHER,\n                )\n\n    async def aevaluate(\n        self,\n        query: Optional[str] = None,\n        response: Optional[str] = None,\n        contexts: Optional[Sequence[str]] = None,\n        second_response: Optional[str] = None,\n        reference: Optional[str] = None,\n        sleep_time_in_seconds: int = 0,\n        **kwargs: Any,\n    ) -> EvaluationResult:\n        del kwargs  # Unused\n        del contexts  # Unused\n\n        if query is None or response is None or second_response is None:\n            raise ValueError(\n                \"query, response, second_response, and reference must be provided\"\n            )\n\n        await asyncio.sleep(sleep_time_in_seconds)\n\n        eval_result = await self._get_eval_result(\n            query, response, second_response, reference\n        )\n        if self._enforce_consensus and not eval_result.invalid_result:\n            # Flip the order of the answers and see if the answer is consistent\n            # (which means that the score should flip from 0 to 1 and vice-versa)\n            # if not, then we return a tie\n            flipped_eval_result = await self._get_eval_result(\n                query, second_response, response, reference\n            )\n            if not flipped_eval_result.invalid_result:\n                resolved_eval_result = await self._resolve_results(\n                    eval_result, flipped_eval_result","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/pairwise.py#L232-L268","documentation":"PairwiseComparisonEvaluator.aevaluate requires query, response, and second_response (the message text omits that reference is actually optional in this check — the guard tests only the first three). contexts is accepted but explicitly deleted as unused. None among those three triggers the ValueError before any judging occurs.","triggerScenarios":"Awaiting aevaluate with second_response=None (e.g. only one candidate answer available), or query/response None from a failed upstream step; note reference=None does NOT trigger this error despite the message wording.","commonSituations":"Comparing answers where one pipeline produced no output; looping over pairs where a candidate is missing; assuming reference is required because the error text lists it.","solutions":["Provide non-None query, response, and second_response to aevaluate","Skip comparisons where either candidate answer is missing rather than passing None","Remember contexts is ignored by this evaluator — don't rely on it; reference may be None safely"],"exampleFix":"# before\nresult = await evaluator.aevaluate(\n    query=q, response=a1, second_response=None, reference=ref\n)\n\n# after\nif a1 is None or a2 is None:\n    skip(\"missing candidate answer\")\nresult = await evaluator.aevaluate(\n    query=q, response=a1, second_response=a2, reference=ref\n)","handlingStrategy":"validation","validationCode":"def pairwise_args_ok(query, response, second_response) -> bool:\n    return all(x is not None for x in (query, response, second_response))\n\nif pairwise_args_ok(q, a1, a2):\n    result = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)","typeGuard":null,"tryCatchPattern":"try:\n    result = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)\nexcept ValueError as e:\n    if \"must be provided\" in str(e):\n        skip(\"incomplete pairwise sample\")\n    else:\n        raise","preventionTips":["Ensure both candidate answers exist before scheduling pairwise comparisons","Remember reference is optional and contexts is ignored for this evaluator","Pair samples defensively: zip candidates only when both are present"],"tags":["validation","evaluation","pairwise","async","required-args"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}