run-llama/llama_index · error · ValueError

query, response, second_response, and reference must be prov

Error message

query, response, second_response, and reference must be provided

What it means

PairwiseComparisonEvaluator.aevaluate requires query, response, and second_response (the message text omits that reference is actually optional in this check — the guard tests only the first three). contexts is accepted but explicitly deleted as unused. None among those three triggers the ValueError before any judging occurs.

Source

Thrown at llama-index-core/llama_index/core/evaluation/pairwise.py:250

                    feedback="",
                    pairwise_source=EvaluationSource.NEITHER,
                )

    async def aevaluate(
        self,
        query: Optional[str] = None,
        response: Optional[str] = None,
        contexts: Optional[Sequence[str]] = None,
        second_response: Optional[str] = None,
        reference: Optional[str] = None,
        sleep_time_in_seconds: int = 0,
        **kwargs: Any,
    ) -> EvaluationResult:
        del kwargs  # Unused
        del contexts  # Unused

        if query is None or response is None or second_response is None:
            raise ValueError(
                "query, response, second_response, and reference must be provided"
            )

        await asyncio.sleep(sleep_time_in_seconds)

        eval_result = await self._get_eval_result(
            query, response, second_response, reference
        )
        if self._enforce_consensus and not eval_result.invalid_result:
            # Flip the order of the answers and see if the answer is consistent
            # (which means that the score should flip from 0 to 1 and vice-versa)
            # if not, then we return a tie
            flipped_eval_result = await self._get_eval_result(
                query, second_response, response, reference
            )
            if not flipped_eval_result.invalid_result:
                resolved_eval_result = await self._resolve_results(
                    eval_result, flipped_eval_result

View on GitHub (pinned to afd0fef371)

Solutions

1. Provide non-None query, response, and second_response to aevaluate
2. Skip comparisons where either candidate answer is missing rather than passing None
3. Remember contexts is ignored by this evaluator — don't rely on it; reference may be None safely

Example fix

# before
result = await evaluator.aevaluate(
    query=q, response=a1, second_response=None, reference=ref
)

# after
if a1 is None or a2 is None:
    skip("missing candidate answer")
result = await evaluator.aevaluate(
    query=q, response=a1, second_response=a2, reference=ref
)

def pairwise_args_ok(query, response, second_response) -> bool:
    return all(x is not None for x in (query, response, second_response))

if pairwise_args_ok(q, a1, a2):
    result = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)

try:
    result = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)
except ValueError as e:
    if "must be provided" in str(e):
        skip("incomplete pairwise sample")
    else:
        raise

When it happens

Trigger: Awaiting aevaluate with second_response=None (e.g. only one candidate answer available), or query/response None from a failed upstream step; note reference=None does NOT trigger this error despite the message wording.

Common situations: Comparing answers where one pipeline produced no output; looping over pairs where a candidate is missing; assuming reference is required because the error text lists it.

Related errors

• query, contexts, and response must be provided
• query, contexts, and response must be provided
• names and results_arr must have same length.
• Retrieved ids and expected ids must be provided
• query and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/f94edc4b415617ba. Report an issue: GitHub.