run-llama/llama_index · error · ValueError

Impossible score results. Total amount of votes is 2.

Error message

Impossible score results. Total amount of votes is 2.

What it means

In PairwiseComparisonEvaluator with enforce_consensus=True, the query is judged twice (original and flipped answer order). Each judge yields a score in {0,1}, so votes_1 + votes_2 must equal exactly 2; a float discrepancy (or a None/None score path leaving both vote counts at their defaults in a way that doesn't sum to 2) raises this ValueError as a sanity check on impossible vote totals.

Source

Thrown at llama-index-core/llama_index/core/evaluation/pairwise.py:204

            flipped_eval_result (EvaluationResult): Result when answer_2 is shown first

        Returns:
            EvaluationResult: The final evaluation result

        """
        # add pairwise_source to eval_result and flipped_eval_result
        eval_result.pairwise_source = EvaluationSource.ORIGINAL
        flipped_eval_result.pairwise_source = EvaluationSource.FLIPPED

        # count the votes for each of the 2 answers
        votes_1 = 0.0
        votes_2 = 0.0
        if eval_result.score is not None and flipped_eval_result.score is not None:
            votes_1 = eval_result.score + (1 - flipped_eval_result.score)
            votes_2 = (1 - eval_result.score) + flipped_eval_result.score

        if votes_1 + votes_2 != 2:  # each round, the judge can give a total of 1 vote
            raise ValueError("Impossible score results. Total amount of votes is 2.")

        # get the judges (original and flipped) who voted for answer_1
        voters_1 = [eval_result] * (eval_result.score == 1.0) + [
            flipped_eval_result
        ] * (flipped_eval_result.score == 0.0)

        # get the judges (original and flipped) who voted for answer_2
        voters_2 = [eval_result] * (eval_result.score == 0.0) + [
            flipped_eval_result
        ] * (flipped_eval_result.score == 1.0)

        if votes_1 > votes_2:
            return voters_1[0]  # return any voter for answer_1
        elif votes_2 > votes_1:
            return voters_2[0]  # return any vote for answer_2
        else:
            if (
                eval_result.score == 0.5

View on GitHub (pinned to afd0fef371)

Solutions

1. Disable consensus enforcement: PairwiseComparisonEvaluator(..., enforce_consensus=False)
2. Use a judge/prompt that yields strictly binary verdicts (0 or 1) so the vote invariant holds
3. Pin the judge's temperature to 0 and verify eval_result.score values are exactly 0.0 or 1.0 before consensus aggregation

Example fix

# before
evaluator = PairwiseComparisonEvaluator(llm=judge_llm, enforce_consensus=True)
result = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)

# after
evaluator = PairwiseComparisonEvaluator(llm=judge_llm, enforce_consensus=False)
result = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)

BINARY = {0.0, 1.0}
if enforce_consensus and eval_result.score in BINARY and flipped.score in BINARY:
    result = combine(eval_result, flipped)
else:
    result = eval_result  # skip consensus arithmetic on non-binary scores

try:
    result = await evaluator.aevaluate(...)
except ValueError as e:
    if "Impossible score results" in str(e):
        logger.warning("judge produced non-binary scores; rerunning without consensus")
        evaluator._enforce_consensus = False
        result = await evaluator.aevaluate(...)
    else:
        raise

When it happens

Trigger: enable_enforce_consensus=True and judge scores that are not exact 0.0/1.0 (e.g. 0.7) so votes_1 + votes_2 != 2.0 due to float arithmetic; judge returning fractional scores; float precision drift making the strict != comparison true.

Common situations: Using a judge LLM whose output parser maps to fractional scores instead of binary; models that answer with graded preferences; the strict equality check tripping on floating-point representation of sums like 0.1+0.9.

Related errors

• The response is invalid
• query, response, second_response, and reference must be prov
• The response is invalid
• query and response must be provided
• The response is invalid

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/a067f4e0b43aa2bd. Report an issue: GitHub.