{"record":{"id":"a067f4e0b43aa2bd","repo":"run-llama/llama_index","slug":"impossible-score-results-total-amount-of-votes-is","errorCode":null,"errorMessage":"Impossible score results. Total amount of votes is 2.","messagePattern":"Impossible score results\\. Total amount of votes is 2\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/pairwise.py","lineNumber":204,"sourceCode":"            flipped_eval_result (EvaluationResult): Result when answer_2 is shown first\n\n        Returns:\n            EvaluationResult: The final evaluation result\n\n        \"\"\"\n        # add pairwise_source to eval_result and flipped_eval_result\n        eval_result.pairwise_source = EvaluationSource.ORIGINAL\n        flipped_eval_result.pairwise_source = EvaluationSource.FLIPPED\n\n        # count the votes for each of the 2 answers\n        votes_1 = 0.0\n        votes_2 = 0.0\n        if eval_result.score is not None and flipped_eval_result.score is not None:\n            votes_1 = eval_result.score + (1 - flipped_eval_result.score)\n            votes_2 = (1 - eval_result.score) + flipped_eval_result.score\n\n        if votes_1 + votes_2 != 2:  # each round, the judge can give a total of 1 vote\n            raise ValueError(\"Impossible score results. Total amount of votes is 2.\")\n\n        # get the judges (original and flipped) who voted for answer_1\n        voters_1 = [eval_result] * (eval_result.score == 1.0) + [\n            flipped_eval_result\n        ] * (flipped_eval_result.score == 0.0)\n\n        # get the judges (original and flipped) who voted for answer_2\n        voters_2 = [eval_result] * (eval_result.score == 0.0) + [\n            flipped_eval_result\n        ] * (flipped_eval_result.score == 1.0)\n\n        if votes_1 > votes_2:\n            return voters_1[0]  # return any voter for answer_1\n        elif votes_2 > votes_1:\n            return voters_2[0]  # return any vote for answer_2\n        else:\n            if (\n                eval_result.score == 0.5","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/pairwise.py#L186-L222","documentation":"In PairwiseComparisonEvaluator with enforce_consensus=True, the query is judged twice (original and flipped answer order). Each judge yields a score in {0,1}, so votes_1 + votes_2 must equal exactly 2; a float discrepancy (or a None/None score path leaving both vote counts at their defaults in a way that doesn't sum to 2) raises this ValueError as a sanity check on impossible vote totals.","triggerScenarios":"enable_enforce_consensus=True and judge scores that are not exact 0.0/1.0 (e.g. 0.7) so votes_1 + votes_2 != 2.0 due to float arithmetic; judge returning fractional scores; float precision drift making the strict != comparison true.","commonSituations":"Using a judge LLM whose output parser maps to fractional scores instead of binary; models that answer with graded preferences; the strict equality check tripping on floating-point representation of sums like 0.1+0.9.","solutions":["Disable consensus enforcement: PairwiseComparisonEvaluator(..., enforce_consensus=False)","Use a judge/prompt that yields strictly binary verdicts (0 or 1) so the vote invariant holds","Pin the judge's temperature to 0 and verify eval_result.score values are exactly 0.0 or 1.0 before consensus aggregation"],"exampleFix":"# before\nevaluator = PairwiseComparisonEvaluator(llm=judge_llm, enforce_consensus=True)\nresult = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)\n\n# after\nevaluator = PairwiseComparisonEvaluator(llm=judge_llm, enforce_consensus=False)\nresult = await evaluator.aevaluate(query=q, response=a1, second_response=a2, reference=ref)","handlingStrategy":"validation","validationCode":"BINARY = {0.0, 1.0}\nif enforce_consensus and eval_result.score in BINARY and flipped.score in BINARY:\n    result = combine(eval_result, flipped)\nelse:\n    result = eval_result  # skip consensus arithmetic on non-binary scores","typeGuard":null,"tryCatchPattern":"try:\n    result = await evaluator.aevaluate(...)\nexcept ValueError as e:\n    if \"Impossible score results\" in str(e):\n        logger.warning(\"judge produced non-binary scores; rerunning without consensus\")\n        evaluator._enforce_consensus = False\n        result = await evaluator.aevaluate(...)\n    else:\n        raise","preventionTips":["Keep enforce_consensus=False unless the judge reliably outputs binary verdicts","Log scores from both passes to detect fractional verdicts early","Use a deterministic judge (temperature 0) for pairwise comparisons"],"tags":["evaluation","pairwise","llm-verdict","consensus","float-precision"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}