{"record":{"id":"ddc90dbfaa72bb9e","repo":"aaif-goose/goose","slug":"failed-to-parse-tie-breaker-response-after-max-re","errorCode":null,"errorMessage":"Failed to parse tie-breaker response after {max_retries} attempts: {str(e)}","messagePattern":"Failed to parse tie-breaker response after (.+?) attempts: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/bench-postprocess-scripts/llm-judges/llm_judge.py","lineNumber":146,"sourceCode":"                        ],\n                        temperature=0.9\n                    )\n                    \n                    response_text = response.choices[0].message.content.strip()\n                    try:\n                        evaluation = json.loads(response_text)\n                        score = float(evaluation.get(\"score\", 0.0))\n                        score = max(0.0, min(score, rubric_max_score))\n                        scores.append(score)\n                        print(f\"Tie-breaker score: {score}\")\n                        score_counts = Counter(scores)\n                        break  # Successfully parsed, exit retry loop\n                    except (json.JSONDecodeError, ValueError) as e:\n                        retry_count += 1\n                        print(f\"Error parsing tie-breaker response as JSON (attempt {retry_count}/{max_retries}): {str(e)}\")\n                        print(f\"Response text: {response_text}\")\n                        if retry_count == max_retries:\n                            raise ValueError(f\"Failed to parse tie-breaker response after {max_retries} attempts: {str(e)}\")\n                        print(\"Retrying tie-breaker...\")\n                        time.sleep(1)  # Wait 1 second before retrying\n                        continue\n                except Exception as e:\n                    # For other exceptions (API errors, etc.), raise immediately\n                    print(f\"API error in tie-breaker: {str(e)}\")\n                    raise\n        \n        # Get the most common score\n        most_common_score = score_counts.most_common(1)[0][0]\n        print(f\"Most common score: {most_common_score} (occurred {score_counts[most_common_score]} times)\")\n        return most_common_score\n            \n    except Exception as e:\n        if \"OPENAI_API_KEY\" in str(e):\n            raise  # Re-raise API key errors\n        print(f\"Error evaluating with OpenAI: {str(e)}\")\n        raise ValueError(f\"OpenAI evaluation failed: {str(e)}\")","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/scripts/bench-postprocess-scripts/llm-judges/llm_judge.py#L128-L164","documentation":"Raised by the tie-breaker stage of evaluate_with_openai. When the initial judge runs produce no single most-common score (e.g. all N scores differ), the script runs one extra deciding call; if that response also fails to parse as JSON (or lacks a numeric 'score') for max_retries consecutive attempts, this error aborts. Same parsing contract and retry/sleep(1) loop as error 58, just on the deciding vote.","triggerScenarios":"An even or small number of judge runs where every score is distinct, forcing the tie-breaker; the tie-breaker response arriving fenced in markdown or as prose; the 'score' field non-numeric so float() raises inside the shared try block.","commonSituations":"Low run counts (1-2 samples) making ties the norm; inconsistent models producing scattered scores; the same format-drift causes as error 58 hitting exactly when a tie-break is needed.","solutions":["Apply the same hardening as the main loop: strip markdown fences and/or use response_format={'type': 'json_object'}","Use an odd number of judge runs so a strict majority usually exists and the tie-breaker rarely triggers","Lower temperature and keep the output-instructions block verbatim in the tie-breaker prompt","Raise max_retries for models that intermittently emit prose"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import json, re\n\ndef parse_tiebreaker_response(text: str) -> float:\n    stripped = re.sub(r'^```(?:json)?\\s*|\\s*```$', '', text.strip(), flags=re.MULTILINE)\n    evaluation = json.loads(stripped)\n    return float(evaluation['score'])","typeGuard":null,"tryCatchPattern":"for attempt in range(max_retries):\n    try:\n        tie_score = parse_tiebreaker_response(response_text)\n        break\n    except (json.JSONDecodeError, ValueError, KeyError):\n        if attempt == max_retries - 1:\n            raise ValueError(f'Failed to parse tie-breaker response after {max_retries} attempts')\n        time.sleep(2 ** attempt)","preventionTips":["Use an odd number of judge runs so a strict majority usually avoids the tie-breaker entirely","Harden the tie-breaker call the same way as the main loop: json_object response format, fence stripping, temperature 0","Treat recurring tie-breaker parse failures as a prompt-format defect, not bad luck — fix the prompt's JSON instruction"],"tags":["openai","json","parsing","retry","llm-judge","tie-breaker"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}