aaif-goose/goose · error · ValueError

Failed to parse tie-breaker response after {max_retries} att

Error message

Failed to parse tie-breaker response after {max_retries} attempts: {str(e)}

What it means

Raised by the tie-breaker stage of evaluate_with_openai. When the initial judge runs produce no single most-common score (e.g. all N scores differ), the script runs one extra deciding call; if that response also fails to parse as JSON (or lacks a numeric 'score') for max_retries consecutive attempts, this error aborts. Same parsing contract and retry/sleep(1) loop as error 58, just on the deciding vote.

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/llm_judge.py:146

                        ],
                        temperature=0.9
                    )
                    
                    response_text = response.choices[0].message.content.strip()
                    try:
                        evaluation = json.loads(response_text)
                        score = float(evaluation.get("score", 0.0))
                        score = max(0.0, min(score, rubric_max_score))
                        scores.append(score)
                        print(f"Tie-breaker score: {score}")
                        score_counts = Counter(scores)
                        break  # Successfully parsed, exit retry loop
                    except (json.JSONDecodeError, ValueError) as e:
                        retry_count += 1
                        print(f"Error parsing tie-breaker response as JSON (attempt {retry_count}/{max_retries}): {str(e)}")
                        print(f"Response text: {response_text}")
                        if retry_count == max_retries:
                            raise ValueError(f"Failed to parse tie-breaker response after {max_retries} attempts: {str(e)}")
                        print("Retrying tie-breaker...")
                        time.sleep(1)  # Wait 1 second before retrying
                        continue
                except Exception as e:
                    # For other exceptions (API errors, etc.), raise immediately
                    print(f"API error in tie-breaker: {str(e)}")
                    raise
        
        # Get the most common score
        most_common_score = score_counts.most_common(1)[0][0]
        print(f"Most common score: {most_common_score} (occurred {score_counts[most_common_score]} times)")
        return most_common_score
            
    except Exception as e:
        if "OPENAI_API_KEY" in str(e):
            raise  # Re-raise API key errors
        print(f"Error evaluating with OpenAI: {str(e)}")
        raise ValueError(f"OpenAI evaluation failed: {str(e)}")

View on GitHub (pinned to 3810898a74)

Solutions

Apply the same hardening as the main loop: strip markdown fences and/or use response_format={'type': 'json_object'}
Use an odd number of judge runs so a strict majority usually exists and the tie-breaker rarely triggers
Lower temperature and keep the output-instructions block verbatim in the tie-breaker prompt
Raise max_retries for models that intermittently emit prose

Defensive patterns

Strategy: retry

Validate before calling

import json, re

def parse_tiebreaker_response(text: str) -> float:
    stripped = re.sub(r'^```(?:json)?\s*|\s*```$', '', text.strip(), flags=re.MULTILINE)
    evaluation = json.loads(stripped)
    return float(evaluation['score'])

Try / catch

for attempt in range(max_retries):
    try:
        tie_score = parse_tiebreaker_response(response_text)
        break
    except (json.JSONDecodeError, ValueError, KeyError):
        if attempt == max_retries - 1:
            raise ValueError(f'Failed to parse tie-breaker response after {max_retries} attempts')
        time.sleep(2 ** attempt)

Prevention

Use an odd number of judge runs so a strict majority usually avoids the tie-breaker entirely
Harden the tie-breaker call the same way as the main loop: json_object response format, fence stripping, temperature 0
Treat recurring tie-breaker parse failures as a prompt-format defect, not bad luck — fix the prompt's JSON instruction

When it happens

Trigger: An even or small number of judge runs where every score is distinct, forcing the tie-breaker; the tie-breaker response arriving fenced in markdown or as prose; the 'score' field non-numeric so float() raises inside the shared try block.

Common situations: Low run counts (1-2 samples) making ties the norm; inconsistent models producing scattered scores; the same format-drift causes as error 58 hitting exactly when a tie-break is needed.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/ddc90dbfaa72bb9e. Report an issue: GitHub.