aaif-goose/goose · error · ValueError

Failed to parse OpenAI evaluation response after {max_retrie

Error message

Failed to parse OpenAI evaluation response after {max_retries} attempts: {str(e)}

What it means

Raised by evaluate_with_openai when the judge's chat response failed to parse as JSON (or its 'score' could not be coerced to float) on max_retries consecutive attempts. Each attempt re-asks the model, sleeps 1 second between tries, and prints the offending response text; only when the retry budget is exhausted does it raise. The usual root cause is the model wrapping the JSON in markdown fences or adding prose despite the output instructions embedded in the system prompt.

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/llm_judge.py:104

                        ],
                        temperature=0.9
                    )
                    
                    # Extract and parse JSON from response
                    response_text = response.choices[0].message.content.strip()
                    try:
                        evaluation = json.loads(response_text)
                        score = float(evaluation.get("score", 0.0))
                        score = max(0.0, min(score, rubric_max_score))
                        scores.append(score)
                        print(f"Run {i+1} score: {score}")
                        break  # Successfully parsed, exit retry loop
                    except (json.JSONDecodeError, ValueError) as e:
                        retry_count += 1
                        print(f"Error parsing OpenAI response as JSON (attempt {retry_count}/{max_retries}): {str(e)}")
                        print(f"Response text: {response_text}")
                        if retry_count == max_retries:
                            raise ValueError(f"Failed to parse OpenAI evaluation response after {max_retries} attempts: {str(e)}")
                        print("Retrying...")
                        time.sleep(1)  # Wait 1 second before retrying
                        continue
                except Exception as e:
                    # For other exceptions (API errors, etc.), raise immediately
                    print(f"API error: {str(e)}")
                    raise
        
        # Count occurrences of each score
        score_counts = Counter(scores)
        
        # If there's no single most common score (all scores are different), run one more time
        if len(scores) == 3 and max(score_counts.values()) == 1:
            print("No majority score found. Running tie-breaker...")
            max_retries = 5
            retry_count = 0
            
            while retry_count < max_retries:

View on GitHub (pinned to 3810898a74)

Solutions

Strip markdown fences before parsing: remove leading ```/```json and trailing ``` lines from response_text
Request structured output from the API: pass response_format={'type': 'json_object'} when creating the completion
Lower temperature (e.g. 0) and move the output instructions to the end of the prompt so they dominate
Raise max_retries for flaky models; note the retry loop sleeps only 1s, so keep budgets modest

Example fix

# before
evaluation = json.loads(response_text)
score = float(evaluation.get('score', 0.0))

# after
stripped = re.sub(r'^```(?:json)?\s*|\s*```$', '', response_text.strip(), flags=re.MULTILINE)
evaluation = json.loads(stripped)
score = float(evaluation['score'])  # KeyError/ValueError still feeds the retry loop

Defensive patterns

Strategy: retry

Validate before calling

import json, re

def parse_judge_response(text: str) -> float:
    stripped = re.sub(r'^```(?:json)?\s*|\s*```$', '', text.strip(), flags=re.MULTILINE)
    evaluation = json.loads(stripped)
    return float(evaluation['score'])  # raises before the retry budget is wasted

Try / catch

for attempt in range(max_retries):
    try:
        score = parse_judge_response(response_text)
        break
    except (json.JSONDecodeError, ValueError, KeyError):
        if attempt == max_retries - 1:
            raise ValueError(f'Failed to parse OpenAI evaluation response after {max_retries} attempts')
        time.sleep(2 ** attempt)

Prevention

Pass response_format={'type': 'json_object'} on the completion call
Strip markdown fences before json.loads — it is the most common single failure
Run with temperature 0 and keep the output-instructions block at the end of the prompt
Score must be a bare numeric field; validate its type before float()

When it happens

Trigger: The model replies with ```json ... ``` fences around the object; the model returns prose ('The score is 2') instead of JSON; the 'score' field is a non-numeric string so float() raises ValueError inside the same try; a slower model consistently ignoring the JSON format instruction across all retries.

Common situations: Judges run with models that are loose about output formats; temperature settings that encourage chatty answers; prompts where the rubric wording drowns out the output-instructions block; API changes returning error text in message content that then fails json.loads.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/1ccbc7230189ee09. Report an issue: GitHub.