{"record":{"id":"1ccbc7230189ee09","repo":"aaif-goose/goose","slug":"failed-to-parse-openai-evaluation-response-after","errorCode":null,"errorMessage":"Failed to parse OpenAI evaluation response after {max_retries} attempts: {str(e)}","messagePattern":"Failed to parse OpenAI evaluation response after (.+?) attempts: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/bench-postprocess-scripts/llm-judges/llm_judge.py","lineNumber":104,"sourceCode":"                        ],\n                        temperature=0.9\n                    )\n                    \n                    # Extract and parse JSON from response\n                    response_text = response.choices[0].message.content.strip()\n                    try:\n                        evaluation = json.loads(response_text)\n                        score = float(evaluation.get(\"score\", 0.0))\n                        score = max(0.0, min(score, rubric_max_score))\n                        scores.append(score)\n                        print(f\"Run {i+1} score: {score}\")\n                        break  # Successfully parsed, exit retry loop\n                    except (json.JSONDecodeError, ValueError) as e:\n                        retry_count += 1\n                        print(f\"Error parsing OpenAI response as JSON (attempt {retry_count}/{max_retries}): {str(e)}\")\n                        print(f\"Response text: {response_text}\")\n                        if retry_count == max_retries:\n                            raise ValueError(f\"Failed to parse OpenAI evaluation response after {max_retries} attempts: {str(e)}\")\n                        print(\"Retrying...\")\n                        time.sleep(1)  # Wait 1 second before retrying\n                        continue\n                except Exception as e:\n                    # For other exceptions (API errors, etc.), raise immediately\n                    print(f\"API error: {str(e)}\")\n                    raise\n        \n        # Count occurrences of each score\n        score_counts = Counter(scores)\n        \n        # If there's no single most common score (all scores are different), run one more time\n        if len(scores) == 3 and max(score_counts.values()) == 1:\n            print(\"No majority score found. Running tie-breaker...\")\n            max_retries = 5\n            retry_count = 0\n            \n            while retry_count < max_retries:","sourceCodeStart":86,"sourceCodeEnd":122,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/scripts/bench-postprocess-scripts/llm-judges/llm_judge.py#L86-L122","documentation":"Raised by evaluate_with_openai when the judge's chat response failed to parse as JSON (or its 'score' could not be coerced to float) on max_retries consecutive attempts. Each attempt re-asks the model, sleeps 1 second between tries, and prints the offending response text; only when the retry budget is exhausted does it raise. The usual root cause is the model wrapping the JSON in markdown fences or adding prose despite the output instructions embedded in the system prompt.","triggerScenarios":"The model replies with ```json ... ``` fences around the object; the model returns prose ('The score is 2') instead of JSON; the 'score' field is a non-numeric string so float() raises ValueError inside the same try; a slower model consistently ignoring the JSON format instruction across all retries.","commonSituations":"Judges run with models that are loose about output formats; temperature settings that encourage chatty answers; prompts where the rubric wording drowns out the output-instructions block; API changes returning error text in message content that then fails json.loads.","solutions":["Strip markdown fences before parsing: remove leading ```/```json and trailing ``` lines from response_text","Request structured output from the API: pass response_format={'type': 'json_object'} when creating the completion","Lower temperature (e.g. 0) and move the output instructions to the end of the prompt so they dominate","Raise max_retries for flaky models; note the retry loop sleeps only 1s, so keep budgets modest"],"exampleFix":"# before\nevaluation = json.loads(response_text)\nscore = float(evaluation.get('score', 0.0))\n\n# after\nstripped = re.sub(r'^```(?:json)?\\s*|\\s*```$', '', response_text.strip(), flags=re.MULTILINE)\nevaluation = json.loads(stripped)\nscore = float(evaluation['score'])  # KeyError/ValueError still feeds the retry loop","handlingStrategy":"retry","validationCode":"import json, re\n\ndef parse_judge_response(text: str) -> float:\n    stripped = re.sub(r'^```(?:json)?\\s*|\\s*```$', '', text.strip(), flags=re.MULTILINE)\n    evaluation = json.loads(stripped)\n    return float(evaluation['score'])  # raises before the retry budget is wasted","typeGuard":null,"tryCatchPattern":"for attempt in range(max_retries):\n    try:\n        score = parse_judge_response(response_text)\n        break\n    except (json.JSONDecodeError, ValueError, KeyError):\n        if attempt == max_retries - 1:\n            raise ValueError(f'Failed to parse OpenAI evaluation response after {max_retries} attempts')\n        time.sleep(2 ** attempt)","preventionTips":["Pass response_format={'type': 'json_object'} on the completion call","Strip markdown fences before json.loads — it is the most common single failure","Run with temperature 0 and keep the output-instructions block at the end of the prompt","Score must be a bare numeric field; validate its type before float()"],"tags":["openai","json","parsing","retry","llm-judge","benchmark"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}