{"record":{"id":"a0c2918c10319863","repo":"microsoft/autogen","slug":"error-in-coding-document-with-openai","errorCode":null,"errorMessage":"Error in coding document with OpenAI","messagePattern":"Error in coding document with OpenAI","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/packages/agbench/src/agbench/linter/coders/oai_coder.py","lineNumber":146,"sourceCode":"* muddled-task-execution -- unclear what kind of tasks were muddled\n* task-completion-gaps -- too high level\nThe above names are too high level and unclear. Please DO NOT use such names.\n    \"\"\",\n                    },\n                    {\n                        \"role\": \"user\",\n                        \"content\": doc.text,\n                    },\n                ],\n                response_format=CodeList,\n            )\n\n            message = completion.choices[0].message\n            if message.parsed and len(message.parsed.code_list) > 0:\n                coded_document = CodedDocument(doc=doc, codes=set(message.parsed.code_list))\n            else:\n                print(message.refusal)\n                raise ValueError(\"Error in coding document with OpenAI\")\n        else:\n            code_to_str = \"\\n\".join(\n                [\n                    (\n                        f\"\\n---\\nCode Name: {code.name}\\n\"\n                        f\"Definition: {code.definition}\\n\"\n                        f\"Examples: {code.examples}\\n---\\n\"\n                    )\n                    for code in code_set\n                ]\n            )\n\n            completion = self.client.beta.chat.completions.parse(\n                model=self.model,\n                messages=[\n                    {\n                        \"role\": \"system\",\n                        \"content\": \"\"\"You are an expert qualitative researcher.","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/microsoft/autogen/blob/027ecf0a379bcc1d09956d46d12d44a3ad9cee14/python/packages/agbench/src/agbench/linter/coders/oai_coder.py#L128-L164","documentation":"ValueError raised inside agbench's OAIQualitativeCoder.code_document when the OpenAI structured-output completion comes back without usable codes — either the message was a refusal (printed first via message.refusal) or message.parsed.code_list was empty. The coder deliberately fails instead of returning an empty coding result.","triggerScenarios":"Calling code_document on a log whose content triggers the model's refusal behavior (embedded unsafe-looking content from benchmark transcripts); the model returning an empty code list for input it considers irrelevant; a response_format/CodeList schema mismatch so message.parsed is null.","commonSituations":"Benchmark logs containing adversarial prompts or tool output that safety filters reject; very short or non-log text the model declines to code; OpenAI SDK or model version changes altering structured-output behavior; missing/invalid OPENAI_API_KEY leading to degenerate responses.","solutions":["Look at the printed refusal message — it states why OpenAI declined; adjust or truncate the offending log content if it contains refused material.","Retry the call: refusals and empty parses are often transient; the coder is a single completion, so re-running code_command may succeed.","Pin compatible openai SDK / model versions and verify response_format=CodeList still parses (non-null message.parsed) on your SDK version.","Ensure OPENAI_API_KEY is set and valid so the completion is real rather than an error path."],"exampleFix":"# before\ncompletion = client.chat.completions.parse(model=..., messages=[...], response_format=CodeList)\nif message.parsed and len(message.parsed.code_list) > 0: ...\nelse: raise ValueError(\"Error in coding document with OpenAI\")\n\n# after (retry with backoff on refusal/empty)\nfor attempt in range(3):\n    completion = client.chat.completions.parse(model=..., messages=[...], response_format=CodeList)\n    parsed = completion.choices[0].message.parsed\n    if parsed and parsed.code_list:\n        return CodedDocument(doc=doc, codes=set(parsed.code_list))\n    time.sleep(2 ** attempt)\nraise ValueError(\"OpenAI returned no codes after retries\")","handlingStrategy":"retry","validationCode":"# Sanity-check the input before paying for a completion\nif not doc.text or len(doc.text.strip()) < 20:\n    raise ValueError(\"Document too short to code reliably\")","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        return coder.code_document(doc)\n    except ValueError as e:\n        if \"OpenAI\" not in str(e):\n            raise\n        time.sleep(2 ** attempt)  # backoff and retry refusals/empty parses\nraise ValueError(\"OpenAI returned no codes after retries\")","preventionTips":["Print and read message.refusal before retrying so real content problems are fixed, not retried.","Set and verify OPENAI_API_KEY; confirm the openai SDK version supports response_format with your schema.","Wrap code_document in a bounded retry with backoff for transient refusals and empty parses."],"tags":["python","openai","llm","retry"],"backgroundTag":null,"analyzedSha":"027ecf0a379bcc1d09956d46d12d44a3ad9cee14","analyzedAt":"2026-08-15T03:38:00.719Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}