microsoft/autogen · error · ValueError

Error in coding document with OpenAI

Error message

Error in coding document with OpenAI

What it means

ValueError raised inside agbench's OAIQualitativeCoder.code_document when the OpenAI structured-output completion comes back without usable codes — either the message was a refusal (printed first via message.refusal) or message.parsed.code_list was empty. The coder deliberately fails instead of returning an empty coding result.

Source

Thrown at python/packages/agbench/src/agbench/linter/coders/oai_coder.py:146

* muddled-task-execution -- unclear what kind of tasks were muddled
* task-completion-gaps -- too high level
The above names are too high level and unclear. Please DO NOT use such names.
    """,
                    },
                    {
                        "role": "user",
                        "content": doc.text,
                    },
                ],
                response_format=CodeList,
            )

            message = completion.choices[0].message
            if message.parsed and len(message.parsed.code_list) > 0:
                coded_document = CodedDocument(doc=doc, codes=set(message.parsed.code_list))
            else:
                print(message.refusal)
                raise ValueError("Error in coding document with OpenAI")
        else:
            code_to_str = "\n".join(
                [
                    (
                        f"\n---\nCode Name: {code.name}\n"
                        f"Definition: {code.definition}\n"
                        f"Examples: {code.examples}\n---\n"
                    )
                    for code in code_set
                ]
            )

            completion = self.client.beta.chat.completions.parse(
                model=self.model,
                messages=[
                    {
                        "role": "system",
                        "content": """You are an expert qualitative researcher.

View on GitHub (pinned to 027ecf0a37)

Solutions

1. Look at the printed refusal message — it states why OpenAI declined; adjust or truncate the offending log content if it contains refused material.
2. Retry the call: refusals and empty parses are often transient; the coder is a single completion, so re-running code_command may succeed.
3. Pin compatible openai SDK / model versions and verify response_format=CodeList still parses (non-null message.parsed) on your SDK version.
4. Ensure OPENAI_API_KEY is set and valid so the completion is real rather than an error path.

Example fix

# before
completion = client.chat.completions.parse(model=..., messages=[...], response_format=CodeList)
if message.parsed and len(message.parsed.code_list) > 0: ...
else: raise ValueError("Error in coding document with OpenAI")

# after (retry with backoff on refusal/empty)
for attempt in range(3):
    completion = client.chat.completions.parse(model=..., messages=[...], response_format=CodeList)
    parsed = completion.choices[0].message.parsed
    if parsed and parsed.code_list:
        return CodedDocument(doc=doc, codes=set(parsed.code_list))
    time.sleep(2 ** attempt)
raise ValueError("OpenAI returned no codes after retries")

# Sanity-check the input before paying for a completion
if not doc.text or len(doc.text.strip()) < 20:
    raise ValueError("Document too short to code reliably")

for attempt in range(3):
    try:
        return coder.code_document(doc)
    except ValueError as e:
        if "OpenAI" not in str(e):
            raise
        time.sleep(2 ** attempt)  # backoff and retry refusals/empty parses
raise ValueError("OpenAI returned no codes after retries")

When it happens

Trigger: Calling code_document on a log whose content triggers the model's refusal behavior (embedded unsafe-looking content from benchmark transcripts); the model returning an empty code list for input it considers irrelevant; a response_format/CodeList schema mismatch so message.parsed is null.

Common situations: Benchmark logs containing adversarial prompts or tool output that safety filters reject; very short or non-log text the model declines to code; OpenAI SDK or model version changes altering structured-output behavior; missing/invalid OPENAI_API_KEY leading to degenerate responses.

Related errors

• Failed to fetch galleries
• Failed to fetch gallery
• Failed to code the document.
• Unknown tool '{name}'. Please choose from: {tool_names}
• Failed to get login URL

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/a0c2918c10319863. Report an issue: GitHub.