BoundaryML/baml · error · anyhow::Error

BAML internal error (openai-responses): assistant messages m

Error message

BAML internal error (openai-responses): assistant messages must be text; media not supported for assistant in Responses API

What it means

When serializing chat messages for the OpenAI Responses API, BAML found a media (non-text) content part attached to an `assistant`-role message. The Responses API only permits text (`output_text`) content on assistant messages, so BAML bails with an internal-error marker instead of sending a request OpenAI would reject. This indicates chat history containing images/audio/PDF in an assistant turn reached the Responses API serializer.

Source

Thrown at engine/baml-runtime/src/internal/llm_client/primitive/openai/openai_client.rs:133

    role: &str,
    allowed_metadata: &AllowedRoleMetadata,
) -> Result<serde_json::Value> {
    match part {
        ChatMessagePart::Text(text) => {
            let content_type = if role == "assistant" {
                "output_text"
            } else {
                "input_text"
            };
            Ok(json!({
                "type": content_type,
                "text": text
            }))
        }
        ChatMessagePart::Media(media) => {
            // For assistant role, we only support text outputs in Responses API.
            if role == "assistant" {
                anyhow::bail!(
                    "BAML internal error (openai-responses): assistant messages must be text; media not supported for assistant in Responses API"
                );
            }
            match media.media_type {
                baml_types::BamlMediaType::Image => {
                    let image_url = match &media.content {
                        baml_types::BamlMediaContent::Url(url_content) => url_content.url.clone(),
                        baml_types::BamlMediaContent::Base64(b64_media) => {
                            format!(
                                "data:{};base64,{}",
                                media.mime_type_as_ok()?,
                                b64_media.base64
                            )
                        }
                        baml_types::BamlMediaContent::File(_) => {
                            anyhow::bail!(
                                "BAML internal error (openai-responses): image file should have been resolved, not processed directly."
                            );

View on GitHub (pinned to bd85ce9dee)

Solutions

Remove media parts from assistant-role messages in your prompt/history; keep assistant turns text-only.
Move media to user or system messages, which the Responses API accepts as input_image/input_audio/input_file.
If the media came from a stored previous response, strip non-text parts before re-sending the history.
If media-in-assistant is essential, use the standard OpenAI Chat Completions client instead of the Responses API strategy.

Example fix

// before: assistant turn containing media in chat history
{ role: "assistant", content: [media(image)] }

// after: keep assistant text-only, move media to the user turn
{ role: "assistant", content: [text("here is the analysis")] }
{ role: "user", content: [image("https://...")] }

Defensive patterns

Strategy: validation

Validate before calling

for msg in &history {
    if msg.role == "assistant"
        && msg.parts.iter().any(|p| matches!(p, ChatMessagePart::Media(_)))
    {
        panic!("assistant messages must be text-only for Responses API");
    }
}

Type guard

fn is_text_only(msg: &RenderedChatMessage) -> bool {
    msg.role != "assistant"
        || msg.content.iter().all(|p| matches!(p, ChatMessagePart::Text(_)))
}

Try / catch

match result {
    Err(e) if e.to_string().contains("assistant messages must be text") => {
        // strip media from assistant turns and retry once
    }
    r => r?,
}

Prevention

Keep media in user/system messages only; treat assistant turns as text-only in history builders.
Strip non-text parts when persisting and replaying assistant responses.
Audit few-shot examples for media in assistant roles before switching to the Responses API.

When it happens

Trigger: Calling an LLM function whose client uses the OpenAI Responses API while the chat history includes an assistant message containing media (e.g. prior response carried an image, or a multi-turn conversation had media inserted into the assistant role). Hit in `responses_content_part` when `role == "assistant"` and `ChatMessagePart::Media` is matched (openai_client.rs:133).

Common situations: Replaying conversation history where a previous model reply included generated media; constructing few-shot examples with image outputs under assistant turns; migrating a client from the standard Chat Completions API (which tolerates media in assistant messages differently) to the Responses API.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/d2acedec6b631201. Report an issue: GitHub.