BoundaryML/baml · error · anyhow::Error

BAML internal error (openai-responses): audio must be base64

Error message

BAML internal error (openai-responses): audio must be base64 encoded for Responses API

What it means

The OpenAI Responses API input_audio content part only accepts base64-encoded audio data with a format string. When an audio media part carried a URL or file reference instead of `BamlMediaContent::Base64`, BAML bails rather than sending an unsupported shape. Remote or local file audio must be converted to base64 before it can be sent through the Responses API.

Source

Thrown at engine/baml-runtime/src/internal/llm_client/primitive/openai/openai_client.rs:173

                        "type": "input_image",
                        "detail": "auto",
                        "image_url": image_url
                    }))
                }
                baml_types::BamlMediaType::Audio => match &media.content {
                    baml_types::BamlMediaContent::Base64(b64_media) => {
                        let mime_type = media.mime_type_as_ok()?;
                        let format = mime_type.strip_prefix("audio/").unwrap_or(&mime_type);
                        Ok(json!({
                            "type": "input_audio",
                            "input_audio": {
                                "data": b64_media.base64,
                                "format": format
                            }
                        }))
                    }
                    _ => {
                        anyhow::bail!(
                            "BAML internal error (openai-responses): audio must be base64 encoded for Responses API"
                        );
                    }
                },
                baml_types::BamlMediaType::Pdf => match &media.content {
                    baml_types::BamlMediaContent::Url(url_content) => Ok(json!({
                        "type": "input_file",
                        "file_url": url_content.url,
                        "filename": "document.pdf"
                    })),
                    baml_types::BamlMediaContent::File(file_content) => {
                        anyhow::bail!(
                            "BAML internal error (openai-responses): Local PDF files are not supported by OpenAI Responses API - use file_url for remote files or upload file and use file_id. File path: {:?}",
                            file_content.relpath
                        );
                    }
                    baml_types::BamlMediaContent::Base64(b64_media) => Ok(json!({
                        "type": "input_file",

View on GitHub (pinned to bd85ce9dee)

Solutions

  1. Provide the audio as base64 (`audio { b64 ... }`) so the Responses API path can emit `input_audio`.
  2. If the audio lives at a URL, download it and convert to base64 before passing it to the function.
  3. If it is a local file, read and base64-encode it in your client code or use BAML's file-loading/media resolution to inline it.
  4. Alternatively, switch the client to the standard OpenAI Chat Completions strategy if you need URL-based audio.

Example fix

// before: URL audio on a Responses API client
audio { url "https://example.com/clip.mp3" }

// after: base64 audio
audio { b64 "SUQzBAAAAAAA..." } // with mime_type audio/mp3
Defensive patterns

Strategy: validation

Validate before calling

if let BamlMediaContent::Url(_) | BamlMediaContent::File(_) = audio.content {
    // pre-fetch and base64-encode before sending
    let bytes = fetch_bytes(&audio)?;
    audio = to_base64_media(bytes, &audio.mime_type);
}

Type guard

fn audio_is_base64(media: &BamlMedia) -> bool {
    matches!(media.content, BamlMediaContent::Base64(_))
}

Try / catch

match result {
    Err(e) if e.to_string().contains("audio must be base64 encoded") => {
        // download/encode the audio to base64 and retry
    }
    r => r?,
}

Prevention

When it happens

Trigger: Sending audio to a function whose client uses the OpenAI Responses API with the audio provided as a URL (`BamlMediaContent::Url`) or as a local file (`BamlMediaContent::File`); only the Base64 variant is accepted (openai_client.rs:172-176, the `_` arm of the Audio match).

Common situations: Pointing `audio { url ... }` at a hosted recording while using a Responses-API model (e.g. gpt-4o-audio via Responses); passing a local audio file path; reusing prompt media config written for another provider that accepts URLs.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/3db02316ba55555f. Report an issue: GitHub.