BoundaryML/baml · error · anyhow::Error

Video input is only supported on OpenAI's Realtime API (/v1/

Error message

Video input is only supported on OpenAI's Realtime API (/v1/realtime), not on chat completions. Consider extracting frames from the video as images instead. See: https://platform.openai.com/docs/guides/realtime

What it means

The OpenAI chat completions endpoint does not accept video input; BAML enforces this by bailing with a pointer to OpenAI's Realtime API, which is the only OpenAI surface that supports video. It suggests extracting video frames as images as a workaround.

Source

Thrown at engine/baml-runtime/src/internal/llm_client/primitive/openai/openai_client.rs:894

                        content.insert(
                            payload_key.into(),
                            json!({
                                "filename": "document.pdf",
                                "file_data": format!("data:{};base64,{}", media.mime_type_as_ok()?, b64_media.base64)
                            }),
                        );
                    }
                    BamlMediaContent::File(media_file) => {
                        // For files, we need to resolve them to base64 first
                        anyhow::bail!(
                            "BAML internal error (openai): Pdf file should have been resolved to base64 before this stage."
                        );
                    }
                }
            }
            BamlMediaType::Video => {
                // OpenAI video is only supported on the Realtime API (/v1/realtime), not on chat completions
                anyhow::bail!(
                    "Video input is only supported on OpenAI's Realtime API (/v1/realtime), not on chat completions. \
                    Consider extracting frames from the video as images instead. \
                    See: https://platform.openai.com/docs/guides/realtime"
                );
            }
        }
        Ok(content)
    }

    fn role_to_message(
        &self,
        content: &RenderedChatMessage,
    ) -> Result<serde_json::Map<String, serde_json::Value>> {
        let mut message = serde_json::Map::new();
        message.insert("role".into(), json!(content.role));

        let strategy = self.get_provider_strategy();
        let formatted_content =

View on GitHub (pinned to bd85ce9dee)

Solutions

Extract representative frames from the video and send them as image media instead
Route video to OpenAI's Realtime API (/v1/realtime) if live video is required
Use a provider that supports video input in chat requests (e.g. Gemini) in your BAML client config
Sample the video with ffmpeg and attach frames in the prompt

Example fix

// before (shell + baml)
ffmpeg -i video.mp4 frame_%03d.jpg
// after (baml)
image { url "./frame_001.jpg" } image { url "./frame_002.jpg" }

Defensive patterns

Strategy: validation

Validate before calling

// block video media for openai chat-completions clients
if (media.type === "video" && provider === "openai" && api === "chat-completions") {
  throw new Error("Video not supported on chat completions; extract frames or use Realtime API");
}

Type guard

function videoAllowed(provider: string, api: string): boolean {
  return provider === "openai" && api === "realtime";
}

Try / catch

// try video prompt, fall back to frame-extracted images
try {
  return await fn.run({ video_url });
} catch (e) {
  if (String(e).includes("Video input is only supported")) {
    const frames = extractFrames(video_url);
    return await fn.run({ frames });
  }
  throw e;
}

Prevention

Extract video frames with ffmpeg before prompts when using OpenAI chat models
Configure a Gemini/other client for video-capable prompts
Document provider media capabilities in your prompt test suite

When it happens

Trigger: A ChatMessagePart with BamlMediaType::Video reaches to_media_message for the OpenAI chat completions converter (any video media in a prompt sent to OpenAI chat completions).

Common situations: Prompts that pass video files/URLs to gpt-4o-class models via chat completions, after seeing video support advertised for other providers or the Realtime API.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/4775b0f8f8c271c9. Report an issue: GitHub.