langchain-ai/langchain · error · ValueError

OpenAI messages can only support text and image data. Receiv

Error message

OpenAI messages can only support text and image data. Received content block with media of type: {block['mime_type']}

What it means

After validating a VertexAI `media` block's keys, the converter checks that `mime_type` contains `"image"`. OpenAI's chat API accepts only text and image inputs, so video (`video/mp4`), audio (`audio/wav`), or application media cannot be represented and the conversion raises with the offending MIME type. This is a capability restriction of the target format, not a malformed block.

Source

Thrown at libs/core/langchain_core/messages/utils.py:1932

                        text = text["text"]
                    content.append({"type": "text", "text": text})
                # VertexAI format
                elif block.get("type") == "media":
                    if missing := [k for k in ("mime_type", "data") if k not in block]:
                        err = (
                            f"Unrecognized content block at "
                            f"messages[{i}].content[{j}] has 'type': "
                            f"'media' but does not have key(s) {missing}. Full "
                            f"content block:\n\n{block}"
                        )
                        raise ValueError(err)
                    if "image" not in block["mime_type"]:
                        err = (
                            f"OpenAI messages can only support text and image data."
                            f" Received content block with media of type:"
                            f" {block['mime_type']}"
                        )
                        raise ValueError(err)
                    b64_image = _bytes_to_b64_str(block["data"])
                    content.append(
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": (f"data:{block['mime_type']};base64,{b64_image}")
                            },
                        }
                    )
                elif (
                    block.get("type") in {"thinking", "reasoning"}
                    or pass_through_unknown_blocks
                ):
                    content.append(block)
                else:
                    err = (
                        f"Unrecognized content block at "
                        f"messages[{i}].content[{j}] does not match OpenAI, "

View on GitHub (pinned to e32fa9a52e)

Solutions

1. Filter out non-image media blocks before conversion (see defense validationCode).
2. If audio input is required, use a provider that supports it natively (e.g. Gemini) rather than converting to OpenAI format.
3. For video, extract representative frames as images and replace the media block with `image_url` blocks.
4. Set expectations: text and images only when the destination is OpenAI.

Example fix

// before
content = [{"type": "media", "mime_type": "video/mp4", "data": raw}]
convert_to_openai_messages([HumanMessage(content=content)])

// after
content = [b for b in content if not (b.get("type") == "media" and "image" not in b.get("mime_type", ""))]
convert_to_openai_messages([HumanMessage(content=content)])

def is_openai_supported_media(b: dict) -> bool:
    return not (b.get("type") == "media" and "image" not in str(b.get("mime_type", "")))

# strip unsupported modalities before converting to OpenAI format
blocks = [b for b in blocks if is_openai_supported_media(b)]

def is_image_media(b: dict) -> bool:
    return b.get("type") == "media" and "image" in str(b.get("mime_type", ""))

try:
    oai = convert_to_openai_messages(messages)
except ValueError as e:
    if "can only support text and image" in str(e):
        messages = filter_non_image_media(messages)  # your helper
        oai = convert_to_openai_messages(messages)

When it happens

Trigger: `{"type": "media", "mime_type": "video/mp4", "data": ...}` or `{"mime_type": "audio/ogg", ...}` passed to `convert_to_openai_messages`; Gemini conversations containing inline video/audio parts forwarded to an OpenAI-compatible model.

Common situations: Routing the same multimodal history to both Gemini and OpenAI models; assuming `convert_to_openai_messages` silently drops unsupported modalities (it does not, unless `pass_through_unknown_blocks` applies — and it does not apply here since the block matched the `media` branch).

Related errors

• Unrecognized content block at messages[{i}].content[{j}] has
• Unrecognized content block at messages[{i}].content[{j}] has
• Unrecognized content block at messages[{i}].content[{j}] has
• Unrecognized content block at messages[{i}].content[{j}] has
• Unrecognized content block at messages[{i}].content[{j}] has

AI-assisted analysis of langchain-ai/langchain@e32fa9a52e (2026-08-14). Data as JSON: /api/errors/3d358ab6b8e178eb. Report an issue: GitHub.