microsoft/autogen · error · RuntimeError

model {model_family} does not support vision.

Error message

model {model_family} does not support vision.

What it means

When an MCP client requests server-side sampling with image content, the host converts it to AutoGen's Image type. If the configured model_info says the model family lacks vision support, RuntimeError('model {family} does not support vision.') is raised rather than sending an unusable request. model_info comes from the chat model client the sampling host was configured with.

Source

Thrown at python/packages/autogen-ext/src/autogen_ext/tools/mcp/_host/_sampling.py:48

    Handles text and image content conversion, with vision model validation for images.

    Args:
        content: MCP content object (text, image, or audio)
        model_info: Optional model information for vision capability checking

    Returns:
        Converted content as string or Image object

    Raises:
        RuntimeError: If image content is provided but model doesn't support vision
        ValueError: If content type is unsupported
    """
    if content.type == "text":
        return content.text
    elif content.type == "image":
        if model_info and not model_info.get("vision", False):
            model_family = model_info.get("family", "unknown")
            raise RuntimeError(f"model {model_family} does not support vision.")

        # Decode base64 image data and create PIL Image
        image_data = base64.b64decode(content.data)
        pil_image = PILImage.open(io.BytesIO(image_data))
        return Image.from_pil(pil_image)
    else:
        raise ValueError(f"Unsupported content type: {content.type}")


def parse_sampling_message(message: mcp_types.SamplingMessage, model_info: ModelInfo | None = None) -> LLMMessage:
    """Convert MCP sampling messages to AutoGen LLM messages.

    Args:
        message: MCP sampling message with role and content
        model_info: Optional model information for content parsing

    Returns:
        Converted AutoGen LLM message (UserMessage or AssistantMessage)

View on GitHub (pinned to 027ecf0a37)

Solutions

1. Switch the sampling host's chat model to a vision-capable model (e.g. gpt-4o family).
2. If the model genuinely supports images, provide a correct model_info dict with vision=True (and the right family) when constructing the model client.
3. On the server side, stop sending image content to non-vision models, or make image blocks optional.

Example fix

# before
client = OpenAIChatCompletionClient(model="gpt-3.5-turbo")  # vision=False
# MCP server sampling request with image -> RuntimeError

# after
client = OpenAIChatCompletionClient(
    model="gpt-4o",
    model_info={
        "vision": True,
        "function_calling": True,
        "json_output": True,
        "family": ModelFamily.GPT_4O,
    },
)

info = client.model_info or {}
has_image = any(b.get("type") == "image" for b in message_content)
if has_image and not info.get("vision", False):
    raise ValueError("reject sampling request: model lacks vision")

def model_supports_vision(model_info: dict | None) -> bool:
    return bool(model_info and model_info.get("vision", False))

try:
    msg = parse_sampling_message(message, model_info)
except RuntimeError as e:
    if "does not support vision" in str(e):
        return error_result("vision not supported by this model")
    raise

When it happens

Trigger: An MCP server sends a sampling request containing content with type='image' while the host's model_info has vision=False (or the family's capability entry omits vision). parse_content is called per content block during parse_sampling_message.

Common situations: Using a text-only model (e.g. a completion-style or small local model) with an MCP server that attaches screenshots/images; model_info defaults being conservative (vision not set) for a custom or newly added model family; a custom model_info dict built by hand without the vision key.

Related errors

• Unsupported content type: {content.type}
• Model does not support vision and image was provided
• Multi-part messages such as those containing images are curr
• Unknown content type: {part}
• Model does not support vision and image was provided

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/b7fd5b8f281e27bd. Report an issue: GitHub.