microsoft/autogen · error · ValueError

Model does not support vision and image was provided

Error message

Model does not support vision and image was provided

What it means

OllamaChatCompletionClient validates messages before sending them to the Ollama API. If the resolved model_info declares vision=False and any UserMessage in the conversation contains an Image object in its multimodal content list, the client raises ValueError instead of making the request. This is a pre-flight capability check so unsupported image input fails fast rather than producing a confusing server-side error.

Source

Thrown at python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py:577

            else:
                raise ValueError(f"json_output must be a boolean or a Pydantic model class, got {type(json_output)}")

        if "format" in create_args:
            # Handle the case where format is set from create_args.
            if json_output is not None:
                raise ValueError("json_output and format cannot be set at the same time. Use json_output instead.")
            assert response_format_value is None
            response_format_value = create_args["format"]
            # Remove format from create_args to prevent passing it twice.
            del create_args["format"]

        # TODO: allow custom handling.
        # For now we raise an error if images are present and vision is not supported
        if self.model_info["vision"] is False:
            for message in messages:
                if isinstance(message, UserMessage):
                    if isinstance(message.content, list) and any(isinstance(x, Image) for x in message.content):
                        raise ValueError("Model does not support vision and image was provided")

        if self.model_info["json_output"] is False and json_output is True:
            raise ValueError("Model does not support JSON output.")

        ollama_messages_nested = [to_ollama_type(m) for m in messages]
        ollama_messages = [item for sublist in ollama_messages_nested for item in sublist]

        if self.model_info["function_calling"] is False and len(tools) > 0:
            raise ValueError("Model does not support function calling and tools were provided")

        converted_tools: List[OllamaTool] = []

        # Handle tool_choice parameter in a way that is compatible with Ollama API.
        if isinstance(tool_choice, Tool):
            # If tool_choice is a Tool, convert it to OllamaTool.
            converted_tools = convert_tools([tool_choice])
        elif tool_choice == "none":
            # No tool choice, do not pass tools to the API.

View on GitHub (pinned to 027ecf0a37)

Solutions

1. Switch to a vision-capable Ollama model (e.g. llava, llama3.2-vision, qwen2-vl) and pass model_info={'vision': True, ...} so the capability flag is accurate
2. Filter Image parts out of UserMessage content before sending if vision is not needed
3. If your model genuinely supports vision but is not in the known list, supply explicit model_info with 'vision': True to the client constructor

Example fix

// before
client = OllamaChatCompletionClient(model='llama3.1')
await client.create([UserMessage(source='user', content=['describe this', Image.from_file('cat.png')])])
# ValueError: Model does not support vision and image was provided

// after
client = OllamaChatCompletionClient(
    model='llava',
    model_info={'vision': True, 'function_calling': True, 'json_output': True, 'family': ModelFamily.UNKNOWN, 'structured_output': False},
)
await client.create([UserMessage(source='user', content=['describe this', Image.from_file('cat.png')])])

from autogen_core.models import UserMessage, Image
from autogen_core import CancellationToken

def assert_vision_ok(model_info: dict, messages) -> None:
    if model_info.get('vision', False) is False:
        for m in messages:
            if isinstance(m, UserMessage) and isinstance(m.content, list):
                if any(isinstance(p, Image) for p in m.content):
                    raise RuntimeError(f'Model lacks vision but message from {m.source} contains an Image')

from autogen_core.models import UserMessage, Image
from typing import Any

def has_image_part(msg: Any) -> bool:
    return (isinstance(msg, UserMessage) and isinstance(msg.content, list)
            and any(isinstance(p, Image) for p in msg.content))

try:
    result = await client.create(messages)
except ValueError as e:
    if 'does not support vision' in str(e):
        messages = [strip_images(m) for m in messages]  # or switch model
        result = await client.create(messages)
    else:
        raise

When it happens

Trigger: Calling create()/create_stream() on OllamaChatCompletionClient where model_info['vision'] is False (either explicitly set or defaulted for a text-only model such as llama3.1) while at least one UserMessage has content=[..., Image(...), ...]. Even one image in one user message triggers it.

Common situations: Using a text-only Ollama model (llama3, mistral, qwen2:7b) with a multimodal agent pattern copied from a vision example; forgetting to pass model_info for a vision-capable local model so it defaults to vision=False; publishing an image in a chat context that reaches the model.

Related errors

• Unknown content type: {part}
• Invalid aggregate message {reason}
• Unsupported content type {item.GetType()}
• Only TextContent and ImageContent are allowed in MultiModalM
• Image content cannot be converted to text

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/e5ddabc8775cf63f. Report an issue: GitHub.