microsoft/autogen · error · ValueError

The model does not support function calling. MultimodalWebSu

Error message

The model does not support function calling. MultimodalWebSurfer requires a model that supports function calling.

What it means

MultimodalWebSurfer drives the browser by emitting tool/function calls, so it inspects model_client.model_info['function_calling'] at construction and rejects models that declare no function-calling support with ValueError. This is a hard capability requirement, not a runtime degradation — the agent cannot operate without structured tool calls.

Source

Thrown at python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py:236

        animate_actions: bool = False,
        to_save_screenshots: bool = False,
        use_ocr: bool = False,
        browser_channel: str | None = None,
        browser_data_dir: str | None = None,
        to_resize_viewport: bool = True,
        playwright: Playwright | None = None,
        context: BrowserContext | None = None,
    ):
        """
        Initialize the MultimodalWebSurfer.
        """
        super().__init__(name, description)
        if debug_dir is None and to_save_screenshots:
            raise ValueError(
                "Cannot save screenshots without a debug directory. Set it using the 'debug_dir' parameter. The debug directory is created if it does not exist."
            )
        if model_client.model_info["function_calling"] is False:
            raise ValueError(
                "The model does not support function calling. MultimodalWebSurfer requires a model that supports function calling."
            )

        self._model_client = model_client
        self.headless = headless
        self.browser_channel = browser_channel
        self.browser_data_dir = browser_data_dir
        self.start_page = start_page or self.DEFAULT_START_PAGE
        self.downloads_folder = downloads_folder
        self.debug_dir = debug_dir
        self.to_save_screenshots = to_save_screenshots
        self.use_ocr = use_ocr
        self.to_resize_viewport = to_resize_viewport
        self.animate_actions = animate_actions

        # Call init to set these in case not set
        self._playwright: Playwright | None = playwright
        self._context: BrowserContext | None = context

View on GitHub (pinned to 027ecf0a37)

Solutions

1. Use a function-calling-capable model, e.g. AzureOpenAIChatCompletionClient with gpt-4o / gpt-4-turbo
2. If wrapping a custom client, set model_info={..., 'function_calling': True} only if the backend truly supports tools
3. In tests, use ReplayChatCompletionClient with a model_info that declares function_calling and vision
4. Note vision ('vision': True) is also required for this multimodal agent

Example fix

// before
client = ReplayChatCompletionClient(chat_completions)  # default info: no function calling
surfer = MultimodalWebSurfer(name='s', model_client=client)  # ValueError

// after
from autogen_core.models import ModelInfo
client = ReplayChatCompletionClient(chat_completions, model_info=ModelInfo(vision=True, function_calling=True, json_output=False, family='unknown', structured_output=False))
surfer = MultimodalWebSurfer(name='s', model_client=client)

def supports_web_surfer(client) -> bool:
    info = client.model_info
    return bool(info.get('function_calling')) and bool(info.get('vision'))

from autogen_core.models import ChatCompletionClient

def is_web_surfer_capable(client: ChatCompletionClient) -> bool:
    info = client.model_info
    return bool(info.get('function_calling')) and bool(info.get('vision'))

When it happens

Trigger: Constructing MultimodalWebSurfer with a ChatCompletionClient whose model_info sets function_calling=False, e.g. some embedding/small local models, replay/seed clients used in tests, or a custom model_info dict that omits the key with a falsy default.

Common situations: Swapping in a cheaper or local model (e.g. via a custom client) that lacks tool-calling, mis-declared model_info when wrapping third-party endpoints, running deterministic tests with a mock client that did not set function_calling=True.

Related errors

• Please set OPENAI_API_KEY environment variable.
• Please set OPENAI_API_KEY environment variable.
• Cannot save screenshots without a debug directory. Set it us
• Failed to get login URL
• Authentication failed

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/1bd0c9b73cd26254. Report an issue: GitHub.