What causes "Unexpected response type from LlamaCpp model." in microsoft/autogen?

Pin a known-good llama-cpp-python version (e.g. pip install 'llama-cpp-python==0.3.x') matching the autogen-ext release's tested range

microsoft/autogen · error · ValueError

Unexpected response type from LlamaCpp model.

Error message

Unexpected response type from LlamaCpp model.

What it means

After awaiting create_chat_completion (run in an executor), the client requires the result to be a dict, since it immediately indexes response['usage'] and response['choices']. If llama-cpp-python returns something else (e.g. an object, None, or a changed return shape after a monkeypatch or version drift), this ValueError fires. In practice it signals a broken or incompatible llama-cpp-python installation.

Source

Thrown at python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py:333

        if self.model_info["function_calling"]:
            # Run this in on the event loop to avoid blocking.
            response_future = asyncio.get_event_loop().run_in_executor(
                None,
                lambda: self.llm.create_chat_completion(
                    messages=converted_messages, tools=convert_tools(tools), stream=False, **create_args
                ),
            )
        else:
            response_future = asyncio.get_event_loop().run_in_executor(
                None, lambda: self.llm.create_chat_completion(messages=converted_messages, stream=False, **create_args)
            )
        if cancellation_token:
            cancellation_token.link_future(response_future)
        response = await response_future

        if not isinstance(response, dict):
            raise ValueError("Unexpected response type from LlamaCpp model.")

        self._total_usage["prompt_tokens"] += response["usage"]["prompt_tokens"]
        self._total_usage["completion_tokens"] += response["usage"]["completion_tokens"]

        # Parse the response
        response_tool_calls: ChatCompletionTool | None = None
        response_text: str | None = None
        if "choices" in response and len(response["choices"]) > 0:
            if "message" in response["choices"][0]:
                response_text = response["choices"][0]["message"]["content"]
            if "tool_calls" in response["choices"][0]:
                response_tool_calls = response["choices"][0]["tool_calls"]  # type: ignore

        content: List[FunctionCall] | str = ""
        thought: str | None = None
        if response_tool_calls:
            content = []
            for tool_call in response_tool_calls:

View on GitHub (pinned to 027ecf0a37)

Solutions

Pin a known-good llama-cpp-python version (e.g. pip install 'llama-cpp-python==0.3.x') matching the autogen-ext release's tested range
Remove stream=True from extra_create_args — use create_stream (which itself raises NotImplementedError) or omit it
If testing with a fake Llama, make create_chat_completion return a dict with 'usage' and 'choices' keys

Example fix

# before
result = await client.create(messages, extra_create_args={"stream": True})

# after
result = await client.create(messages)

Defensive patterns

Strategy: fallback

Validate before calling

installed = importlib.metadata.version("llama-cpp-python")
if tuple(int(x) for x in installed.split(".")[:2]) < (0, 2):
    raise RuntimeError(f"llama-cpp-python {installed} too old; upgrade to a tested release")

Try / catch

try:
    result = await client.create(messages)
except ValueError as e:
    if "Unexpected response type" in str(e):
        logger.error("llama-cpp-python returned a non-dict; check version/patches")
        raise
    raise

Prevention

Pin llama-cpp-python to the version range autogen-ext tests against
Never pass stream=True via extra_create_args to create()
In tests, make fake Llama.create_chat_completion return dict responses with usage/choices

When it happens

Trigger: llama-cpp-python version whose create_chat_completion returns a non-dict (API drift between the installed wheel and what autogen-ext expects); a mocked/patched Llama object in tests returning MagicMock or a tuple; streaming accidentally enabled inside create_args via extra_create_args (stream=True) so the return value is a generator.

Common situations: Pip-installed a pre-release or very old llama-cpp-python; passing extra_create_args={'stream': True} to a non-streaming call; unit tests substituting a fake Llama whose create_chat_completion returns a string; ABI-mismatched wheel returning error objects.

Related errors

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/52cd29540fb4d191. Report an issue: GitHub.