microsoft/autogen · error · NotImplementedError

Stream not yet implemented for LlamaCppChatCompletionClient

Error message

Stream not yet implemented for LlamaCppChatCompletionClient

What it means

LlamaCppChatCompletionClient.create_stream() raises NotImplementedError unconditionally after its tool_choice validation — streaming has never been implemented for this client. The subsequent yield '' is dead code making the function an async generator. Any attempt to stream from a llama.cpp model fails here.

Source

Thrown at python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py:426

        messages: Sequence[LLMMessage],
        *,
        tools: Sequence[Tool | ToolSchema] = [],
        tool_choice: Tool | Literal["auto", "required", "none"] = "auto",
        # None means do not override the default
        # A value means to override the client default - often specified in the constructor
        json_output: Optional[bool | type[BaseModel]] = None,
        extra_create_args: Mapping[str, Any] = {},
        cancellation_token: Optional[CancellationToken] = None,
    ) -> AsyncGenerator[Union[str, CreateResult], None]:
        # Validate tool_choice parameter even though streaming is not implemented
        if tool_choice != "auto" and tool_choice != "none":
            if not self.model_info["function_calling"]:
                raise ValueError("tool_choice specified but model does not support function calling")
            if len(tools) == 0:
                raise ValueError("tool_choice specified but no tools provided")
            logger.warning("tool_choice parameter specified but may not be supported by llama-cpp-python")

        raise NotImplementedError("Stream not yet implemented for LlamaCppChatCompletionClient")
        yield ""

    # Implement abstract methods
    def actual_usage(self) -> RequestUsage:
        return RequestUsage(
            prompt_tokens=self._total_usage.get("prompt_tokens", 0),
            completion_tokens=self._total_usage.get("completion_tokens", 0),
        )

    @property
    def capabilities(self) -> ModelInfo:
        return self.model_info

    def count_tokens(
        self,
        messages: Sequence[SystemMessage | UserMessage | AssistantMessage | FunctionExecutionResultMessage],
        **kwargs: Any,
    ) -> int:

View on GitHub (pinned to 027ecf0a37)

Solutions

  1. Use the non-streaming API: result = await client.create(messages)
  2. If chunked output is required, simulate it by yielding the complete result once it arrives, or buffer create() output yourself
  3. Catch NotImplementedError at the runner level and fall back to create() when the client lacks streaming support

Example fix

# before
async for chunk in client.create_stream(messages):
    print(chunk, end="")

# after
result = await client.create(messages)
print(result.content)
Defensive patterns

Strategy: fallback

Validate before calling

if type(client).create_stream is LlamaCppChatCompletionClient.create_stream:
    result = await client.create(messages)  # streaming unsupported
else:
    async for chunk in client.create_stream(messages):
        ...

Try / catch

try:
    async for chunk in client.create_stream(messages):
        print(chunk, end="")
except NotImplementedError:
    result = await client.create(messages)
    print(result.content)

Prevention

When it happens

Trigger: Calling client.create_stream(...) on LlamaCppChatCompletionClient in any form; frameworks that auto-detect and prefer streaming (e.g. some UI runtimes) triggering it implicitly.

Common situations: Swapping an OpenAI client for the llama.cpp client in a streaming chat UI; a shared runner that iterates create_stream when a stream=True config flag is set.

Related errors

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/28ecca4cb04e3598. Report an issue: GitHub.