{"record":{"id":"28ecca4cb04e3598","repo":"microsoft/autogen","slug":"stream-not-yet-implemented-for-llamacppchatcomplet","errorCode":null,"errorMessage":"Stream not yet implemented for LlamaCppChatCompletionClient","messagePattern":"Stream not yet implemented for LlamaCppChatCompletionClient","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py","lineNumber":426,"sourceCode":"        messages: Sequence[LLMMessage],\n        *,\n        tools: Sequence[Tool | ToolSchema] = [],\n        tool_choice: Tool | Literal[\"auto\", \"required\", \"none\"] = \"auto\",\n        # None means do not override the default\n        # A value means to override the client default - often specified in the constructor\n        json_output: Optional[bool | type[BaseModel]] = None,\n        extra_create_args: Mapping[str, Any] = {},\n        cancellation_token: Optional[CancellationToken] = None,\n    ) -> AsyncGenerator[Union[str, CreateResult], None]:\n        # Validate tool_choice parameter even though streaming is not implemented\n        if tool_choice != \"auto\" and tool_choice != \"none\":\n            if not self.model_info[\"function_calling\"]:\n                raise ValueError(\"tool_choice specified but model does not support function calling\")\n            if len(tools) == 0:\n                raise ValueError(\"tool_choice specified but no tools provided\")\n            logger.warning(\"tool_choice parameter specified but may not be supported by llama-cpp-python\")\n\n        raise NotImplementedError(\"Stream not yet implemented for LlamaCppChatCompletionClient\")\n        yield \"\"\n\n    # Implement abstract methods\n    def actual_usage(self) -> RequestUsage:\n        return RequestUsage(\n            prompt_tokens=self._total_usage.get(\"prompt_tokens\", 0),\n            completion_tokens=self._total_usage.get(\"completion_tokens\", 0),\n        )\n\n    @property\n    def capabilities(self) -> ModelInfo:\n        return self.model_info\n\n    def count_tokens(\n        self,\n        messages: Sequence[SystemMessage | UserMessage | AssistantMessage | FunctionExecutionResultMessage],\n        **kwargs: Any,\n    ) -> int:","sourceCodeStart":408,"sourceCodeEnd":444,"githubUrl":"https://github.com/microsoft/autogen/blob/027ecf0a379bcc1d09956d46d12d44a3ad9cee14/python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py#L408-L444","documentation":"LlamaCppChatCompletionClient.create_stream() raises NotImplementedError unconditionally after its tool_choice validation — streaming has never been implemented for this client. The subsequent yield '' is dead code making the function an async generator. Any attempt to stream from a llama.cpp model fails here.","triggerScenarios":"Calling client.create_stream(...) on LlamaCppChatCompletionClient in any form; frameworks that auto-detect and prefer streaming (e.g. some UI runtimes) triggering it implicitly.","commonSituations":"Swapping an OpenAI client for the llama.cpp client in a streaming chat UI; a shared runner that iterates create_stream when a stream=True config flag is set.","solutions":["Use the non-streaming API: result = await client.create(messages)","If chunked output is required, simulate it by yielding the complete result once it arrives, or buffer create() output yourself","Catch NotImplementedError at the runner level and fall back to create() when the client lacks streaming support"],"exampleFix":"# before\nasync for chunk in client.create_stream(messages):\n    print(chunk, end=\"\")\n\n# after\nresult = await client.create(messages)\nprint(result.content)","handlingStrategy":"fallback","validationCode":"if type(client).create_stream is LlamaCppChatCompletionClient.create_stream:\n    result = await client.create(messages)  # streaming unsupported\nelse:\n    async for chunk in client.create_stream(messages):\n        ...","typeGuard":null,"tryCatchPattern":"try:\n    async for chunk in client.create_stream(messages):\n        print(chunk, end=\"\")\nexcept NotImplementedError:\n    result = await client.create(messages)\n    print(result.content)","preventionTips":["Check the client's docs/capabilities for streaming support before wiring a streaming UI","Wrap streaming in a helper with a create() fallback for clients without stream support","Treat NotImplementedError from third-party clients as a expected branch, not a bug"],"tags":["llama-cpp","streaming","not-implemented","api-limitation"],"backgroundTag":null,"analyzedSha":"027ecf0a379bcc1d09956d46d12d44a3ad9cee14","analyzedAt":"2026-08-15T03:38:00.719Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}