{"record":{"id":"a7a36915aae6b0ed","repo":"zylon-ai/private-gpt","slug":"llm-does-not-support-structured-chat","errorCode":null,"errorMessage":"LLM does not support structured chat.","messagePattern":"LLM does not support structured chat\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/multimodality/audio_handler.py","lineNumber":488,"sourceCode":"        messages: list[ChatMessage],\n        **kwargs: Any,\n    ) -> Any:\n        try:\n            async with retry_context(\n                tries=self._num_max_retries,\n                jitter=self._retry_jitter,\n                logger=logger,\n            ) as retry:\n                seed = kwargs.pop(\"seed\", None) or 0\n                count = 0\n\n                async def _call() -> Any:\n                    nonlocal count\n                    count += 1\n\n                    structured_chat = getattr(self._llm, \"astructured_chat\", None)\n                    if not callable(structured_chat):\n                        raise NotImplementedError(\n                            \"LLM does not support structured chat.\"\n                        )\n\n                    new_kwargs = kwargs.copy()\n                    new_kwargs[\"seed\"] = str(seed) + str(count)\n\n                    return await structured_chat(response_model, messages, **new_kwargs)\n\n                return await retry(_call)\n        except MODEL_NOT_AVAILABLE_EXCEPTION_TYPES as e:\n            raise ModelNotAvailableError(\n                \"Model server is not available or request failed.\"\n            ) from e\n        except Exception:\n            raise\n\n\nclass AudioProcessingWorkflow(Workflow):","sourceCodeStart":470,"sourceCodeEnd":506,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/multimodality/audio_handler.py#L470-L506","documentation":"Raised inside the audio handler's structured-chat retry closure when the configured LLM object has no callable astructured_chat attribute (checked via getattr each attempt). The audio processing workflow needs structured (schema-constrained) output to parse transcription/analysis results; LLM backends that do not implement the astructured_chat interface get NotImplementedError instead of an AttributeError deep in the call. It is a capability mismatch: the configured audio_multimodal_llm does not support the structured-chat API.","triggerScenarios":"Running AudioProcessingWorkflow (or the structured chat helper around audio_handler.py:488) with an LLM wrapper/backend lacking astructured_chat — e.g. a mock in tests, a minimal OpenAI-compatible wrapper, or an older llama-index LLM class; passing a plain LLM where a structured-capable one is required.","commonSituations":"Swapping the multimodal LLM backend to a custom/in-house wrapper; upgrading llama-index where the structured-chat method was renamed/removed for some classes; test doubles not implementing the full interface.","solutions":["Use an LLM class that implements astructured_chat (llama-index structured-LLM interface) for audio_multimodal_llm.","If wrapping an OpenAI-compatible endpoint, implement async def astructured_chat(response_model, messages, **kwargs) using JSON/tool-call mode on the wrapper.","In tests, patch or implement astructured_chat on the fake LLM.","Check hasattr(llm, 'astructured_chat') at wiring time to fail fast with a clearer message."],"exampleFix":"# before\nworkflow = AudioProcessingWorkflow(audio_multimodal_llm=plain_llm)  # no astructured_chat\n\n# after\nclass StructuredCapableLLM(PlainLLM):\n    async def astructured_chat(self, response_model, messages, **kwargs):\n        return await run_structure(self.acompletion(messages), response_model)\nworkflow = AudioProcessingWorkflow(audio_multimodal_llm=StructuredCapableLLM(...))","handlingStrategy":"type-guard","validationCode":"if not callable(getattr(audio_multimodal_llm, 'astructured_chat', None)):\n    raise ConfigError('audio LLM must implement astructured_chat')","typeGuard":"def supports_structured_chat(llm) -> bool:\n    return callable(getattr(llm, 'astructured_chat', None))","tryCatchPattern":"try:\n    result = await run_structured_audio_chat(...)\nexcept NotImplementedError as e:\n    if 'structured chat' in str(e):\n        raise ConfigError('swap in a structured-capable LLM') from e","preventionTips":["Assert the astructured_chat capability when wiring AudioProcessingWorkflow, not at first call.","Keep test doubles faithful: fakes used for audio flows must implement astructured_chat."],"tags":["multimodal","audio","llm","interface-mismatch","not-implemented"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}