zylon-ai/private-gpt · error · NotImplementedError

LLM does not support structured chat.

Error message

LLM does not support structured chat.

What it means

Raised inside the audio handler's structured-chat retry closure when the configured LLM object has no callable astructured_chat attribute (checked via getattr each attempt). The audio processing workflow needs structured (schema-constrained) output to parse transcription/analysis results; LLM backends that do not implement the astructured_chat interface get NotImplementedError instead of an AttributeError deep in the call. It is a capability mismatch: the configured audio_multimodal_llm does not support the structured-chat API.

Source

Thrown at private_gpt/components/multimodality/audio_handler.py:488

        messages: list[ChatMessage],
        **kwargs: Any,
    ) -> Any:
        try:
            async with retry_context(
                tries=self._num_max_retries,
                jitter=self._retry_jitter,
                logger=logger,
            ) as retry:
                seed = kwargs.pop("seed", None) or 0
                count = 0

                async def _call() -> Any:
                    nonlocal count
                    count += 1

                    structured_chat = getattr(self._llm, "astructured_chat", None)
                    if not callable(structured_chat):
                        raise NotImplementedError(
                            "LLM does not support structured chat."
                        )

                    new_kwargs = kwargs.copy()
                    new_kwargs["seed"] = str(seed) + str(count)

                    return await structured_chat(response_model, messages, **new_kwargs)

                return await retry(_call)
        except MODEL_NOT_AVAILABLE_EXCEPTION_TYPES as e:
            raise ModelNotAvailableError(
                "Model server is not available or request failed."
            ) from e
        except Exception:
            raise


class AudioProcessingWorkflow(Workflow):

View on GitHub (pinned to 4a030776a3)

Solutions

1. Use an LLM class that implements astructured_chat (llama-index structured-LLM interface) for audio_multimodal_llm.
2. If wrapping an OpenAI-compatible endpoint, implement async def astructured_chat(response_model, messages, **kwargs) using JSON/tool-call mode on the wrapper.
3. In tests, patch or implement astructured_chat on the fake LLM.
4. Check hasattr(llm, 'astructured_chat') at wiring time to fail fast with a clearer message.

Example fix

# before
workflow = AudioProcessingWorkflow(audio_multimodal_llm=plain_llm)  # no astructured_chat

# after
class StructuredCapableLLM(PlainLLM):
    async def astructured_chat(self, response_model, messages, **kwargs):
        return await run_structure(self.acompletion(messages), response_model)
workflow = AudioProcessingWorkflow(audio_multimodal_llm=StructuredCapableLLM(...))

if not callable(getattr(audio_multimodal_llm, 'astructured_chat', None)):
    raise ConfigError('audio LLM must implement astructured_chat')

def supports_structured_chat(llm) -> bool:
    return callable(getattr(llm, 'astructured_chat', None))

try:
    result = await run_structured_audio_chat(...)
except NotImplementedError as e:
    if 'structured chat' in str(e):
        raise ConfigError('swap in a structured-capable LLM') from e

When it happens

Trigger: Running AudioProcessingWorkflow (or the structured chat helper around audio_handler.py:488) with an LLM wrapper/backend lacking astructured_chat — e.g. a mock in tests, a minimal OpenAI-compatible wrapper, or an older llama-index LLM class; passing a plain LLM where a structured-capable one is required.

Common situations: Swapping the multimodal LLM backend to a custom/in-house wrapper; upgrading llama-index where the structured-chat method was renamed/removed for some classes; test doubles not implementing the full interface.

Related errors

• Audio blocks found but no audio-capable LLM provided.
• Failed to describe audio in the message.
• Failed to describe images in the message.
• RemoteTokenizeTokenizer only supports text tokenization
• TikTokenTokenizer only supports text token counting

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/a7a36915aae6b0ed. Report an issue: GitHub.