zylon-ai/private-gpt · error · NotImplementedError

LLM does not support structured chat.

Error message

LLM does not support structured chat.

What it means

Same capability check as the audio handler, but in the image handler's retried structured-chat loop: before each attempt (including retries after image-reduction passes) it verifies `callable(getattr(self._llm, 'astructured_chat', None))` and raises `NotImplementedError` if the LLM wrapper cannot produce Pydantic-validated structured output. It fires before any request is sent.

Source

Thrown at private_gpt/components/multimodality/image_handler.py:365

            async with retry_context(
                tries=self._num_max_retries,
                jitter=self._retry_jitter,
                logger=logger,
            ) as retry:
                seed = kwargs.pop("seed", None) or 0
                semaphore_manager: SemaphoreManager | None = kwargs.pop(
                    "semaphore_manager", None
                )
                count = 0
                max_iterations = kwargs.pop("max_iterations", 3)

                async def _call() -> Any:
                    nonlocal count
                    count += 1

                    structured_chat = getattr(self._llm, "astructured_chat", None)
                    if not callable(structured_chat):
                        raise NotImplementedError(
                            "LLM does not support structured chat."
                        )

                    new_kwargs = kwargs.copy()
                    new_kwargs["seed"] = str(seed) + str(count)

                    try:
                        current_messages = messages
                        if count > 1:
                            current_messages = self._reduce_images_in_messages(
                                messages, count - 1
                            )
                            logger.info(
                                f"Retry {count}: Reduced image quality (iteration {count - 1}/{max_iterations})"
                            )

                        return await structured_chat(
                            response_model, current_messages, **new_kwargs

View on GitHub (pinned to 4a030776a3)

Solutions

1. Point the multimodal LLM setting at a provider that implements `astructured_chat`
2. Add/alias `astructured_chat` on the custom LLM wrapper (delegate to structured-output support or parse into the response model)
3. For tests, provide a fake LLM with an async `astructured_chat` method

Example fix

// before
llm = CustomChatLLM()  # no astructured_chat
handler = ImageHandler(llm, ...)
await handler.extract(...)  # NotImplementedError

// after
class CustomChatLLM:
    async def astructured_chat(self, response_model, messages, **kwargs):
        resp = await self.achat(messages)
        return response_model.model_validate_json(resp.content)

if not callable(getattr(image_llm, "astructured_chat", None)):
    raise ValueError("image LLM must implement astructured_chat")

def supports_structured_images(llm: Any) -> bool:
    return callable(getattr(llm, "astructured_chat", None))

try:
    await image_handler.extract(...)
except NotImplementedError:
    # permanent capability gap; fail fast, do not retry
    raise

When it happens

Trigger: Running image extraction/description pipelines with an LLM class lacking `astructured_chat`; first attempt and any retry (the `count > 1` image-reduction path) both re-enter `_call` and re-check; using a custom or stub LLM injected into the image handler.

Common situations: Configuring a chat-only or completion-only LLM backend for multimodal image work; test doubles that don't mirror the real LLM surface; llama-index version drift renaming structured-output methods; local models behind a minimal wrapper.

Related errors

• Failed to describe images in the message.
• INVALID_REQUEST_ERROR
• REQUEST_TOO_LARGE_ERROR
• Last item is a FlexibleModel, expected a specific output_cls
• No items returned from astream_structured_predict

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/79352d364fbdb03e. Report an issue: GitHub.