What causes "Multimodal synthesis requires a chat LLM." in run-llama/llama_index?

Pass a chat LLM explicitly: ResponseSynthesizer(multimodal=True, llm=OpenAI(model='gpt-4o-mini')).

run-llama/llama_index · error · ValueError

Multimodal synthesis requires a chat LLM.

Error message

Multimodal synthesis requires a chat LLM.

What it means

BaseSynthesizer.__init__ with multimodal=True requires the configured LLM to be a chat model (is_chat_model check), because multimodal synthesis sends image content via chat messages, which completion-style LLMs cannot express. The check runs against self._llm, which may come from the constructor argument or Settings.llm.

Source

Thrown at llama-index-core/llama_index/core/response_synthesizers/base.py:92

        output_cls: Optional[Type[BaseModel]] = None,
        empty_response: Optional[str] = None,
        multimodal: bool = False,
    ) -> None:
        """Init params."""
        self._llm = llm or Settings.llm

        if callback_manager:
            self._llm.callback_manager = callback_manager

        self._callback_manager = callback_manager or Settings.callback_manager
        self._streaming = streaming
        self._output_cls = output_cls
        self._empty_response = empty_response or "Empty Response"
        self._multimodal = multimodal
        self._prompt_helper: PromptHelper
        if multimodal:
            if not is_chat_model(self._llm):
                raise ValueError("Multimodal synthesis requires a chat LLM.")
        self._prompt_helper = (
            prompt_helper
            or Settings._prompt_helper
            or PromptHelper.from_llm_metadata(
                self._llm.metadata,
            )
        )
        self._chat_prompt_helper = (
            chat_prompt_helper
            or Settings._chat_prompt_helper
            or ChatPromptHelper.from_llm_metadata(
                self._llm.metadata,
            )
        )

    def _empty_response_generator(self) -> Generator[str, None, None]:
        yield self._empty_response

View on GitHub (pinned to afd0fef371)

Solutions

Pass a chat LLM explicitly: ResponseSynthesizer(multimodal=True, llm=OpenAI(model='gpt-4o-mini')).
Set Settings.llm to a chat model (any LLM class whose is_chat_model is True) before constructing the engine.
If you do not need image input, drop multimodal=True.
Verify with llama_index.core.utils.is_chat_model(Settings.llm) at startup.

Example fix

# before
synth = ResponseSynthesizer(multimodal=True)  # Settings.llm is a completion model

# after
from llama_index.llms.openai import OpenAI
synth = ResponseSynthesizer(multimodal=True, llm=OpenAI(model="gpt-4o"))

Defensive patterns

Strategy: validation

Validate before calling

from llama_index.core.utils import is_chat_model
assert is_chat_model(llm), f"multimodal synthesis requires a chat LLM, got {type(llm)}"

Type guard

from llama_index.core.llms import ChatLLM
from llama_index.core.utils import is_chat_model

def is_chat_llm(llm) -> bool:
    return is_chat_model(llm)

Prevention

Always pass an explicit chat LLM when multimodal=True.
Check is_chat_model(Settings.llm) during app startup.

When it happens

Trigger: Building a query engine or ResponseSynthesizer with multimodal=True while Settings.llm (or the passed llm) is a completion model such as an OpenAI 'text-davinci-*'-style or HuggingFaceHub completion LLM; setting multimodal=True in a RetrieverQueryEngine factory; a default env where a non-chat LLM was installed globally.

Common situations: Multimodal RAG demos that forget to pass llm=OpenAI(model='gpt-4o'); environments where Settings.llm was set to a legacy completion model earlier; integration tests with a stub completion LLM.

Related errors

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/3a0e310e9481a18e. Report an issue: GitHub.