microsoft/semantic-kernel · error · ServiceInitializationError

When using a multi-modal model, a template must be specified

Error message

When using a multi-modal model, a template must be specified. Please provide a ONNXTemplate in the constructor.

What it means

Raised by the OnnxGenAIChatCompletion constructor after super().__init__() completes, when the loaded model's genai_config.json indicates a vision/multi-modal model (enable_multi_modality is True) but no ONNXTemplate was passed. Multi-modal ONNX models require a custom chat template to format inputs correctly because the default tokenizer.apply_chat_template may not handle image/audio tokens. Raised as ServiceInitializationError.

Source

Thrown at python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_chat_completion.py:85

                env_file_path=env_file_path,
                env_file_encoding=env_file_encoding,
            )
        except ValidationError as e:
            raise ServiceInitializationError(f"Error creating OnnxGenAISettings: {e}") from e

        if settings.chat_model_folder is None:
            raise ServiceInitializationError(
                "AI model path is not provided. Please provide the 'ai_model_path' parameter in the constructor. "
                "OR set the 'ONNX_GEN_AI_CHAT_MODEL_FOLDER' environment variable."
            )

        if ai_model_id is None:
            ai_model_id = settings.chat_model_folder

        super().__init__(ai_model_id=ai_model_id, ai_model_path=settings.chat_model_folder, template=template, **kwargs)

        if self.enable_multi_modality and template is None:
            raise ServiceInitializationError(
                "When using a multi-modal model, a template must be specified."
                " Please provide a ONNXTemplate in the constructor."
            )

    @override
    async def _inner_get_chat_message_contents(
        self,
        chat_history: "ChatHistory",
        settings: "PromptExecutionSettings",
    ) -> list["ChatMessageContent"]:
        """Create chat message contents, in the number specified by the settings.

        Args:
            chat_history : A list of chats in a chat_history object, that can be
                rendered into messages from system, user, assistant and tools.
            settings : Settings for the request.

        Returns:

View on GitHub (pinned to c028a0c7dc)

Solutions

1. Pass an ONNXTemplate instance: OnnxGenAIChatCompletion(ai_model_path='...', template=Phi3VisionTemplate())
2. If you intended to use a text-only model, point ai_model_path at a non-vision model folder
3. Check the model's genai_config.json to confirm whether 'vision' is in the model config

Example fix

// before
chat = OnnxGenAIChatCompletion(ai_model_path='/models/phi-3-vision')
// after
from semantic_kernel.connectors.ai.onnx.utils import ONNXTemplate
chat = OnnxGenAIChatCompletion(ai_model_path='/models/phi-3-vision', template=my_vision_template)

import json, os

model_path = '/path/to/vision-model'
config_file = os.path.join(model_path, 'genai_config.json')
with open(config_file) as f:
    config = json.load(f)

is_multimodal = 'vision' in config.get('model', {})
template = my_onnx_template if is_multimodal else None

chat = OnnxGenAIChatCompletion(ai_model_path=model_path, template=template)

from semantic_kernel.exceptions.service_exceptions import ServiceInitializationError

try:
    chat = OnnxGenAIChatCompletion(ai_model_path=model_path)
except ServiceInitializationError as e:
    if 'multi-modal model, a template must be specified' in str(e):
        chat = OnnxGenAIChatCompletion(ai_model_path=model_path, template=my_template)
    else:
        raise

When it happens

Trigger: Constructing OnnxGenAIChatCompletion(ai_model_path='/path/to/vision-model') where genai_config.json has a 'model' dict containing the key 'vision', without passing template=some_onnx_template.

Common situations: Downloading a multi-modal model (e.g. Phi-3-vision) and reusing the same constructor call as for a text-only model; forgetting that vision models need a template; not knowing which ONNXTemplate subclass to use.

Related errors

• Error creating OnnxGenAISettings: {e}
• AI model path is not provided. Please provide the 'ai_model_
• The model does not support multi-modality
• Image Content URI needs to be set, because onnx can only wor
• Audio Content URI needs to be set, because onnx can only wor

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/c6d2832dc06d4ecf. Report an issue: GitHub.