{"record":{"id":"c6d2832dc06d4ecf","repo":"microsoft/semantic-kernel","slug":"when-using-a-multi-modal-model-a-template-must-be","errorCode":null,"errorMessage":"When using a multi-modal model, a template must be specified. Please provide a ONNXTemplate in the constructor.","messagePattern":"When using a multi-modal model, a template must be specified\\. Please provide a ONNXTemplate in the constructor\\.","errorType":"exception","errorClass":"ServiceInitializationError","httpStatus":null,"severity":"error","filePath":"python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_chat_completion.py","lineNumber":85,"sourceCode":"                env_file_path=env_file_path,\n                env_file_encoding=env_file_encoding,\n            )\n        except ValidationError as e:\n            raise ServiceInitializationError(f\"Error creating OnnxGenAISettings: {e}\") from e\n\n        if settings.chat_model_folder is None:\n            raise ServiceInitializationError(\n                \"AI model path is not provided. Please provide the 'ai_model_path' parameter in the constructor. \"\n                \"OR set the 'ONNX_GEN_AI_CHAT_MODEL_FOLDER' environment variable.\"\n            )\n\n        if ai_model_id is None:\n            ai_model_id = settings.chat_model_folder\n\n        super().__init__(ai_model_id=ai_model_id, ai_model_path=settings.chat_model_folder, template=template, **kwargs)\n\n        if self.enable_multi_modality and template is None:\n            raise ServiceInitializationError(\n                \"When using a multi-modal model, a template must be specified.\"\n                \" Please provide a ONNXTemplate in the constructor.\"\n            )\n\n    @override\n    async def _inner_get_chat_message_contents(\n        self,\n        chat_history: \"ChatHistory\",\n        settings: \"PromptExecutionSettings\",\n    ) -> list[\"ChatMessageContent\"]:\n        \"\"\"Create chat message contents, in the number specified by the settings.\n\n        Args:\n            chat_history : A list of chats in a chat_history object, that can be\n                rendered into messages from system, user, assistant and tools.\n            settings : Settings for the request.\n\n        Returns:","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_chat_completion.py#L67-L103","documentation":"Raised by the OnnxGenAIChatCompletion constructor after super().__init__() completes, when the loaded model's genai_config.json indicates a vision/multi-modal model (enable_multi_modality is True) but no ONNXTemplate was passed. Multi-modal ONNX models require a custom chat template to format inputs correctly because the default tokenizer.apply_chat_template may not handle image/audio tokens. Raised as ServiceInitializationError.","triggerScenarios":"Constructing OnnxGenAIChatCompletion(ai_model_path='/path/to/vision-model') where genai_config.json has a 'model' dict containing the key 'vision', without passing template=some_onnx_template.","commonSituations":"Downloading a multi-modal model (e.g. Phi-3-vision) and reusing the same constructor call as for a text-only model; forgetting that vision models need a template; not knowing which ONNXTemplate subclass to use.","solutions":["Pass an ONNXTemplate instance: OnnxGenAIChatCompletion(ai_model_path='...', template=Phi3VisionTemplate())","If you intended to use a text-only model, point ai_model_path at a non-vision model folder","Check the model's genai_config.json to confirm whether 'vision' is in the model config"],"exampleFix":"// before\nchat = OnnxGenAIChatCompletion(ai_model_path='/models/phi-3-vision')\n// after\nfrom semantic_kernel.connectors.ai.onnx.utils import ONNXTemplate\nchat = OnnxGenAIChatCompletion(ai_model_path='/models/phi-3-vision', template=my_vision_template)","handlingStrategy":"validation","validationCode":"import json, os\n\nmodel_path = '/path/to/vision-model'\nconfig_file = os.path.join(model_path, 'genai_config.json')\nwith open(config_file) as f:\n    config = json.load(f)\n\nis_multimodal = 'vision' in config.get('model', {})\ntemplate = my_onnx_template if is_multimodal else None\n\nchat = OnnxGenAIChatCompletion(ai_model_path=model_path, template=template)","typeGuard":null,"tryCatchPattern":"from semantic_kernel.exceptions.service_exceptions import ServiceInitializationError\n\ntry:\n    chat = OnnxGenAIChatCompletion(ai_model_path=model_path)\nexcept ServiceInitializationError as e:\n    if 'multi-modal model, a template must be specified' in str(e):\n        chat = OnnxGenAIChatCompletion(ai_model_path=model_path, template=my_template)\n    else:\n        raise","preventionTips":["Check genai_config.json for a 'vision' key before constructing the service","Keep a registry of ONNXTemplate instances matched to your downloaded models","Pass the template proactively for any model you suspect is multi-modal"],"tags":["onnx","multimodal","template","initialization"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}