{"record":{"id":"aa8854d630866d6a","repo":"microsoft/semantic-kernel","slug":"failed-inference-with-onnx","errorCode":null,"errorMessage":"Failed Inference with ONNX","messagePattern":"Failed Inference with ONNX","errorType":"exception","errorClass":"ServiceInvalidResponseError","httpStatus":null,"severity":"error","filePath":"python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_completion_base.py","lineNumber":93,"sourceCode":"                input_tokens = self.tokenizer.encode(prompt)\n                generator.append_tokens(input_tokens)\n            else:\n                # With the use of Pybind in ONNX there is currently no way to load images from bytes\n                # We can only open images & audios from a file path currently\n                if images is not None:\n                    images = OnnxRuntimeGenAi.Images.open(*[str(image.uri) for image in images])\n                if audios is not None:\n                    audios = OnnxRuntimeGenAi.Audios.open(*[str(audio.uri) for audio in audios])\n                input_tokens = self.tokenizer(prompt, images=images, audios=audios)\n                generator.set_inputs(input_tokens)\n\n            while not generator.is_done():\n                generator.generate_next_token()\n                new_token_choices = [self.tokenizer_stream.decode(token) for token in generator.get_next_tokens()]\n                yield new_token_choices\n            del generator\n        except Exception as ex:\n            raise ServiceInvalidResponseError(\"Failed Inference with ONNX\", ex) from ex\n\n    async def _generate_next_token(\n        self,\n        prompt: str,\n        settings: OnnxGenAIPromptExecutionSettings,\n        images: list[ImageContent] | None = None,\n        audios: list[AudioContent] | None = None,\n    ):\n        token_choices: list[str] = []\n        async for new_token_choice in self._generate_next_token_async(prompt, settings, images, audios=audios):\n            # zip only works if the lists are the same length\n            if len(token_choices) == 0:\n                token_choices = new_token_choice\n            else:\n                token_choices = [old_token + new_token for old_token, new_token in zip(token_choices, new_token_choice)]\n        return token_choices\n","sourceCodeStart":75,"sourceCodeEnd":110,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_completion_base.py#L75-L110","documentation":"Raised inside _generate_next_token_async when any Exception occurs during the ONNX inference loop — encoding the prompt, setting generator inputs, calling generate_next_token, or decoding tokens. The broad except clause wraps all exceptions as ServiceInvalidResponseError chained via 'from ex'. This is a runtime inference failure, distinct from initialization failures.","triggerScenarios":"Calling get_chat_message_contents, get_streaming_chat_message_contents, or text completion when the ONNX generator fails during token generation. Common root causes: malformed prompt after template application, image/audio file not found at the uri during Images.open/Audios.open, generator memory exhaustion, or incompatible search options.","commonSituations":"Image/audio URI pointing at a file that was deleted or moved after construction; setting search options in OnnxGenAIPromptExecutionSettings that the model does not support; passing a prompt that fails encoding; running out of memory with a large model on limited hardware.","solutions":["Inspect the chained exception (__cause__) for the specific ONNX runtime error","If using multimodal, verify all image.uri and audio.uri files exist and are readable at inference time","Check that settings (max_tokens, temperature, etc.) are valid for the model","Monitor GPU/CPU memory during inference and reduce context size if needed"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import os\nfrom semantic_kernel.contents import ImageContent, AudioContent\n\ndef validate_multimodal_uris(chat_history):\n    for msg in chat_history.messages:\n        for item in msg.items:\n            if isinstance(item, (ImageContent, AudioContent)):\n                if not item.uri or not os.path.exists(str(item.uri).replace('file://', '')):\n                    raise ValueError(f'Multimodal content URI missing or file not found: {item.uri}')","typeGuard":null,"tryCatchPattern":"from semantic_kernel.exceptions.service_exceptions import ServiceInvalidResponseError\n\ntry:\n    result = await chat.get_chat_message_contents(chat_history=history, settings=settings)\nexcept ServiceInvalidResponseError as e:\n    cause = e.__cause__\n    logger.error('ONNX inference failed: %s. Root cause: %s', e, cause)\n    raise","preventionTips":["Verify all image and audio file URIs exist on disk right before inference","Keep multimodal temp files alive (do not garbage-collect) for the duration of the call","Validate OnnxGenAIPromptExecutionSettings search options against model capabilities","Log e.__cause__ to pinpoint the ONNX runtime-level error"],"tags":["onnx","inference","runtime","response-error"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}