What causes "Failed Inference with ONNX" in microsoft/semantic-kernel?

Inspect the chained exception (__cause__) for the specific ONNX runtime error

microsoft/semantic-kernel · error · ServiceInvalidResponseError

Failed Inference with ONNX

Error message

Failed Inference with ONNX

What it means

Raised inside _generate_next_token_async when any Exception occurs during the ONNX inference loop — encoding the prompt, setting generator inputs, calling generate_next_token, or decoding tokens. The broad except clause wraps all exceptions as ServiceInvalidResponseError chained via 'from ex'. This is a runtime inference failure, distinct from initialization failures.

Source

Thrown at python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_completion_base.py:93

                input_tokens = self.tokenizer.encode(prompt)
                generator.append_tokens(input_tokens)
            else:
                # With the use of Pybind in ONNX there is currently no way to load images from bytes
                # We can only open images & audios from a file path currently
                if images is not None:
                    images = OnnxRuntimeGenAi.Images.open(*[str(image.uri) for image in images])
                if audios is not None:
                    audios = OnnxRuntimeGenAi.Audios.open(*[str(audio.uri) for audio in audios])
                input_tokens = self.tokenizer(prompt, images=images, audios=audios)
                generator.set_inputs(input_tokens)

            while not generator.is_done():
                generator.generate_next_token()
                new_token_choices = [self.tokenizer_stream.decode(token) for token in generator.get_next_tokens()]
                yield new_token_choices
            del generator
        except Exception as ex:
            raise ServiceInvalidResponseError("Failed Inference with ONNX", ex) from ex

    async def _generate_next_token(
        self,
        prompt: str,
        settings: OnnxGenAIPromptExecutionSettings,
        images: list[ImageContent] | None = None,
        audios: list[AudioContent] | None = None,
    ):
        token_choices: list[str] = []
        async for new_token_choice in self._generate_next_token_async(prompt, settings, images, audios=audios):
            # zip only works if the lists are the same length
            if len(token_choices) == 0:
                token_choices = new_token_choice
            else:
                token_choices = [old_token + new_token for old_token, new_token in zip(token_choices, new_token_choice)]
        return token_choices

View on GitHub (pinned to c028a0c7dc)

Solutions

Inspect the chained exception (__cause__) for the specific ONNX runtime error
If using multimodal, verify all image.uri and audio.uri files exist and are readable at inference time
Check that settings (max_tokens, temperature, etc.) are valid for the model
Monitor GPU/CPU memory during inference and reduce context size if needed

Defensive patterns

Strategy: try-catch

Validate before calling

import os
from semantic_kernel.contents import ImageContent, AudioContent

def validate_multimodal_uris(chat_history):
    for msg in chat_history.messages:
        for item in msg.items:
            if isinstance(item, (ImageContent, AudioContent)):
                if not item.uri or not os.path.exists(str(item.uri).replace('file://', '')):
                    raise ValueError(f'Multimodal content URI missing or file not found: {item.uri}')

Try / catch

from semantic_kernel.exceptions.service_exceptions import ServiceInvalidResponseError

try:
    result = await chat.get_chat_message_contents(chat_history=history, settings=settings)
except ServiceInvalidResponseError as e:
    cause = e.__cause__
    logger.error('ONNX inference failed: %s. Root cause: %s', e, cause)
    raise

Prevention

Verify all image and audio file URIs exist on disk right before inference
Keep multimodal temp files alive (do not garbage-collect) for the duration of the call
Validate OnnxGenAIPromptExecutionSettings search options against model capabilities
Log e.__cause__ to pinpoint the ONNX runtime-level error

When it happens

Trigger: Calling get_chat_message_contents, get_streaming_chat_message_contents, or text completion when the ONNX generator fails during token generation. Common root causes: malformed prompt after template application, image/audio file not found at the uri during Images.open/Audios.open, generator memory exhaustion, or incompatible search options.

Common situations: Image/audio URI pointing at a file that was deleted or moved after construction; setting search options in OnnxGenAIPromptExecutionSettings that the model does not support; passing a prompt that fails encoding; running out of memory with a large model on limited hardware.

Related errors

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/aa8854d630866d6a. Report an issue: GitHub.