dotnet/machinelearning · error · InvalidOperationException

Failed to generate a reply.

Error message

Failed to generate a reply.

What it means

After building the prompt, LlamaCausalLMAgent.GenerateReplyAsync runs the local LLaMA pipeline's Generate method. If Generate returns null (no tokens produced, e.g. input already at or beyond maxLen or the model failed to start), the agent wraps this in InvalidOperationException('Failed to generate a reply.').

Source

Thrown at src/Microsoft.ML.GenAI.LLaMA/LlamaCausalLMAgent.cs:57

    public Task<IMessage> GenerateReplyAsync(IEnumerable<IMessage> messages, GenerateReplyOptions? options = null, CancellationToken cancellationToken = default)
    {
        if (_systemMessage != null)
        {
            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);
            messages = messages.Prepend(systemMessage);
        }
        var input = _templateBuilder.BuildPrompt(messages);
        var maxLen = options?.MaxToken ?? 1024;
        var temperature = options?.Temperature ?? 0.7f;
        var stopTokenSequence = options?.StopSequence ?? [];
        stopTokenSequence = stopTokenSequence.Append("<|eot_id|>").ToArray();

        var output = _pipeline.Generate(
            input,
            maxLen: maxLen,
            temperature: temperature,
            stopSequences: stopTokenSequence) ?? throw new InvalidOperationException("Failed to generate a reply.");

        return Task.FromResult<IMessage>(new TextMessage(Role.Assistant, output, from: this.Name));
    }

#pragma warning disable CS1998 // Async method lacks 'await' operators and will run synchronously
    public async IAsyncEnumerable<IMessage> GenerateStreamingReplyAsync(
#pragma warning restore CS1998 // Async method lacks 'await' operators and will run synchronously
        IEnumerable<IMessage> messages,
        GenerateReplyOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        if (_systemMessage != null)
        {
            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);
            messages = messages.Prepend(systemMessage);
        }
        var input = _templateBuilder.BuildPrompt(messages);
        var maxLen = options?.MaxToken ?? 1024;

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Increase maxLen so it exceeds the prompt token count plus room for the reply.
2. Shorten the conversation history or system prompt to fit the context window.
3. Verify the model weights/pipeline initialized correctly (model path, tokenizer files).
4. Catch the exception and fall back to a smaller history, then retry generation.

Example fix

// before
var reply = await agent.GenerateReplyAsync(messages, maxLen: 256); // long history
// after
var reply = await agent.GenerateReplyAsync(messages.TakeLast(6), maxLen: 2048);

// Estimate prompt tokens; ensure room for the reply before generating.
int promptTokens = tokenizer.CountTokens(prompt);
if (promptTokens + minReplyTokens >= maxLen)
    maxLen = promptTokens + minReplyTokens + 256;

static bool CanGenerate(int promptTokens, int maxLen, int minReply = 64) =>
    promptTokens + minReply < maxLen;

try { reply = await agent.GenerateReplyAsync(messages, maxLen: maxLen); }
catch (InvalidOperationException ex) when (ex.Message == "Failed to generate a reply.")
{ reply = await agent.GenerateReplyAsync(messages.TakeLast(4), maxLen: maxLen * 2); }

When it happens

Trigger: Calling GenerateReplyAsync when the pipeline's Generate returns null — typically when maxLen is smaller than the tokenized prompt length so no new tokens can be produced, an empty/invalid prompt, or a model/pipeline that failed to initialize.

Common situations: maxLen set too small (e.g. default 256) with a long chat history that already fills the context window; model weights not loaded correctly so generation silently yields nothing; very long system prompts pushing input past the limit.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• Please provide a message with content.
• Please provide a message with a valid role. The valid roles
• Invalid role.
• Only text content is supported, but got {item.GetType().Name
• Unsupported role {message.Role}

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cf427b409016a015. Report an issue: GitHub.