dotnet/machinelearning · error · InvalidOperationException

Failed to generate a reply.

Error message

Failed to generate a reply.

What it means

MistralCausalLMAgent.GenerateReplyAsync throws InvalidOperationException when the underlying pipeline's Generate call returns null, which the library interprets as generation producing no reply. This guards downstream code (tool-call parsing, StartsWith checks) from a null reference.

Source

Thrown at src/Microsoft.ML.GenAI.Mistral/MistralCausalLMAgent.cs:61

    public Task<IMessage> GenerateReplyAsync(IEnumerable<IMessage> messages, GenerateReplyOptions? options = null, CancellationToken cancellationToken = default)
    {
        if (_systemMessage != null)
        {
            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);
            messages = messages.Prepend(systemMessage);
        }
        var input = _templateBuilder.BuildPrompt(messages, options?.Functions);
        var maxLen = options?.MaxToken ?? 1024;
        var temperature = options?.Temperature ?? 0.7f;
        var stopTokenSequence = options?.StopSequence ?? [];
        stopTokenSequence = stopTokenSequence.Append(_stopSequence).ToArray();

        var output = _pipeline.Generate(
            input,
            maxLen: maxLen,
            temperature: temperature,
            stopSequences: stopTokenSequence) ?? throw new InvalidOperationException("Failed to generate a reply.");

        // post-process the output for tool call
        if (output.StartsWith("[TOOL_CALLS]"))
        {
            return Task.FromResult<IMessage>(ParseAsToolCallMessage(output));
        }

        return Task.FromResult<IMessage>(new TextMessage(Role.Assistant, output, from: this.Name));
    }

#pragma warning disable CS1998 // Async method lacks 'await' operators and will run synchronously
    public async IAsyncEnumerable<IMessage> GenerateStreamingReplyAsync(
#pragma warning restore CS1998 // Async method lacks 'await' operators and will run synchronously
        IEnumerable<IMessage> messages,
        GenerateReplyOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        if (_systemMessage != null)

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Increase maxLen so it exceeds the prompt length and leaves room for generated tokens.
2. Remove or shorten stop sequences that could match immediately at position 0.
3. Verify the model/tokenizer pipeline is correctly initialized and produces tokens for a simple prompt; inspect pipeline output directly.

Example fix

// before
var reply = await agent.GenerateReplyAsync(chatHistory, maxLen: promptTokens.Count); // cannot generate
// after
var reply = await agent.GenerateReplyAsync(chatHistory, maxLen: promptTokens.Count + 512);

// check budget: maxLen must exceed prompt token count
if (maxLen <= promptTokenCount) throw new ArgumentException("maxLen must exceed prompt length to generate a reply.");

try { reply = await agent.GenerateReplyAsync(history, options: opts); } catch (InvalidOperationException ex) when (ex.Message == "Failed to generate a reply.") { reply = fallbackMessage; }

When it happens

Trigger: Calling GenerateReplyAsync when _pipeline.Generate returns null — e.g. generation stopped before any tokens were emitted, stop sequences matched immediately, or maxLen is too small for the prompt plus at least one token.

Common situations: maxLen set equal to or below the prompt token count so the model cannot emit new tokens; a stop sequence equal to the beginning of the model's output; model/tokenizer misconfiguration producing empty output.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• Failed to generate a reply.
• Failed to generate a reply.
• Logits is null
• Failed to generate a reply.
• Failed to deserialize tool calls.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e6221d6ae1409ea8. Report an issue: GitHub.