dotnet/machinelearning · error · InvalidOperationException

Failed to generate a reply.

Error message

Failed to generate a reply.

What it means

Phi3CausalLMAgent.GenerateReplyAsync calls the underlying pipeline's Generate, which returns null when no text could be produced. The agent converts that null into InvalidOperationException('Failed to generate a reply.') so callers get a clear failure instead of a null message.

Source

Thrown at src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMAgent.cs:56

    public Task<IMessage> GenerateReplyAsync(IEnumerable<IMessage> messages, GenerateReplyOptions? options = null, CancellationToken cancellationToken = default)
    {
        if (_systemMessage != null)
        {
            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);
            messages = messages.Prepend(systemMessage);
        }

        var input = _templateBuilder.BuildPrompt(messages);
        var maxLen = options?.MaxToken ?? 1024;
        var temperature = options?.Temperature ?? 0.7f;
        var stopTokenSequence = options?.StopSequence ?? [];
        stopTokenSequence = stopTokenSequence.Append("<|end|>").ToArray();

        var output = _pipeline.Generate(
            input,
            maxLen: maxLen,
            temperature: temperature,
            stopSequences: stopTokenSequence) ?? throw new InvalidOperationException("Failed to generate a reply.");

        return Task.FromResult<IMessage>(new TextMessage(Role.Assistant, output, from: this.Name));
    }

#pragma warning disable CS1998 // Async method lacks 'await' operators and will run synchronously
    public async IAsyncEnumerable<IMessage> GenerateStreamingReplyAsync(
#pragma warning restore CS1998 // Async method lacks 'await' operators and will run synchronously
        IEnumerable<IMessage> messages,
        GenerateReplyOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        if (_systemMessage != null)
        {
            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);
            messages = messages.Prepend(systemMessage);
        }

        var input = _templateBuilder.BuildPrompt(messages);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Inspect the input prompt/chat template — ensure it is non-empty and correctly formatted for Phi-3.
2. Remove or relax custom stopSequences so generation is not stopped before any token is emitted.
3. Increase maxLen (and verify temperature is in a valid range) and retry.
4. Verify the model weights and tokenizer files load correctly and match the Phi-3 version you target.

Example fix

// before
var reply = await agent.GenerateReplyAsync(history, new Phi3AgentRequestOptions { MaxLen = 1 });
// after
var reply = await agent.GenerateReplyAsync(history, new Phi3AgentRequestOptions { MaxLen = 1024 });

if (string.IsNullOrWhiteSpace(prompt) && history.All(m => string.IsNullOrWhiteSpace(m.GetContent())))
    throw new InvalidOperationException("Chat history is empty; generation would produce no output.");

try { var reply = await agent.GenerateReplyAsync(history, options); } catch (InvalidOperationException ex) when (ex.Message == "Failed to generate a reply.") { logger.LogWarning(ex, "Generation produced no output"); return fallbackMessage; }

When it happens

Trigger: Calling GenerateReplyAsync with an empty prompt, a stop sequence that matches the very first generated token (note '<|end|>' is always appended), or a maxLen/temperature configuration that yields no output tokens.

Common situations: Prompt built from an empty chat history; stopSequences supplied by the caller already containing '<|end|>' plus overly aggressive custom stops; model weights/tokenizer mismatch producing garbage that the pipeline filters out; maxLen set too small to produce any token.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• Failed to generate a reply.
• Logits is null
• Failed to generate a reply.
• Please provide a message with content.
• Please provide a message with a valid role. The valid roles

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/21db878efc2e0876. Report an issue: GitHub.