dotnet/machinelearning · error · InvalidOperationException

Failed to generate a reply.

Error message

Failed to generate a reply.

What it means

The causal LM pipeline's underlying model Generate call returned null, meaning the model produced no text output for the given prompt. The chat client surfaces this as InvalidOperationException because a chat response without content is unusable.

Source

Thrown at src/Microsoft.ML.GenAI.Core/CausalLMPipelineChatClient.cs:45

        IMEAIChatTemplateBuilder chatTemplateBuilder,
        ChatClientMetadata? metadata = null)
    {
        var classNameWithType = $"{nameof(CausalLMPipelineChatClient<TTokenizer, TCausalLMModel>)}<{typeof(TTokenizer).Name}, {typeof(TCausalLMModel).Name}>";
        _metadata = new ChatClientMetadata(providerName: classNameWithType, defaultModelId: typeof(TCausalLMModel).Name);
        _chatTemplateBuilder = chatTemplateBuilder;
        _pipeline = pipeline;
    }

    public virtual Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        var prompt = _chatTemplateBuilder.BuildPrompt(messages, options);
        var stopSequences = options?.StopSequences ?? Array.Empty<string>();

        var output = _pipeline.Generate(
            prompt,
            maxLen: options?.MaxOutputTokens ?? 1024,
            temperature: options?.Temperature ?? 0.7f,
            stopSequences: stopSequences.ToArray()) ?? throw new InvalidOperationException("Failed to generate a reply.");

        var chatMessage = new ChatMessage(ChatRole.Assistant, output);
        return Task.FromResult(new ChatResponse([chatMessage])
        {
            CreatedAt = DateTime.UtcNow,
            FinishReason = ChatFinishReason.Stop,
            ResponseId = Guid.NewGuid().ToString("N"),
        });
    }

#pragma warning disable CS1998 // Async method lacks 'await' operators and will run synchronously
    public virtual async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
#pragma warning restore CS1998 // Async method lacks 'await' operators and will run synchronously
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        var prompt = _chatTemplateBuilder.BuildPrompt(messages, options);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify the prompt after applying the chat template is non-empty and correctly formatted for the loaded model
2. Ensure options.MaxOutputTokens is a positive value (default 1024)
3. Check that stop sequences do not match the beginning of the generated output
4. Confirm the model weights and tokenizer are compatible with the pipeline

Example fix

// before
var response = await chatClient.GetResponseAsync(chatMessages, new ChatOptions { MaxOutputTokens = 0 });
// after
var response = await chatClient.GetResponseAsync(chatMessages, new ChatOptions { MaxOutputTokens = 1024 });

if (string.IsNullOrWhiteSpace(prompt)) throw new ArgumentException("Prompt must not be empty before calling GetResponseAsync");

bool hasOutput = !string.IsNullOrEmpty(_pipeline.Generate(prompt, maxLen, temperature, stopSequences));

try { return await client.GetResponseAsync(messages, options); } catch (InvalidOperationException ex) when (ex.Message == "Failed to generate a reply.") { return ChatResponse with fallback message; }

When it happens

Trigger: Calling GetResponseAsync/GetStreamingResponseAsync on CausalLMPipelineChatClient when the wrapped _pipeline.Generate returns null (e.g. prompt is empty, maxLen truncates to zero output, or the model yields no tokens).

Common situations: Prompt template produces an empty prompt after chat formatting; MaxOutputTokens set to 0; tokenizer/model mismatch causing immediate stop sequence hit.

Understand the failure class

Background: "empty response", "returned no data", "empty embeddings": what HTTP 200-with-empty-body errors mean across libraries — this error's family across 36 libraries.

Related errors

• Logits is null
• Failed to generate a reply.
• The layer count is not enough to cover all layers, did you f
• Invalid state, either qkv_proj or q_proj, k_proj, v_proj sho
• Rope type not implemented

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/ef492854cb30e0a5. Report an issue: GitHub.