{"record":{"id":"cf427b409016a015","repo":"dotnet/machinelearning","slug":"failed-to-generate-a-reply-cf427b","errorCode":null,"errorMessage":"Failed to generate a reply.","messagePattern":"Failed to generate a reply\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.LLaMA/LlamaCausalLMAgent.cs","lineNumber":57,"sourceCode":"\n    public Task<IMessage> GenerateReplyAsync(IEnumerable<IMessage> messages, GenerateReplyOptions? options = null, CancellationToken cancellationToken = default)\n    {\n        if (_systemMessage != null)\n        {\n            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);\n            messages = messages.Prepend(systemMessage);\n        }\n        var input = _templateBuilder.BuildPrompt(messages);\n        var maxLen = options?.MaxToken ?? 1024;\n        var temperature = options?.Temperature ?? 0.7f;\n        var stopTokenSequence = options?.StopSequence ?? [];\n        stopTokenSequence = stopTokenSequence.Append(\"<|eot_id|>\").ToArray();\n\n        var output = _pipeline.Generate(\n            input,\n            maxLen: maxLen,\n            temperature: temperature,\n            stopSequences: stopTokenSequence) ?? throw new InvalidOperationException(\"Failed to generate a reply.\");\n\n        return Task.FromResult<IMessage>(new TextMessage(Role.Assistant, output, from: this.Name));\n    }\n\n#pragma warning disable CS1998 // Async method lacks 'await' operators and will run synchronously\n    public async IAsyncEnumerable<IMessage> GenerateStreamingReplyAsync(\n#pragma warning restore CS1998 // Async method lacks 'await' operators and will run synchronously\n        IEnumerable<IMessage> messages,\n        GenerateReplyOptions? options = null,\n        [EnumeratorCancellation] CancellationToken cancellationToken = default)\n    {\n        if (_systemMessage != null)\n        {\n            var systemMessage = new TextMessage(Role.System, _systemMessage, from: this.Name);\n            messages = messages.Prepend(systemMessage);\n        }\n        var input = _templateBuilder.BuildPrompt(messages);\n        var maxLen = options?.MaxToken ?? 1024;","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.LLaMA/LlamaCausalLMAgent.cs#L39-L75","documentation":"After building the prompt, LlamaCausalLMAgent.GenerateReplyAsync runs the local LLaMA pipeline's Generate method. If Generate returns null (no tokens produced, e.g. input already at or beyond maxLen or the model failed to start), the agent wraps this in InvalidOperationException('Failed to generate a reply.').","triggerScenarios":"Calling GenerateReplyAsync when the pipeline's Generate returns null — typically when maxLen is smaller than the tokenized prompt length so no new tokens can be produced, an empty/invalid prompt, or a model/pipeline that failed to initialize.","commonSituations":"maxLen set too small (e.g. default 256) with a long chat history that already fills the context window; model weights not loaded correctly so generation silently yields nothing; very long system prompts pushing input past the limit.","solutions":["Increase maxLen so it exceeds the prompt token count plus room for the reply.","Shorten the conversation history or system prompt to fit the context window.","Verify the model weights/pipeline initialized correctly (model path, tokenizer files).","Catch the exception and fall back to a smaller history, then retry generation."],"exampleFix":"// before\nvar reply = await agent.GenerateReplyAsync(messages, maxLen: 256); // long history\n// after\nvar reply = await agent.GenerateReplyAsync(messages.TakeLast(6), maxLen: 2048);","handlingStrategy":"fallback","validationCode":"// Estimate prompt tokens; ensure room for the reply before generating.\nint promptTokens = tokenizer.CountTokens(prompt);\nif (promptTokens + minReplyTokens >= maxLen)\n    maxLen = promptTokens + minReplyTokens + 256;","typeGuard":"static bool CanGenerate(int promptTokens, int maxLen, int minReply = 64) =>\n    promptTokens + minReply < maxLen;","tryCatchPattern":"try { reply = await agent.GenerateReplyAsync(messages, maxLen: maxLen); }\ncatch (InvalidOperationException ex) when (ex.Message == \"Failed to generate a reply.\")\n{ reply = await agent.GenerateReplyAsync(messages.TakeLast(4), maxLen: maxLen * 2); }","preventionTips":["Set maxLen well above the prompt token count; never rely on small defaults.","Trim chat history to the last N turns to stay inside the context window.","Verify model and tokenizer load successfully at startup with a smoke generation.","Retry once with a truncated history when generation returns empty."],"tags":["llama","local-inference","generation","context-window"],"backgroundTag":"empty-result-set","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}