microsoft/autogen · error · InvalidOperationException

The response should contain either text or tool calls.

Error message

The response should contain either text or tool calls.

What it means

Thrown by GeminiMessageConnector while aggregating a streaming Gemini response: the bucket of GenerateContentResponse chunks contained neither pure-text parts nor any function-call parts. The connector can only fold streams into TextMessage or ToolCallMessage, so any other part kind (or an inconsistent mix) makes it throw InvalidOperationException.

Source

Thrown at dotnet/src/AutoGen.Gemini/Middleware/GeminiMessageConnector.cs:108

                    var functionCallParts = bucket.Where(m => m.Candidates.Count == 1 && m.Candidates[0].Content.Parts.Count == 1 && m.Candidates[0].Content.Parts[0].DataCase == Part.DataOneofCase.FunctionCall)
                        .Select(m => m.Candidates[0].Content.Parts[0]).ToList();

                    var toolCalls = new List<ToolCall>();
                    foreach (var part in functionCallParts)
                    {
                        var fc = part.FunctionCall;
                        var toolCall = new ToolCall(fc.Name, fc.Args.ToString());

                        toolCalls.Add(toolCall);
                    }

                    var toolCallMessage = new ToolCallMessage(toolCalls, agent.Name);

                    yield return toolCallMessage;
                }
                else
                {
                    throw new InvalidOperationException("The response should contain either text or tool calls.");
                }
            }
        }
    }

    public async Task<IMessage> InvokeAsync(MiddlewareContext context, IAgent agent, CancellationToken cancellationToken = default)
    {
        var messages = ProcessMessage(context.Messages, agent);
        var reply = await agent.GenerateReplyAsync(messages, context.Options, cancellationToken);

        return reply switch
        {
            Core.IMessage<GenerateContentResponse> m => PostProcessMessage(m.Content, agent),
            _ when strictMode => throw new InvalidOperationException($"Unsupported message type: {reply.GetType()}"),
            _ => reply,
        };
    }

View on GitHub (pinned to 027ecf0a37)

Solutions

1. Restrict the request so replies contain only text or function calls (don't request multimodal output with this connector)
2. Upgrade AutoGen.Gemini — newer connectors handle mixed/multimodal streams
3. Catch InvalidOperationException around streaming and fall back to non-streaming GenerateContentAsync for that turn

Example fix

// before: request asks Gemini for image output -> stream has inlineData parts -> throw
// after: text-only generation config
request.GenerationConfig = new GenerationConfig { ResponseModalities = { Modality.Text } };

// Before requesting, constrain output modality to text
request.GenerationConfig ??= new GenerationConfig();
request.GenerationConfig.ResponseMimeTypes.Clear();
request.GenerationConfig.ResponseModalities.Clear();
request.GenerationConfig.ResponseModalities.Add(Modality.Text);

static bool StreamIsFoldable(IEnumerable<GenerateContentResponse> chunks) =>
    chunks.All(c => c.Candidates.Count == 1 &&
                    c.Candidates[0].Content.Parts.All(p => p.DataCase is Part.DataOneofCase.Text or Part.DataOneofCase.FunctionCall));

try { await foreach (var m in geminiAgent.GenerateStreamingReplyAsync(msgs, ct)) { } }
catch (InvalidOperationException e) when (e.Message == "The response should contain either text or tool calls.")
{ /* fall back to non-streaming GenerateReplyAsync for this turn */ }

When it happens

Trigger: A Gemini stream where chunks carry parts whose DataCase is neither Text nor FunctionCall — e.g. inlineData/image parts, embedded metadata, or a stream mixing a text chunk with a function-call chunk such that neither the All(text) nor Any(functionCall) predicate holds.

Common situations: Multimodal replies (images/binary parts) from Gemini models, function-calling streams that also emit a text preamble chunk, or API version changes introducing new part kinds the connector does not recognize.

Related errors

• Unsupported message type: {reply.GetType()}
• Only one choice is supported in streaming response
• Please set OPENAI_API_KEY environment variable.
• Failed to deserialize response
• Value was null.

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/6093761043e1f8ec. Report an issue: GitHub.