external processor unexpectedly set end of stream in…

Error message

external processor unexpectedly set end of stream in response body mutation

What it means

Raised by recvFromProcServerLoop (ext_proc.go:1462) when a response body mutation from the server carries EndOfStream set in its streamed body response. The client controls stream end, so the server setting EOS inside a response body mutation is rejected; failProcStream fails the RPC unless failure_mode_allow bypasses it.

Solutions

On the server, never set body_mutation.streamed_response.end_of_stream (leave it false); EOS is the client's responsibility.
If the server wants to end the gRPC stream, half-close its own send side instead of mutating EOS.
Enable failure_mode_allow so the client bypasses ext_proc rather than failing the RPC.
Audit the server's BodyResponse construction to ensure end_of_stream is never populated.

Example fix

// before: server sets end of stream inside the body mutation
stream.Send(&procpb.ProcessingResponse{Response: &procpb.ProcessingResponse_ResponseBody{
  ResponseBody: &procpb.BodyResponse{
    Response: &procpb.CommonResponse{
      BodyMutation: &procpb.BodyMutation{BodyMutation: &procpb.BodyMutation_StreamedResponse{
        StreamedResponse: &procpb.StreamedBodyResponse{Body: chunk, EndOfStream: true},
      }},
    },
  },
}})

// after: do not set EndOfStream in the body mutation
StreamedResponse: &procpb.StreamedBodyResponse{Body: chunk}

Defensive patterns

Strategy: fallback

Validate before calling

// On the ext_proc SERVER: never populate end_of_stream on the streamed body.
func buildStreamedBody(chunk []byte) *procpb.StreamedBodyResponse {
    return &procpb.StreamedBodyResponse{Body: chunk} // EndOfStream left false
}

Try / catch

filter.failure_mode_allow = true
if st, ok := status.FromError(err); ok && st.Code() == codes.Internal &&
    strings.Contains(st.Message(), "end of stream in response body mutation") {
    // server incorrectly set end_of_stream inside the body mutation
}

Prevention

Server: leave BodyMutation.StreamedResponse.end_of_stream unset (false).
To end the stream, half-close the server's send side instead.
Enable failure_mode_allow so the RPC survives the violation.
Audit BodyResponse construction for any accidental EndOfStream copy from input.

When it happens

Trigger: Triggered when validateBodyResponse returns a streamedResp whose EndOfStream is true (ext_proc.go:1461) inside a response_body message from the server.

Common situations: Server copies EndOfStream from the request body it received, a handler that signals stream end via the body mutation rather than by simply stopping, or misuse of the BodyResponse.body_mutation.streamed_response.end_of_stream field.

Related errors

AI-assisted analysis of grpc/grpc-go@0c51461d27 (2026-08-11). Data as JSON: /api/errors/2a9fcb2613cb967b. Report an issue: GitHub.

Appendix: source

Thrown at internal/xds/httpfilter/extproc/ext_proc.go:1462

			// RPC.
			if cs.config.processingModes.responseHeaderMode == modeSend && !cs.responseHeadersReady.HasFired() {
				cs.failProcStream(fmt.Errorf("external processor sent response body before sending response headers"))
				return
			}

			// If mutated response trailers have been received before receiving the
			// response body message, fail the RPC.
			if cs.config.processingModes.responseTrailerMode == modeSend && cs.responseTrailerReady.HasFired() {
				cs.failProcStream(fmt.Errorf("external processor sent response body after response trailers were already processed"))
				return
			}

			streamedResp, ok := cs.validateBodyResponse(resp.GetResponseBody())
			if !ok {
				return
			}
			if streamedResp.GetEndOfStream() {
				cs.failProcStream(fmt.Errorf("external processor unexpectedly set end of stream in response body mutation"))
				return
			}
			cs.mutatedRespBuffer.Put(streamedResp)

		case resp.GetResponseHeaders() != nil:
			if cs.config.processingModes.responseHeaderMode == modeSkip {
				cs.failProcStream(fmt.Errorf("external processor unexpectedly sent response headers when response header processing is disabled"))
				return
			}
			if !cs.responseHeaderSent.Load() {
				cs.failProcStream(fmt.Errorf("external processor sent response headers before response headers were sent to it"))
				return
			}
			if cs.responseHeadersReady.HasFired() {
				cs.failProcStream(fmt.Errorf("external processor unexpectedly sent duplicate response headers after response headers were already processed"))
				return
			}

View on GitHub (pinned to 0c51461d27)