external processor unexpectedly sent duplicate response trai

Error message

external processor unexpectedly sent duplicate response trailers after response trailers were already processed

What it means

The grpc-go external-processor (extproc) HTTP filter enforces a strict request/response ordering with the sidecar server. Response trailers may be sent by the server at most once per RPC; a one-shot event (responseTrailerReady) records that they have already been received and applied. If a second ProcessingResponse message carrying response_trailers arrives after that event has fired, the client treats it as a protocol violation and fails the extproc stream.

Source

Thrown at internal/xds/httpfilter/extproc/ext_proc.go:1341

				cs.failProcStream(err)
				return
			}
			// Signal that the response header is modified and ready to be sent to the
			// client, so that if there is any buffered response body, it can be sent
			// after the header.
			cs.fireResponseHeadersReady()

		case resp.GetResponseTrailers() != nil:
			if cs.config.processingModes.responseTrailerMode == modeSkip {
				cs.failProcStream(fmt.Errorf("external processor unexpectedly sent response trailers when response trailer processing is disabled"))
				return
			}
			if !cs.trailerSent.Load() {
				cs.failProcStream(fmt.Errorf("external processor sent response trailers before response trailers were sent to it"))
				return
			}
			if cs.responseTrailerReady.HasFired() {
				cs.failProcStream(fmt.Errorf("external processor unexpectedly sent duplicate response trailers after response trailers were already processed"))
				return
			}
			trailer := resp.GetResponseTrailers()
			if err = cs.applyMutations(trailer.GetHeaderMutation(), cs.responseTrailers); err != nil {
				cs.failProcStream(err)
				return
			}
			// Signal that the response trailer is modified and ready to be sent to
			// the client.
			cs.fireResponseTrailerReady()
		}
	}
}

func (cs *clientStream) validateBodyResponse(bodyResp *v3procservicepb.BodyResponse) (*v3procservicepb.StreamedBodyResponse, bool) {
	if status := bodyResp.GetResponse().GetStatus(); status != v3procservicepb.CommonResponse_CONTINUE {
		cs.failProcStream(fmt.Errorf("external processor returned unexpected status %v for body response, expected %v", status, v3procservicepb.CommonResponse_CONTINUE))
		return nil, false

View on GitHub (pinned to 03255a9237)

Solutions

Audit the external processor server logic and ensure it emits a response_trailers ProcessingResponse at most once per RPC (track a per-stream 'trailersSent' flag).
Verify the server only sends trailer messages after the client forwarded response trailers to it, never proactively or on unrelated messages.
If using an Envoy-built or third-party sidecar, upgrade it; duplicate trailer messages are typically a server bug fixed in newer versions.
Enable failure_mode_allow in the extproc filter config so a misbehaving server bypasses instead of failing RPCs while you fix the server.

Example fix

// before: extproc server re-sends trailers
stream.Send(&procservicepb.ProcessingResponse{Response: &procservicepb.ProcessingResponse_ResponseTrailers{ResponseTrailers: tv}})
// ...later, same stream...
stream.Send(&procservicepb.ProcessingResponse{Response: &procservicepb.ProcessingResponse_ResponseTrailers{ResponseTrailers: tv}})

// after: send response trailers at most once per RPC
if !trailersSent {
    stream.Send(&procservicepb.ProcessingResponse{Response: &procservicepb.ProcessingResponse_ResponseTrailers{ResponseTrailers: tv}})
    trailersSent = true
}

Defensive patterns

Strategy: fallback

Validate before calling

// Client-side guard is limited; the stream-level duplicate is a server behavior.
// Tolerate it via the extproc failure mode:
if pm.FailureModeAllow { /* proc errors bypass instead of failing RPCs */ }

Prevention

Treat the extproc server as stateful: track per-stream flags so each message type is emitted at most once.
Enable failure_mode_allow so a server regression degrades gracefully while you investigate.
Add server-side integration tests that assert response-trailers is sent exactly once per RPC.

When it happens

Trigger: The extproc server sends two distinct ProcessingResponse messages both with ResponseTrailers set during a single RPC, after the client has already processed the first one and fired responseTrailerReady.

Common situations: A buggy or stateful extproc proxy that buffers then re-flushes trailer mutations; a custom sidecar that unconditionally echoes trailers on every recv loop; concurrency in the server where two goroutines write trailers to the same stream.

Related errors

AI-assisted analysis of grpc/grpc-go@03255a9237 (2026-08-07). Data as JSON: /api/errors/8b048cfede83b07c. Report an issue: GitHub.