grpc/grpc-go · error

failed to send client headers to external processor server

Error message

failed to send client headers to external processor server: %v

What it means

Returned in observability mode (ext_proc.go:468) when sending the request headers ProcessingRequest to the ext_proc server fails. It is wrapped via handleInitError, so in deny mode the RPC fails with codes.Internal and in allow mode it bypasses ext_proc.

Solutions

1. Enable failure_mode_allow so a header-send failure degrades to direct dataplane instead of failing the RPC.
2. Inspect ext_proc server logs for the immediate close/panic and fix the handler that rejects the first message.
3. Increase server.Timeout (grpc_service timeout) or remove it if the headers round-trip exceeds it.
4. Verify observability_mode is actually intended — if not, disable it to switch to the normal (synchronous) path.

Example fix

// before: header send failure in observability mode fails the RPC
filter.observability_mode = true
filter.failure_mode_allow = false

// after: tolerate send failures; dataplane proceeds without ext_proc
filter.observability_mode = true
filter.failure_mode_allow = true

// Nothing to validate pre-call; the failure happens on send. Confirm the
// observability mode + timeout are sane to reduce the chance.
func saneObservabilityConfig(f baseConfig) error {
    if f.observabilityMode && f.server.Timeout != 0 && f.server.Timeout < 100*time.Millisecond {
        return fmt.Errorf("observability header send likely to exceed server.Timeout=%v", f.server.Timeout)
    }
    return nil
}

err := conn.Invoke(ctx, method, req, resp)
if st, ok := status.FromError(err); ok && st.Code() == codes.Internal &&
    strings.Contains(st.Message(), "failed to send client headers to external processor") {
    // header send failed in observability mode: enable failure_mode_allow to bypass
}

When it happens

Trigger: Triggered when requestHeaderMode is SEND (observability mode, ext_proc.go:466) and ocs.sendToProcessor(ocs.requestHeaders(...)) returns a non-nil error — e.g. the proc stream died, write after close, context deadline, or transport reset right after stream creation.

Common situations: Ext_proc server closes the stream immediately after accept (misconfigured handler), context deadline too short for the configured timeout, connection RST during header send, or server-side panic aborting the stream.

Related errors

• failed to create a stream to external processor
• external processor returned invalid body mutation in body…
• external processor returned unexpected status
• external processor returned unexpected status
• external processor sent response body after response…

AI-assisted analysis of grpc/grpc-go@0c51461d27 (2026-08-11). Data as JSON: /api/errors/695dc2e5c5800b45. Report an issue: GitHub.

Appendix: source

Thrown at internal/xds/httpfilter/extproc/ext_proc.go:468

		onFinishFunc := func(error) {
			time.AfterFunc(ocs.config.deferredCloseTimeout, ocs.procCancel)
		}
		newOpts := append(opts, grpc.OnFinish(onFinishFunc))

		if ocs.dataplaneStream, err = newStream(ocs.ctx, newOpts...); err != nil {
			ocs.cancel()
			return nil, ocs.streamError(err)
		}
		ocs.recordMetric(clientHeadersDurationMetric, timeSince(ocs.clientHeadersStartTime).Seconds())
		// Start background goroutine to receive any messages from the external
		// processor server and discard them.
		go ocs.discardProcessorResponsesLoop()

		// If the request header processing mode is set to "Send", forward the
		// headers to the external processor server.
		if i.config.processingModes.requestHeaderMode == modeSend {
			if err = ocs.sendToProcessor(ocs.requestHeaders(outgoingMD, added)); err != nil {
				return ocs.handleInitError(fmt.Errorf("failed to send client headers to external processor server: %v", err), newStream, opts...)
			}
		}

		return ocs, nil
	}

	// Normal mode.
	cs := &clientStream{
		commonStream:             csCommon,
		procStreamFailed:         grpcsync.NewEvent(),
		procStreamBypass:         grpcsync.NewEvent(),
		mutatedReqBuffer:         buffer.NewUnbounded[*v3procservicepb.StreamedBodyResponse](),
		mutatedRespBuffer:        buffer.NewUnbounded[*v3procservicepb.StreamedBodyResponse](),
		responseHeadersReady:     grpcsync.NewEvent(),
		responseTrailerReady:     grpcsync.NewEvent(),
		dataplaneSetup:           make(chan struct{}),
		procSendCh:               make(chan *v3procservicepb.ProcessingRequest),
		requestForwardLoopDoneCh: make(chan struct{}),

View on GitHub (pinned to 0c51461d27)