cilium/cilium · error

recv: %w

Error message

recv: %w

What it means

fetchResponses wraps a failure from trans.Recv() — the receive half of the xDS gRPC stream — as 'recv: %w', logs it at error level, and pushes it to the error channel after waiting on backoff. If the wrapped error passes isRetriableErr, process() keeps the client alive; otherwise Run terminates. It signals the server closed or broke the response stream.

Source

Thrown at pkg/xds/experimental/client/client.go:271

}

// fetchResponses will pass messages from Recv() calls to queue until Context ctx is done.
func (c *XDSClient[ReqT, RespT]) fetchResponses(ctx context.Context, errCh chan error, trans transport[ReqT, RespT]) {
	defer close(errCh)
	log := c.log.With(logfields.Hint, "fetch-responses")
	backoff := backoff.Exponential{
		Logger:     c.log,
		Min:        c.opts.RetryBackoff.Min,
		Max:        c.opts.RetryBackoff.Max,
		ResetAfter: c.opts.RetryBackoff.Reset,
		Jitter:     true,
		Name:       "xds-client-fetch-responses",
	}
	for {
		resp, err := trans.Recv()
		if err != nil {
			log.Error("Failed to receive message", logfields.Error, err)
			err = fmt.Errorf("recv: %w", err)
			select {
			case <-ctx.Done():
				return
			case errCh <- err:
				backoff.Wait(ctx)
			}
			continue
		}

		select {
		case <-ctx.Done():
			return
		case c.responseQueue <- resp:
		}
	}
}

func (c *XDSClient[ReqT, RespT]) getAllResources(typeUrl string) *xds.VersionedResources {

View on GitHub (pinned to ac7b90affa)

Solutions

1. Configure gRPC keepalive parameters (Time ~20-30s, PermitWithoutStream) so intermediaries do not reap the idle stream.
2. Ensure isRetriableErr covers codes.Unavailable/DeadlineExceeded/Canceled so transient recv failures back off and retry instead of stopping Run.
3. If Recv returns io.EOF, treat as server close and reconnect via the backoff loop rather than fataling.
4. Check L7/L4 proxies in the path and raise their idle/stream timeouts.
5. Watch server logs for GOAWAY or restarts coinciding with the failure time.

Example fix

// before
grpc.Dial(addr, grpc.WithTransportCredentials(creds)) // default keepalive; LB kills idle stream
// after
grpc.Dial(addr, grpc.WithTransportCredentials(creds),
	grpc.WithKeepaliveParams(keepalive.ClientParameters{
		Time: 20 * time.Second, Timeout: 10 * time.Second, PermitWithoutStream: true,
	}))

// pre-flight: confirm the stream path tolerates long-lived connections
// enable keepalive params before Dial
kp := keepalive.ClientParameters{Time: 20 * time.Second, Timeout: 10 * time.Second, PermitWithoutStream: true}

func isRecvErr(err error) bool { return strings.HasPrefix(err.Error(), "recv:") }

err := cl.Run(ctx)
if err != nil && strings.HasPrefix(err.Error(), "recv:") {
	if isTransientGRPCErr(err) || errors.Is(err, io.EOF) { scheduleRestartWithBackoff() }
}

When it happens

Trigger: The xDS server closes the stream (goaway, restart, idle timeout), a proxy terminates long-lived gRPC streams, keepalive/HTTP2 idle timeouts fire, or the connection drops mid-stream, causing Recv to return io.EOF or an rpc error.

Common situations: Idle connection reaped by an L4 load balancer (common with default 350s TCP idle timeouts), Envoy/Istio xDS server redeploy, HTTP/2 GOAWAY during server upgrade, NAT/firewall dropping long-lived connections.

Related errors

• server stream send was blocked for too long
• start transport: %w
• process responses: %w
• ACK send: %w
• nodeId is empty

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/3e2c104f6ad599b5. Report an issue: GitHub.