{"record":{"id":"3e2c104f6ad599b5","repo":"cilium/cilium","slug":"recv-w","errorCode":null,"errorMessage":"recv: %w","messagePattern":"recv: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/xds/experimental/client/client.go","lineNumber":271,"sourceCode":"}\n\n// fetchResponses will pass messages from Recv() calls to queue until Context ctx is done.\nfunc (c *XDSClient[ReqT, RespT]) fetchResponses(ctx context.Context, errCh chan error, trans transport[ReqT, RespT]) {\n\tdefer close(errCh)\n\tlog := c.log.With(logfields.Hint, \"fetch-responses\")\n\tbackoff := backoff.Exponential{\n\t\tLogger:     c.log,\n\t\tMin:        c.opts.RetryBackoff.Min,\n\t\tMax:        c.opts.RetryBackoff.Max,\n\t\tResetAfter: c.opts.RetryBackoff.Reset,\n\t\tJitter:     true,\n\t\tName:       \"xds-client-fetch-responses\",\n\t}\n\tfor {\n\t\tresp, err := trans.Recv()\n\t\tif err != nil {\n\t\t\tlog.Error(\"Failed to receive message\", logfields.Error, err)\n\t\t\terr = fmt.Errorf(\"recv: %w\", err)\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase errCh <- err:\n\t\t\t\tbackoff.Wait(ctx)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn\n\t\tcase c.responseQueue <- resp:\n\t\t}\n\t}\n}\n\nfunc (c *XDSClient[ReqT, RespT]) getAllResources(typeUrl string) *xds.VersionedResources {","sourceCodeStart":253,"sourceCodeEnd":289,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/xds/experimental/client/client.go#L253-L289","documentation":"fetchResponses wraps a failure from trans.Recv() — the receive half of the xDS gRPC stream — as 'recv: %w', logs it at error level, and pushes it to the error channel after waiting on backoff. If the wrapped error passes isRetriableErr, process() keeps the client alive; otherwise Run terminates. It signals the server closed or broke the response stream.","triggerScenarios":"The xDS server closes the stream (goaway, restart, idle timeout), a proxy terminates long-lived gRPC streams, keepalive/HTTP2 idle timeouts fire, or the connection drops mid-stream, causing Recv to return io.EOF or an rpc error.","commonSituations":"Idle connection reaped by an L4 load balancer (common with default 350s TCP idle timeouts), Envoy/Istio xDS server redeploy, HTTP/2 GOAWAY during server upgrade, NAT/firewall dropping long-lived connections.","solutions":["Configure gRPC keepalive parameters (Time ~20-30s, PermitWithoutStream) so intermediaries do not reap the idle stream.","Ensure isRetriableErr covers codes.Unavailable/DeadlineExceeded/Canceled so transient recv failures back off and retry instead of stopping Run.","If Recv returns io.EOF, treat as server close and reconnect via the backoff loop rather than fataling.","Check L7/L4 proxies in the path and raise their idle/stream timeouts.","Watch server logs for GOAWAY or restarts coinciding with the failure time."],"exampleFix":"// before\ngrpc.Dial(addr, grpc.WithTransportCredentials(creds)) // default keepalive; LB kills idle stream\n// after\ngrpc.Dial(addr, grpc.WithTransportCredentials(creds),\n\tgrpc.WithKeepaliveParams(keepalive.ClientParameters{\n\t\tTime: 20 * time.Second, Timeout: 10 * time.Second, PermitWithoutStream: true,\n\t}))","handlingStrategy":"retry","validationCode":"// pre-flight: confirm the stream path tolerates long-lived connections\n// enable keepalive params before Dial\nkp := keepalive.ClientParameters{Time: 20 * time.Second, Timeout: 10 * time.Second, PermitWithoutStream: true}","typeGuard":"func isRecvErr(err error) bool { return strings.HasPrefix(err.Error(), \"recv:\") }","tryCatchPattern":"err := cl.Run(ctx)\nif err != nil && strings.HasPrefix(err.Error(), \"recv:\") {\n\tif isTransientGRPCErr(err) || errors.Is(err, io.EOF) { scheduleRestartWithBackoff() }\n}","preventionTips":["Always configure gRPC client keepalive (20-30s) with PermitWithoutStream.","Raise idle/stream timeouts on load balancers and proxies in the path.","Ensure io.EOF from Recv is treated as a reconnectable server close.","Monitor server restarts/GOAWAYs and expect brief reconnections during deploys."],"tags":["grpc","xds","stream","keepalive","network"],"backgroundTag":"grpc-unavailable","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}