{"record":{"id":"dea5a4e53cc28fa1","repo":"cilium/cilium","slug":"grpc-error-w","errorCode":null,"errorMessage":"gRPC error: %w","messagePattern":"gRPC error: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cilium-cli/connectivity/check/action.go","lineNumber":983,"sourceCode":"\t\t// Blocks, interruptable by context cancelation.\n\t\tres, err := b.Recv()\n\t\tif err != nil {\n\t\t\t// Any of the following errors are expected and signal the end\n\t\t\t// of the read loop.\n\t\t\tif errors.Is(err, io.EOF) ||\n\t\t\t\terrors.Is(err, context.Canceled) ||\n\t\t\t\terrors.Is(err, context.DeadlineExceeded) {\n\t\t\t\ta.Debugf(\"Hubble polling ended: %v\", err)\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\t// Return gracefully on 'canceled' gRPC error.\n\t\t\tif status.Code(err) == codes.Canceled {\n\t\t\t\ta.Debugf(\"Hubble polling ended: %v\", err)\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\treturn fmt.Errorf(\"gRPC error: %w\", err)\n\t\t}\n\n\t\tswitch r := res.GetResponseTypes().(type) {\n\n\t\tcase *observer.GetFlowsResponse_NodeStatus:\n\t\t\t// Handle NodeStatus messages generated by Hubble peers, containing\n\t\t\t// individual node readiness, unavailability, invalid filters etc.\n\n\t\t\tswitch r.NodeStatus.StateChange {\n\t\t\tcase relay.NodeState_NODE_CONNECTED:\n\t\t\t\t// Received first connection event from a Hubble peer, tentatively\n\t\t\t\t// notify the caller that traffic can be generated.\n\t\t\t\ta.Debugf(\"Connected to Hubble node(s) %s\", r.NodeStatus.NodeNames)\n\t\t\t\tonce.Do(func() { ready <- true })\n\n\t\t\tcase relay.NodeState_NODE_UNAVAILABLE:\n\t\t\t\t// An unavailable node will result in the event log being incomplete,\n\t\t\t\t// so the test needs to be aborted.","sourceCodeStart":965,"sourceCodeEnd":1001,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/cilium-cli/connectivity/check/action.go#L965-L1001","documentation":"While consuming the Hubble GetFlows follow stream, the receive loop receives an error. Canceled gRPC codes are treated as graceful shutdown and return nil; any other gRPC error (Unavailable, Internal, DeadlineExceeded, etc.) aborts the action with 'gRPC error: %w'. This means the flow stream broke mid-test.","triggerScenarios":"res.Err() from the stream's response channel returns a non-Canceled gRPC error during the for-loop over flow results — Relay restart, connection reset, context deadline exceeded, or transport failure.","commonSituations":"Hubble Relay pod restarting during a long test run; idle stream timing out through a proxy/load balancer; context deadline from the test timeout expiring; network interruption between client and Relay.","solutions":["Read the wrapped gRPC status (status.Code(err)) to identify cause: Unavailable => Relay down, DeadlineExceeded => increase test timeout","Check hubble-relay logs: kubectl -n kube-system logs deploy/hubble-relay","Re-run the test; transient relay restarts resolve on retry","Increase timeouts and keep port-forward/relay stable for the test duration"],"exampleFix":"// before\nres, err := hubbleClient.GetFlows(ctx, req) // short-lived ctx, fails mid-stream\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)\ndefer cancel()","handlingStrategy":"retry","validationCode":"// ensure stream endpoints are healthy before long polls\nst, err := hubbleClient.ServerStatus(ctx, &observer.ServerStatusRequest{})\nif err != nil || st == nil {\n    return errors.New(\"relay not healthy for follow stream\")\n}","typeGuard":null,"tryCatchPattern":"err := runSuite(ctx)\nif err != nil && strings.Contains(err.Error(), \"gRPC error\") {\n    if status.Code(errors.Unwrap(err)) == codes.Unavailable {\n        time.Sleep(10 * time.Second)\n        err = runSuite(ctx) // retry once\n    }\n}","preventionTips":["Keep relay and agents stable for the test duration (no drains/upgrades)","Set test timeouts comfortably above expected runtime","Beware proxies/LBs dropping idle gRPC streams; enable keepalive","Monitor hubble-relay logs during runs"],"tags":["grpc","hubble","stream","relay"],"backgroundTag":"grpc-stream-error","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}