cilium/cilium · error

gRPC error: %w

Error message

gRPC error: %w

What it means

While consuming the Hubble GetFlows follow stream, the receive loop receives an error. Canceled gRPC codes are treated as graceful shutdown and return nil; any other gRPC error (Unavailable, Internal, DeadlineExceeded, etc.) aborts the action with 'gRPC error: %w'. This means the flow stream broke mid-test.

Source

Thrown at cilium-cli/connectivity/check/action.go:983

		// Blocks, interruptable by context cancelation.
		res, err := b.Recv()
		if err != nil {
			// Any of the following errors are expected and signal the end
			// of the read loop.
			if errors.Is(err, io.EOF) ||
				errors.Is(err, context.Canceled) ||
				errors.Is(err, context.DeadlineExceeded) {
				a.Debugf("Hubble polling ended: %v", err)
				return nil
			}

			// Return gracefully on 'canceled' gRPC error.
			if status.Code(err) == codes.Canceled {
				a.Debugf("Hubble polling ended: %v", err)
				return nil
			}

			return fmt.Errorf("gRPC error: %w", err)
		}

		switch r := res.GetResponseTypes().(type) {

		case *observer.GetFlowsResponse_NodeStatus:
			// Handle NodeStatus messages generated by Hubble peers, containing
			// individual node readiness, unavailability, invalid filters etc.

			switch r.NodeStatus.StateChange {
			case relay.NodeState_NODE_CONNECTED:
				// Received first connection event from a Hubble peer, tentatively
				// notify the caller that traffic can be generated.
				a.Debugf("Connected to Hubble node(s) %s", r.NodeStatus.NodeNames)
				once.Do(func() { ready <- true })

			case relay.NodeState_NODE_UNAVAILABLE:
				// An unavailable node will result in the event log being incomplete,
				// so the test needs to be aborted.

View on GitHub (pinned to ac7b90affa)

Solutions

1. Read the wrapped gRPC status (status.Code(err)) to identify cause: Unavailable => Relay down, DeadlineExceeded => increase test timeout
2. Check hubble-relay logs: kubectl -n kube-system logs deploy/hubble-relay
3. Re-run the test; transient relay restarts resolve on retry
4. Increase timeouts and keep port-forward/relay stable for the test duration

Example fix

// before
res, err := hubbleClient.GetFlows(ctx, req) // short-lived ctx, fails mid-stream
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()

// ensure stream endpoints are healthy before long polls
st, err := hubbleClient.ServerStatus(ctx, &observer.ServerStatusRequest{})
if err != nil || st == nil {
    return errors.New("relay not healthy for follow stream")
}

err := runSuite(ctx)
if err != nil && strings.Contains(err.Error(), "gRPC error") {
    if status.Code(errors.Unwrap(err)) == codes.Unavailable {
        time.Sleep(10 * time.Second)
        err = runSuite(ctx) // retry once
    }
}

When it happens

Trigger: res.Err() from the stream's response channel returns a non-Canceled gRPC error during the for-loop over flow results — Relay restart, connection reset, context deadline exceeded, or transport failure.

Common situations: Hubble Relay pod restarting during a long test run; idle stream timing out through a proxy/load balancer; context deadline from the test timeout expiring; network interruption between client and Relay.

Related errors

• server stream send was blocked for too long
• initiating follow request: %w
• unable to create hubble client: %w
• not healthy
• no listener configured

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/dea5a4e53cc28fa1. Report an issue: GitHub.