cilium/cilium · error

send: %w

Error message

send: %w

What it means

handleObserve wraps a failure from trans.Send() — sending a DiscoveryRequest built from a watcher's observeRequest — as 'send: %w'. Called from the loop goroutine, a non-retriable send error terminates the whole client via 'process loop: %w'. It means the client could not write its subscribe/update request onto the xDS stream.

Source

Thrown at pkg/xds/experimental/client/client.go:370

				req := c.xds.nack(c.node, resp, err)
				err = trans.Send(req)
				if err != nil {
					log.Error("Failed to send NACK", logfields.Error, err)
				}
				backoff.Wait(ctx)
			}
		}
	}
}

// handleObserve creates a flavour-specific request based on observeRequest and sends it on given transport trans.
func (c *XDSClient[ReqT, RespT]) handleObserve(trans transport[ReqT, RespT], obsReq *observeRequest) error {
	req := c.xds.prepareObsReq(obsReq, c.node, c.getAllResources)
	c.log.Debug("Send", logfields.Request, req)

	err := trans.Send(req)
	if err != nil {
		return fmt.Errorf("send: %w", err)
	}
	return nil
}

// handleResponse creates transactions based on flavour-specific responses, applies them to cache.
func (c *XDSClient[ReqT, RespT]) handleResponse(trans transport[ReqT, RespT], resp RespT) error {
	transactions, err := c.xds.tx(resp, c.getAllResources)
	if err != nil {
		return fmt.Errorf("tx: %w", err)
	}
	for _, transaction := range transactions {
		c.log.Debug("cache TX: start",
			logfields.XDSTypeURL, transaction.typeUrl,
			logfields.Upserted, transaction.updated,
			logfields.Deleted, transaction.deleted,
		)
		ver, updated, _ := c.cache.TX(transaction.typeUrl, transaction.updated, transaction.deleted)
		c.log.Debug("cache TX: end",

View on GitHub (pinned to ac7b90affa)

Solutions

1. Check the wrapped gRPC code: Unavailable/Canceled are transient — make sure they are classified retriable so the client reconnects instead of dying.
2. Add gRPC keepalive to keep the stream alive between watcher updates.
3. Verify the stream is still healthy at subscribe time; re-issue the watch after the client reconnects.
4. Check server logs for stream closure or rejection of the request at that timestamp.
5. Confirm the typeUrl in the observeRequest is one the server accepts; some servers close the stream on unsupported types.

Example fix

// before
unsub := cl.AddResourceWatcher(typeUrl, cb) // stream already dead -> send fails, client stops
// after
unsub := cl.AddResourceWatcher(typeUrl, cb)
go func() { if err := <-errCh; isTransient(err) { restartClient() } }() // restart on transient send failure

// re-subscribe watches after reconnect instead of assuming the old stream is live
if !clHealthy.Load() { resubscribeAll() }

func isSendErr(err error) bool { return strings.HasPrefix(err.Error(), "send:") }

err := cl.Run(ctx)
if err != nil && strings.HasPrefix(err.Error(), "send:") {
	if isTransientGRPCErr(err) { scheduleRestartWithBackoff() } else { log.Errorf("fatal send: %v", errors.Unwrap(err)) }
}

When it happens

Trigger: A watcher registers/calls observe and loop calls handleObserve while the gRPC stream is broken or half-closed (server GOAWAY, connection reset, context canceled), so trans.Send returns an rpc error.

Common situations: Watcher added during server restart, stream torn down by idle timeout so the first Send after a quiet period fails, context deadline exceeded while sending on a stalled connection, sending after the server closed the stream.

Related errors

• process loop: %w
• nodeId is empty
• type URL is required for ADS
• mismatching type URL
• unknown type URL

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/e3174b5cea210f44. Report an issue: GitHub.