cilium/cilium · critical

client failed

Error message

client failed

What it means

ErrClientFailure is a sentinel error representing a non-recoverable failure of the DeletionFallbackClient's underlying Cilium client. Delete, Del, and EndpointDelete return it when the client used to communicate with the local Cilium agent cannot be created or has permanently failed, so endpoint deletion requests cannot be delivered. Callers (and the deletion queue) treat it as terminal rather than retryable within that client instance.

Source

Thrown at plugins/cilium-cni/lib/deletion_queue.go:61

	connectionBackoff time.Duration
}

const (
	// the timeout for connecting and obtaining the lock
	// the default of 30 seconds is too long; kubelet will time us out before then
	timeoutDuration = 1500 * time.Millisecond

	// default backoff interval between two subsequent connection attempts to the agent
	connectionBackoffDefault = 5 * time.Second

	// the maximum number of queued deletions allowed, to protect against kubelet insanity
	maxDeletionFiles = 256
)

var (
	// Indicates a non-recoverable error for DeletionFallbackClient.
	ErrClientFailure = errors.New("client failed")
)

func newCiliumClient(timeout time.Duration) (ciliumClient, error) {
	return client.NewDefaultClientWithTimeout(timeout)
}

// NewDeletionFallbackClient creates a new deletion client.
func NewDeletionFallbackClient(logger *slog.Logger) *DeletionFallbackClient {
	return &DeletionFallbackClient{
		logger: logger,

		deleteQueueDir:      defaults.DeleteQueueDir,
		deleteQueueLockfile: defaults.DeleteQueueLockfile,

		newCiliumClientFn: newCiliumClient,

		connectionBackoff: connectionBackoffDefault,
	}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Ensure the Cilium agent on the node is running and healthy (cilium status; agent logs around the deletion time).
2. Retry the pod deletion after the agent recovers; kubelet will re-issue DEL for stale sandboxes.
3. Check the agent socket/endpoint path and permissions used by client.NewDefaultClientWithTimeout.
4. If failures persist, clean up leftover Cilium endpoints via cilium-dbg endpoint list / delete, and investigate agent crash loops.

func isClientFailure(err error) bool {
    return errors.Is(err, deletionqueue.ErrClientFailure)
}

err := endpointDelete(ctx, epID)
if errors.Is(err, deletionqueue.ErrClientFailure) {
    // non-recoverable client state: recreate client or wait for agent recovery,
    // then retry the deletion rather than reusing the failed client
    client = newClient()
    err = endpointDelete(ctx, epID)
}
if err != nil {
    log.Error("endpoint deletion failed", "err", err)
}

When it happens

Trigger: Returned by Delete/Del/EndpointDelete on DeletionFallbackClient when newCiliumClient (client.NewDefaultClientWithTimeout) fails to construct a working client — e.g. the agent socket/endpoint is unusable — or when an in-flight client is marked permanently failed after unrecoverable request errors.

Common situations: Cilium agent down/restarting during pod teardown so kubelet's DEL cannot reach it; agent socket permissions or path issues; timeouts leaving the fallback client in a failed state; node under heavy load during bulk pod deletion.

Related errors

• received empty configuration object from Cilium agent
• invalid IPAM response, missing addressing
• IPAM did provide neither IPv4 nor IPv6 address
• failed to fetch cilium runtime config: %w
• creating github client: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/33dd6145453b0dd3. Report an issue: GitHub.