{"record":{"id":"33dd6145453b0dd3","repo":"cilium/cilium","slug":"client-failed","errorCode":null,"errorMessage":"client failed","messagePattern":"client failed","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"plugins/cilium-cni/lib/deletion_queue.go","lineNumber":61,"sourceCode":"\n\tconnectionBackoff time.Duration\n}\n\nconst (\n\t// the timeout for connecting and obtaining the lock\n\t// the default of 30 seconds is too long; kubelet will time us out before then\n\ttimeoutDuration = 1500 * time.Millisecond\n\n\t// default backoff interval between two subsequent connection attempts to the agent\n\tconnectionBackoffDefault = 5 * time.Second\n\n\t// the maximum number of queued deletions allowed, to protect against kubelet insanity\n\tmaxDeletionFiles = 256\n)\n\nvar (\n\t// Indicates a non-recoverable error for DeletionFallbackClient.\n\tErrClientFailure = errors.New(\"client failed\")\n)\n\nfunc newCiliumClient(timeout time.Duration) (ciliumClient, error) {\n\treturn client.NewDefaultClientWithTimeout(timeout)\n}\n\n// NewDeletionFallbackClient creates a new deletion client.\nfunc NewDeletionFallbackClient(logger *slog.Logger) *DeletionFallbackClient {\n\treturn &DeletionFallbackClient{\n\t\tlogger: logger,\n\n\t\tdeleteQueueDir:      defaults.DeleteQueueDir,\n\t\tdeleteQueueLockfile: defaults.DeleteQueueLockfile,\n\n\t\tnewCiliumClientFn: newCiliumClient,\n\n\t\tconnectionBackoff: connectionBackoffDefault,\n\t}","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/plugins/cilium-cni/lib/deletion_queue.go#L43-L79","documentation":"ErrClientFailure is a sentinel error representing a non-recoverable failure of the DeletionFallbackClient's underlying Cilium client. Delete, Del, and EndpointDelete return it when the client used to communicate with the local Cilium agent cannot be created or has permanently failed, so endpoint deletion requests cannot be delivered. Callers (and the deletion queue) treat it as terminal rather than retryable within that client instance.","triggerScenarios":"Returned by Delete/Del/EndpointDelete on DeletionFallbackClient when newCiliumClient (client.NewDefaultClientWithTimeout) fails to construct a working client — e.g. the agent socket/endpoint is unusable — or when an in-flight client is marked permanently failed after unrecoverable request errors.","commonSituations":"Cilium agent down/restarting during pod teardown so kubelet's DEL cannot reach it; agent socket permissions or path issues; timeouts leaving the fallback client in a failed state; node under heavy load during bulk pod deletion.","solutions":["Ensure the Cilium agent on the node is running and healthy (cilium status; agent logs around the deletion time).","Retry the pod deletion after the agent recovers; kubelet will re-issue DEL for stale sandboxes.","Check the agent socket/endpoint path and permissions used by client.NewDefaultClientWithTimeout.","If failures persist, clean up leftover Cilium endpoints via cilium-dbg endpoint list / delete, and investigate agent crash loops."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func isClientFailure(err error) bool {\n    return errors.Is(err, deletionqueue.ErrClientFailure)\n}","tryCatchPattern":"err := endpointDelete(ctx, epID)\nif errors.Is(err, deletionqueue.ErrClientFailure) {\n    // non-recoverable client state: recreate client or wait for agent recovery,\n    // then retry the deletion rather than reusing the failed client\n    client = newClient()\n    err = endpointDelete(ctx, epID)\n}\nif err != nil {\n    log.Error(\"endpoint deletion failed\", \"err\", err)\n}","preventionTips":["Keep the Cilium agent healthy on nodes with high pod churn.","Verify agent socket path/permissions in the CNI environment.","Use sentinel errors.Is checks instead of string matching.","Re-queue failed DELs so kubelet/controller retries after agent recovery."],"tags":["cilium-agent","client","deletion","sentinel-error"],"backgroundTag":"agent-unreachable","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}