{"record":{"id":"bae98ccfd3c86ab8","repo":"cilium/cilium","slug":"failed-to-wait-for-endpoint-restoration-w-bae98c","errorCode":null,"errorMessage":"failed to wait for endpoint restoration: %w","messagePattern":"failed to wait for endpoint restoration: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/endpointcleanup/cleanup.go","lineNumber":114,"sourceCode":"\t\tjob.OneShot(\"endpoint-cleanup\", func(ctx context.Context, health cell.Health) error {\n\t\t\treturn cleanup.run(ctx)\n\t\t}),\n\t)\n}\n\nfunc (c *cleanup) run(ctx context.Context) error {\n\t// Use restored endpoints to delete local CiliumEndpoints which are not in the restored endpoint cache.\n\t// This will clear out any CiliumEndpoints that may be stale.\n\t// Likely causes for this are Pods having their init container restarted or the node being restarted.\n\t// This must wait for both K8s watcher caches to be synced and local endpoint restoration to be complete.\n\t// Note: Synchronization of endpoints to their CEPs may not be complete at this point, but we only have to\n\t// know what endpoints exist post-restoration in our endpointManager cache to perform cleanup.\n\trestorer, err := c.restorerPromise.Await(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to wait for endpoint restorer promise: %w\", err)\n\t}\n\tif err := restorer.WaitForEndpointRestore(ctx); err != nil {\n\t\treturn fmt.Errorf(\"failed to wait for endpoint restoration: %w\", err)\n\t}\n\n\tvar (\n\t\tretries int\n\t\tbo      = wait.Backoff{\n\t\t\tDuration: 500 * time.Millisecond,\n\t\t\tFactor:   1,\n\t\t\tJitter:   0.1,\n\t\t\tSteps:    5,\n\t\t\tCap:      0,\n\t\t}\n\t)\n\terr = wait.ExponentialBackoffWithContext(ctx, bo, func(ctx context.Context) (done bool, err error) {\n\t\tif c.ciliumEndpointSliceEnabled {\n\t\t\terr = c.cleanStaleCESs(ctx)\n\t\t} else {\n\t\t\terr = c.cleanStaleCEPs(ctx)\n\t\t}","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/endpointcleanup/cleanup.go#L96-L132","documentation":"After obtaining the Restorer, the cleanup job calls restorer.WaitForEndpointRestore(ctx), which blocks until local endpoint restoration finishes. This error indicates that wait failed — typically ctx cancellation during shutdown, or the restorer signaling restoration failed — so stale CiliumEndpoint cleanup is skipped. Skipping is safe-by-design (no CEPs are deleted on uncertain state) but stale CEPs may linger.","triggerScenarios":"restorer.WaitForEndpointRestore(ctx) returns an error in cleanup.run: agent context cancelled mid-restore (shutdown/kill), restore loop hit an unrecoverable error, or a timeout elapsed on a node with a large endpoint count.","commonSituations":"Node reboot or agent upgrade interrupting restoration; pod init-container restarts prolonging restore beyond operator patience; leftover stale CEPs from a previous agent run that never got cleaned because this wait aborted.","solutions":["Check the wrapped cause: context.Canceled at shutdown is expected and harmless; other causes indicate real restore failure","Fix the underlying restore failure (state directory, bpffs, filesystem permissions on /var/run/cilium) and restart the agent","Manually delete known-stale CiliumEndpoint objects (kubectl delete cep) if automated cleanup keeps being skipped","Give the agent time on restart; avoid killing cilium during the restore window so cleanup can complete once"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if errors.Is(err, context.Canceled) {\n    return nil // expected at shutdown; rerun cleanup on next agent start\n}","typeGuard":"func isContextErr(err error) bool {\n    return errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)\n}","tryCatchPattern":"if err := restorer.WaitForEndpointRestore(ctx); err != nil {\n    if isContextErr(err) {\n        log.Info(\"restoration wait cancelled; skipping stale CEP cleanup\")\n        return nil\n    }\n    log.Error(\"endpoint restoration failed; stale CEP cleanup skipped\", \"error\", err)\n}","preventionTips":["Fix restore root causes (state dir persistence, bpffs mount, CT maps) so waits succeed","Schedule agent restarts/upgrades outside traffic peaks; allow full restore before SIGKILL","Periodically reconcile stale CiliumEndpoints manually or via a lower-risk GC if this wait aborts repeatedly","Give large nodes adequate CPU/memory so restore finishes within expected windows"],"tags":["cilium","kubernetes","endpoint-restore","context-cancelled","stale-ceps"],"backgroundTag":"endpoint-restore-wait-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}