{"record":{"id":"1b9e392c083d0769","repo":"cilium/cilium","slug":"failed-to-get-ciliumendpoint-store-w","errorCode":null,"errorMessage":"failed to get CiliumEndpoint store: %w","messagePattern":"failed to get CiliumEndpoint store: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/endpointcleanup/cleanup.go","lineNumber":157,"sourceCode":"\t\t\t)\n\t\t\tif resiliency.IsRetryable(err) {\n\t\t\t\treturn false, nil\n\t\t\t}\n\t\t\treturn true, err\n\t\t}\n\t\treturn true, nil\n\t})\n\tif err != nil {\n\t\tc.log.Error(\"Failed to clean up stale CEPs after multiple attempts\", logfields.Error, err)\n\t}\n\treturn err\n}\n\nfunc (c *cleanup) cleanStaleCEPs(ctx context.Context) error {\n\tvar errs error\n\tstore, err := c.ciliumEndpoint.Store(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get CiliumEndpoint store: %w\", err)\n\t}\n\n\tln, err := c.localNodeStore.Get(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get local node: %w\", err)\n\t}\n\n\tobjs, err := store.ByIndex(\"localNode\", node.GetCiliumEndpointNodeIP(ln))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get indexed CiliumEndpointSlice from store: %w\", err)\n\t}\n\tfor _, cep := range objs {\n\t\tif cep.Networking.NodeIP == node.GetCiliumEndpointNodeIP(ln) && c.endpointsCache.LookupCEPName(cep.Namespace+\"/\"+cep.Name) == nil {\n\t\t\tif err := c.deleteCiliumEndpoint(ctx, cep.Namespace, cep.Name, &cep.ObjectMeta.UID); err != nil {\n\t\t\t\terrs = errors.Join(errs, err)\n\t\t\t}\n\t\t}\n\t}","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/endpointcleanup/cleanup.go#L139-L175","documentation":"In Cilium's endpoint cleanup job, cleanStaleCEPs calls ciliumEndpoint.Store(ctx) to obtain a synced, read-only store of CiliumEndpoint resources. This error wraps the failure of that call: the resource's informer could not start or synchronize its cache within the given context (or the context was canceled). It is thrown so the startup GC job can abort and the caller (run) can decide to retry via resiliency.IsRetryable.","triggerScenarios":"resource.Resource[*types.CiliumEndpoint].Store(ctx) returns an error when the CiliumEndpoints informer fails to start/sync: the CiliumEndpoint CRD (cilium.io/v2) is not registered in the cluster, the kube-apiserver is unreachable, RBAC denies list/watch on ciliumendpoints, or ctx is canceled during agent shutdown before caches sync.","commonSituations":"Cilium agent upgraded onto a cluster where the CiliumEndpoint CRD wasn't updated by `cilium install`/helm; kube-apiserver briefly unavailable or rate-limiting during agent boot; missing ClusterRole rules for ciliumendpoints.cilium.io; agent shutdown (ctx canceled) racing the cleanup job.","solutions":["Ensure the cilium.io/v2 CiliumEndpoint CRD is installed and current: run `cilium install`/upgrade or `kubectl apply` the CRDs from the matching Cilium version and verify with `kubectl get crd ciliumendpoints.cilium.io`.","Check connectivity and RBAC to kube-apiserver: `kubectl auth can-i list ciliumendpoints.cilium.io` as the agent's ServiceAccount; fix ClusterRole/Binding if denied.","Inspect agent logs for the wrapped root cause (e.g. 'context deadline exceeded' or watch errors) and verify API server health/latency.","Restart the agent once the API server is reachable; the job retries with backoff (5 x 500ms) for transient failures.","If this happens at shutdown, ignore it — it is a benign context-cancellation during teardown."],"exampleFix":"// before: cleanup job races agent shutdown, Store(ctx) fails with context canceled\nstore, err := c.ciliumEndpoint.Store(ctx)\n// after: give the store a fresh bounded deadline independent of a canceling parent ctx\nstoreCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)\ndefer cancel()\nstore, err := c.ciliumEndpoint.Store(storeCtx)","handlingStrategy":"retry","validationCode":"// before invoking the cleanup path, check the CRD and RBAC prerequisites\nkubectl get crd ciliumendpoints.cilium.io\nkubectl auth can-i list ciliumendpoints.cilium.io --as=system:serviceaccount:cilium:cilium\ncurl -sk https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/healthz","typeGuard":"func storeReady[T any](res resource.Resource[T]) bool {\n    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\n    defer cancel()\n    _, err := res.Store(ctx)\n    return err == nil\n}","tryCatchPattern":"err := cleanup.cleanStaleCEPs(ctx)\nif err != nil {\n    if resiliency.IsRetryable(err) || strings.Contains(err.Error(), \"context deadline exceeded\") {\n        return retryWithBackoff(ctx, cleanup.cleanStaleCEPs) // transient: informer not synced yet\n    }\n    return fmt.Errorf(\"non-retryable cleanup failure, check CRD/RBAC: %w\", err)\n}","preventionTips":["Always apply/upgrade CRDs together with the Cilium agent version (helm `--skip-crds=false` or `cilium-cli` install).","Verify RBAC for ciliumendpoints.cilium.io list/watch in CI before rolling out agent upgrades.","Ensure the cleanup job only starts after watcher caches signal readiness (as run() does via the restorer promise).","Treat context-canceled variants at shutdown as benign; filter them from alerting."],"tags":["kubernetes","cilium","informer","crd","startup"],"backgroundTag":"informer-cache-sync-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}