cilium/cilium · error

failed to get CiliumEndpoint store: %w

Error message

failed to get CiliumEndpoint store: %w

What it means

In Cilium's endpoint cleanup job, cleanStaleCEPs calls ciliumEndpoint.Store(ctx) to obtain a synced, read-only store of CiliumEndpoint resources. This error wraps the failure of that call: the resource's informer could not start or synchronize its cache within the given context (or the context was canceled). It is thrown so the startup GC job can abort and the caller (run) can decide to retry via resiliency.IsRetryable.

Source

Thrown at pkg/endpointcleanup/cleanup.go:157

			)
			if resiliency.IsRetryable(err) {
				return false, nil
			}
			return true, err
		}
		return true, nil
	})
	if err != nil {
		c.log.Error("Failed to clean up stale CEPs after multiple attempts", logfields.Error, err)
	}
	return err
}

func (c *cleanup) cleanStaleCEPs(ctx context.Context) error {
	var errs error
	store, err := c.ciliumEndpoint.Store(ctx)
	if err != nil {
		return fmt.Errorf("failed to get CiliumEndpoint store: %w", err)
	}

	ln, err := c.localNodeStore.Get(ctx)
	if err != nil {
		return fmt.Errorf("failed to get local node: %w", err)
	}

	objs, err := store.ByIndex("localNode", node.GetCiliumEndpointNodeIP(ln))
	if err != nil {
		return fmt.Errorf("failed to get indexed CiliumEndpointSlice from store: %w", err)
	}
	for _, cep := range objs {
		if cep.Networking.NodeIP == node.GetCiliumEndpointNodeIP(ln) && c.endpointsCache.LookupCEPName(cep.Namespace+"/"+cep.Name) == nil {
			if err := c.deleteCiliumEndpoint(ctx, cep.Namespace, cep.Name, &cep.ObjectMeta.UID); err != nil {
				errs = errors.Join(errs, err)
			}
		}
	}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Ensure the cilium.io/v2 CiliumEndpoint CRD is installed and current: run `cilium install`/upgrade or `kubectl apply` the CRDs from the matching Cilium version and verify with `kubectl get crd ciliumendpoints.cilium.io`.
2. Check connectivity and RBAC to kube-apiserver: `kubectl auth can-i list ciliumendpoints.cilium.io` as the agent's ServiceAccount; fix ClusterRole/Binding if denied.
3. Inspect agent logs for the wrapped root cause (e.g. 'context deadline exceeded' or watch errors) and verify API server health/latency.
4. Restart the agent once the API server is reachable; the job retries with backoff (5 x 500ms) for transient failures.
5. If this happens at shutdown, ignore it — it is a benign context-cancellation during teardown.

Example fix

// before: cleanup job races agent shutdown, Store(ctx) fails with context canceled
store, err := c.ciliumEndpoint.Store(ctx)
// after: give the store a fresh bounded deadline independent of a canceling parent ctx
storeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
store, err := c.ciliumEndpoint.Store(storeCtx)

// before invoking the cleanup path, check the CRD and RBAC prerequisites
kubectl get crd ciliumendpoints.cilium.io
kubectl auth can-i list ciliumendpoints.cilium.io --as=system:serviceaccount:cilium:cilium
curl -sk https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/healthz

func storeReady[T any](res resource.Resource[T]) bool {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    _, err := res.Store(ctx)
    return err == nil
}

err := cleanup.cleanStaleCEPs(ctx)
if err != nil {
    if resiliency.IsRetryable(err) || strings.Contains(err.Error(), "context deadline exceeded") {
        return retryWithBackoff(ctx, cleanup.cleanStaleCEPs) // transient: informer not synced yet
    }
    return fmt.Errorf("non-retryable cleanup failure, check CRD/RBAC: %w", err)
}

When it happens

Trigger: resource.Resource[*types.CiliumEndpoint].Store(ctx) returns an error when the CiliumEndpoints informer fails to start/sync: the CiliumEndpoint CRD (cilium.io/v2) is not registered in the cluster, the kube-apiserver is unreachable, RBAC denies list/watch on ciliumendpoints, or ctx is canceled during agent shutdown before caches sync.

Common situations: Cilium agent upgraded onto a cluster where the CiliumEndpoint CRD wasn't updated by `cilium install`/helm; kube-apiserver briefly unavailable or rate-limiting during agent boot; missing ClusterRole rules for ciliumendpoints.cilium.io; agent shutdown (ctx canceled) racing the cleanup job.

Related errors

• failed to get CiliumEndpointSlice store: %w
• unable to create CiliumNetworkPolicy %s: %w
• unable to retrieve CRD %s: %w
• unable to retrieve CiliumLocalRedirectPolicy %s: %w
• failed to collect Cilium cluster-wide network policies: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/1b9e392c083d0769. Report an issue: GitHub.