cilium/cilium · error

failed to wait for endpoint restoration: %w

Error message

failed to wait for endpoint restoration: %w

What it means

After obtaining the Restorer, the cleanup job calls restorer.WaitForEndpointRestore(ctx), which blocks until local endpoint restoration finishes. This error indicates that wait failed — typically ctx cancellation during shutdown, or the restorer signaling restoration failed — so stale CiliumEndpoint cleanup is skipped. Skipping is safe-by-design (no CEPs are deleted on uncertain state) but stale CEPs may linger.

Source

Thrown at pkg/endpointcleanup/cleanup.go:114

		job.OneShot("endpoint-cleanup", func(ctx context.Context, health cell.Health) error {
			return cleanup.run(ctx)
		}),
	)
}

func (c *cleanup) run(ctx context.Context) error {
	// Use restored endpoints to delete local CiliumEndpoints which are not in the restored endpoint cache.
	// This will clear out any CiliumEndpoints that may be stale.
	// Likely causes for this are Pods having their init container restarted or the node being restarted.
	// This must wait for both K8s watcher caches to be synced and local endpoint restoration to be complete.
	// Note: Synchronization of endpoints to their CEPs may not be complete at this point, but we only have to
	// know what endpoints exist post-restoration in our endpointManager cache to perform cleanup.
	restorer, err := c.restorerPromise.Await(ctx)
	if err != nil {
		return fmt.Errorf("failed to wait for endpoint restorer promise: %w", err)
	}
	if err := restorer.WaitForEndpointRestore(ctx); err != nil {
		return fmt.Errorf("failed to wait for endpoint restoration: %w", err)
	}

	var (
		retries int
		bo      = wait.Backoff{
			Duration: 500 * time.Millisecond,
			Factor:   1,
			Jitter:   0.1,
			Steps:    5,
			Cap:      0,
		}
	)
	err = wait.ExponentialBackoffWithContext(ctx, bo, func(ctx context.Context) (done bool, err error) {
		if c.ciliumEndpointSliceEnabled {
			err = c.cleanStaleCESs(ctx)
		} else {
			err = c.cleanStaleCEPs(ctx)
		}

View on GitHub (pinned to ac7b90affa)

Solutions

  1. Check the wrapped cause: context.Canceled at shutdown is expected and harmless; other causes indicate real restore failure
  2. Fix the underlying restore failure (state directory, bpffs, filesystem permissions on /var/run/cilium) and restart the agent
  3. Manually delete known-stale CiliumEndpoint objects (kubectl delete cep) if automated cleanup keeps being skipped
  4. Give the agent time on restart; avoid killing cilium during the restore window so cleanup can complete once
Defensive patterns

Strategy: try-catch

Validate before calling

if errors.Is(err, context.Canceled) {
    return nil // expected at shutdown; rerun cleanup on next agent start
}

Type guard

func isContextErr(err error) bool {
    return errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)
}

Try / catch

if err := restorer.WaitForEndpointRestore(ctx); err != nil {
    if isContextErr(err) {
        log.Info("restoration wait cancelled; skipping stale CEP cleanup")
        return nil
    }
    log.Error("endpoint restoration failed; stale CEP cleanup skipped", "error", err)
}

Prevention

When it happens

Trigger: restorer.WaitForEndpointRestore(ctx) returns an error in cleanup.run: agent context cancelled mid-restore (shutdown/kill), restore loop hit an unrecoverable error, or a timeout elapsed on a node with a large endpoint count.

Common situations: Node reboot or agent upgrade interrupting restoration; pod init-container restarts prolonging restore beyond operator patience; leftover stale CEPs from a previous agent run that never got cleaned because this wait aborted.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/bae98ccfd3c86ab8. Report an issue: GitHub.