What causes "unable to create node event handler: %w" in cilium/cilium?

Inspect the wrapped root cause in the operator logs and address the underlying API/CRD/RBAC failure it names.

unable to create node event handler: %w

Error message

unable to create node event handler: %w

What it means

The cilium-nodes-watcher one-shot job invokes its node-event-handler factory, which internally constructs and starts the IPAM allocator (see clusterpool.go). When that factory returns an error — CRD missing, API failure, RBAC denial, informer setup failure — the job wraps it as 'unable to create node event handler'. The job then aborts, and the chained pod-store retrieval never runs, so the operator's node-based IPAM does not function.

Source

Thrown at operator/pkg/ipam/nodewatcher.go:38

	k8sClient "github.com/cilium/cilium/pkg/k8s/client"
	"github.com/cilium/cilium/pkg/k8s/resource"
	slim_corev1 "github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1"
	"github.com/cilium/cilium/pkg/option"
	"github.com/cilium/cilium/pkg/time"
)

func newNodeWatcherJobFactory(
	pods resource.Resource[*slim_corev1.Pod],
	ciliumNodes resource.Resource[*cilium_api_v2.CiliumNode],
	daemonCfg *option.DaemonConfig,
) allocator.NodeWatcherJobFactory {
	return func(nmFactory allocator.NodeEventHandlerFactory) job.Job {
		return job.OneShot(
			"cilium-nodes-watcher",
			func(ctx context.Context, _ cell.Health) error {
				nm, err := nmFactory(ctx)
				if err != nil {
					return fmt.Errorf("unable to create node event handler: %w", err)
				}

				// The NodeEventHandler uses operatorWatchers.PodStore for IPAM surge allocation.
				podStore, err := pods.Store(ctx)
				if err != nil {
					return fmt.Errorf("unable to retrieve Pod store from Pod resource watcher: %w", err)
				}
				operatorWatchers.PodStore = podStore.CacheStore()

				withResync := daemonCfg.IPAM == ipamOption.IPAMClusterPool || daemonCfg.IPAM == ipamOption.IPAMMultiPool
				watchCiliumNodes(ctx, ciliumNodes, nm, withResync)

				nm.Stop()

				return nil
			},
			// An IPAM allocator that cannot be brought up (e.g. the initial
			// cloud API synchronization failed) leaves the operator unable to

View on GitHub (pinned to ac7b90affa)

Solutions

Inspect the wrapped root cause in the operator logs and address the underlying API/CRD/RBAC failure it names.
Verify CRD presence and version alignment: kubectl get crd ciliumnodes.cilium.io and compare against the operator image tag.
Check operator RBAC for ciliumnodes verbs (get/list/watch/update).
Confirm apiserver reachability from the operator pod, then restart the operator to re-run the one-shot job.
If the subsequent 'unable to retrieve Pod store' error also appears, fix pod watcher startup as well before restarting.

Example fix

// before: job fails because factory error is only wrapped, root cause missed in triage
// after: surface and act on the wrapped cause
err := job.Wait(ctx)
var wrapped *fmt.WrapError
if errors.As(err, &target) || strings.Contains(err.Error(), "unable to create node event handler") {
    log.WithError(err).Error("node watcher failed; verify ciliumnodes CRD + RBAC")
    os.Exit(1) // restart operator after fixing CRD/RBAC
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate prerequisites before enabling the cilium-nodes-watcher job:
if _, err := k8sClient.RESTConfig(); err != nil { return err }
if err := k8sClient.List(ctx, &ciliumv2.CiliumNodeList{}); err != nil {
    return fmt.Errorf("CiliumNode API not usable; node watcher will fail: %w", err)
}
if _, err := pods.Store(ctx); err != nil {
    return fmt.Errorf("pod store not available; node watcher will fail: %w", err)
}

Type guard

func isNodeEventHandlerError(err error) bool {
    return err != nil && strings.Contains(err.Error(), "unable to create node event handler")
}

Try / catch

if err := job.Start(ctx); err != nil {
    if isNodeEventHandlerError(err) {
        // surface the wrapped allocator cause (CRD/RBAC/apiserver)
        log.WithError(errors.Unwrap(err)).Error("node event handler creation failed")
        return // allow supervisor/job retry with backoff
    }
    return err
}

Prevention

Deploy CRDs and RBAC in the same release step as the operator image upgrade.
Run a preflight job that lists CiliumNodes and fetches the pod store before the watcher job starts.
Catch and unwrap this error in job supervision to expose the true allocator root cause.
Keep pod watcher healthy — the same job needs pods.Store(ctx) right after handler creation.
Monitor ciliumnodes API latency/errors; watch-creation failures surface here.

When it happens

Trigger: nmFactory(ctx) (the closure at operator/pkg/ipam/clusterpool.go:106 calling allocator.Start) returns error; same root causes as the ClusterPool allocator start: missing ciliumnodes CRD, API server unreachable, RBAC denied on CiliumNode resources, or informer/watch creation failure.

Common situations: Fresh cluster installs where CRD application failed or was skipped; version skew between cilium agents and operator images; restricted environments (OPA/Gatekeeper) blocking CRD access; kube-apiserver outage during operator (re)start.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/175fce694f35045c. Report an issue: GitHub.