{"record":{"id":"175fce694f35045c","repo":"cilium/cilium","slug":"unable-to-create-node-event-handler-w","errorCode":null,"errorMessage":"unable to create node event handler: %w","messagePattern":"unable to create node event handler: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"operator/pkg/ipam/nodewatcher.go","lineNumber":38,"sourceCode":"\tk8sClient \"github.com/cilium/cilium/pkg/k8s/client\"\n\t\"github.com/cilium/cilium/pkg/k8s/resource\"\n\tslim_corev1 \"github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1\"\n\t\"github.com/cilium/cilium/pkg/option\"\n\t\"github.com/cilium/cilium/pkg/time\"\n)\n\nfunc newNodeWatcherJobFactory(\n\tpods resource.Resource[*slim_corev1.Pod],\n\tciliumNodes resource.Resource[*cilium_api_v2.CiliumNode],\n\tdaemonCfg *option.DaemonConfig,\n) allocator.NodeWatcherJobFactory {\n\treturn func(nmFactory allocator.NodeEventHandlerFactory) job.Job {\n\t\treturn job.OneShot(\n\t\t\t\"cilium-nodes-watcher\",\n\t\t\tfunc(ctx context.Context, _ cell.Health) error {\n\t\t\t\tnm, err := nmFactory(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"unable to create node event handler: %w\", err)\n\t\t\t\t}\n\n\t\t\t\t// The NodeEventHandler uses operatorWatchers.PodStore for IPAM surge allocation.\n\t\t\t\tpodStore, err := pods.Store(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"unable to retrieve Pod store from Pod resource watcher: %w\", err)\n\t\t\t\t}\n\t\t\t\toperatorWatchers.PodStore = podStore.CacheStore()\n\n\t\t\t\twithResync := daemonCfg.IPAM == ipamOption.IPAMClusterPool || daemonCfg.IPAM == ipamOption.IPAMMultiPool\n\t\t\t\twatchCiliumNodes(ctx, ciliumNodes, nm, withResync)\n\n\t\t\t\tnm.Stop()\n\n\t\t\t\treturn nil\n\t\t\t},\n\t\t\t// An IPAM allocator that cannot be brought up (e.g. the initial\n\t\t\t// cloud API synchronization failed) leaves the operator unable to","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/operator/pkg/ipam/nodewatcher.go#L20-L56","documentation":"The cilium-nodes-watcher one-shot job invokes its node-event-handler factory, which internally constructs and starts the IPAM allocator (see clusterpool.go). When that factory returns an error — CRD missing, API failure, RBAC denial, informer setup failure — the job wraps it as 'unable to create node event handler'. The job then aborts, and the chained pod-store retrieval never runs, so the operator's node-based IPAM does not function.","triggerScenarios":"nmFactory(ctx) (the closure at operator/pkg/ipam/clusterpool.go:106 calling allocator.Start) returns error; same root causes as the ClusterPool allocator start: missing ciliumnodes CRD, API server unreachable, RBAC denied on CiliumNode resources, or informer/watch creation failure.","commonSituations":"Fresh cluster installs where CRD application failed or was skipped; version skew between cilium agents and operator images; restricted environments (OPA/Gatekeeper) blocking CRD access; kube-apiserver outage during operator (re)start.","solutions":["Inspect the wrapped root cause in the operator logs and address the underlying API/CRD/RBAC failure it names.","Verify CRD presence and version alignment: kubectl get crd ciliumnodes.cilium.io and compare against the operator image tag.","Check operator RBAC for ciliumnodes verbs (get/list/watch/update).","Confirm apiserver reachability from the operator pod, then restart the operator to re-run the one-shot job.","If the subsequent 'unable to retrieve Pod store' error also appears, fix pod watcher startup as well before restarting."],"exampleFix":"// before: job fails because factory error is only wrapped, root cause missed in triage\n// after: surface and act on the wrapped cause\nerr := job.Wait(ctx)\nvar wrapped *fmt.WrapError\nif errors.As(err, &target) || strings.Contains(err.Error(), \"unable to create node event handler\") {\n    log.WithError(err).Error(\"node watcher failed; verify ciliumnodes CRD + RBAC\")\n    os.Exit(1) // restart operator after fixing CRD/RBAC\n}","handlingStrategy":"try-catch","validationCode":"// Validate prerequisites before enabling the cilium-nodes-watcher job:\nif _, err := k8sClient.RESTConfig(); err != nil { return err }\nif err := k8sClient.List(ctx, &ciliumv2.CiliumNodeList{}); err != nil {\n    return fmt.Errorf(\"CiliumNode API not usable; node watcher will fail: %w\", err)\n}\nif _, err := pods.Store(ctx); err != nil {\n    return fmt.Errorf(\"pod store not available; node watcher will fail: %w\", err)\n}","typeGuard":"func isNodeEventHandlerError(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"unable to create node event handler\")\n}","tryCatchPattern":"if err := job.Start(ctx); err != nil {\n    if isNodeEventHandlerError(err) {\n        // surface the wrapped allocator cause (CRD/RBAC/apiserver)\n        log.WithError(errors.Unwrap(err)).Error(\"node event handler creation failed\")\n        return // allow supervisor/job retry with backoff\n    }\n    return err\n}","preventionTips":["Deploy CRDs and RBAC in the same release step as the operator image upgrade.","Run a preflight job that lists CiliumNodes and fetches the pod store before the watcher job starts.","Catch and unwrap this error in job supervision to expose the true allocator root cause.","Keep pod watcher healthy — the same job needs pods.Store(ctx) right after handler creation.","Monitor ciliumnodes API latency/errors; watch-creation failures surface here."],"tags":["kubernetes","ipam","crd","operator","job"],"backgroundTag":"clusterpool-allocator-start-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}