{"record":{"id":"5bd598e81c66ee08","repo":"cilium/cilium","slug":"failed-to-migrate-node-q-w","errorCode":null,"errorMessage":"failed to migrate node %q: %w","messagePattern":"failed to migrate node %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"operator/pkg/ipam/allocator/multipool/multipool.go","lineNumber":133,"sourceCode":"\t\t\t\t\t\t\tif err != nil {\n\t\t\t\t\t\t\t\treturn fmt.Errorf(\"failed to get CiliumNode store, migration to multi-pool IPAM failed: %w\", err)\n\t\t\t\t\t\t\t}\n\n\t\t\t\t\t\t\twp := workerpool.NewWithContext(ctx, max(p.MultiPoolCfg.MigrationWorkers, 1))\n\t\t\t\t\t\t\tdefer wp.Close()\n\n\t\t\t\t\t\t\titer := store.IterKeys()\n\t\t\t\t\t\t\tfor iter.Next() {\n\t\t\t\t\t\t\t\tkey := iter.Key()\n\t\t\t\t\t\t\t\twp.Submit(\n\t\t\t\t\t\t\t\t\tkey.Name,\n\t\t\t\t\t\t\t\t\tfunc(ctx context.Context) error {\n\t\t\t\t\t\t\t\t\t\tvar errs []error\n\t\t\t\t\t\t\t\t\t\tif err := migrateNode(\n\t\t\t\t\t\t\t\t\t\t\tctx, store, p.Clientset.CiliumV2().CiliumNodes(),\n\t\t\t\t\t\t\t\t\t\t\tkey, p.MultiPoolCfg.IPAMDefaultPool,\n\t\t\t\t\t\t\t\t\t\t); err != nil {\n\t\t\t\t\t\t\t\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to migrate node %q: %w\", key.Name, err))\n\n\t\t\t\t\t\t\t\t\t\t\tif err := updateStatusForFailure(\n\t\t\t\t\t\t\t\t\t\t\t\tctx, store, p.Clientset.CiliumV2().CiliumNodes(),\n\t\t\t\t\t\t\t\t\t\t\t\tkey, err,\n\t\t\t\t\t\t\t\t\t\t\t); err != nil {\n\t\t\t\t\t\t\t\t\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to update CiliumNode status for node %q after migration failure: %w\", key.Name, err))\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\treturn errors.Join(errs...)\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t)\n\t\t\t\t\t\t\t}\n\n\t\t\t\t\t\t\ttasks, err := wp.Drain()\n\t\t\t\t\t\t\tif err != nil {\n\t\t\t\t\t\t\t\tp.Logger.ErrorContext(\n\t\t\t\t\t\t\t\t\tctx, \"Failed to drain worker pool for multi-pool migration\",\n\t\t\t\t\t\t\t\t\tlogfields.Error, err,","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/operator/pkg/ipam/allocator/multipool/multipool.go#L115-L151","documentation":"During cluster-pool to multi-pool IPAM migration, each CiliumNode is processed by migrateNode(), which rewrites the node's IPAM pools to the default multi-pool. Any per-node failure is wrapped as 'failed to migrate node %q' and joined into the migration controller's error list.","triggerScenarios":"The workerpool runs migrateNode(ctx, store, ciliumNodeClient, key, IPAMDefaultPool) for a node key and it returns an error (e.g. CiliumNode update conflicts, object not in store, API update rejected); the error is wrapped with the node name.","commonSituations":"Frequent CiliumNode updates from cilium-agent causing optimistic-concurrency conflicts; CiliumNode missing from the local store (deleted mid-migration); API server throttling or admission webhooks rejecting the update; insufficient RBAC to update CiliumNode status.","solutions":["Inspect the wrapped inner error for the specific node to find the real cause (conflict vs not-found vs RBAC)","Re-run/restart migration: the controller retries nodes whose migration did not complete","If conflicts dominate, reduce concurrent churn on CiliumNodes during migration and ensure the operator can update CiliumNode objects/status","Check node status — updateStatusForFailure annotates the failing node with the reason"],"exampleFix":"// before\n// operator RBAC: ciliumnodes [get, list]\n// after\n// ciliumnodes [get, list, watch, update, patch] so migrateNode can rewrite pools","handlingStrategy":"retry","validationCode":"// check permissions before migration\nif err := auth.Can(ctx, clientset, \"update\", \"cilium.io\", \"ciliumnodes\"); err != nil {\n    return fmt.Errorf(\"operator cannot update CiliumNodes: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := migrateNode(ctx, store, nodes, key, pool); err != nil {\n    if apierrors.IsConflict(err) || apierrors.IsTooManyRequests(err) {\n        return retryWithBackoff(err) // transient; requeue this node\n    }\n    logger.Error(\"node migration failed permanently\", \"node\", key.Name, \"err\", err)\n    return err\n}","preventionTips":["Schedule migration during low cluster churn to reduce CiliumNode update conflicts","Ensure RBAC allows update/patch on ciliumnodes and ciliumnodes/status","Monitor the per-node failure status annotations after migration to catch stragglers"],"tags":["cilium","ipam","migration","kubernetes"],"backgroundTag":"resource-update-conflict","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}