What causes "failed to migrate node %q: %w" in cilium/cilium?

Inspect the wrapped inner error for the specific node to find the real cause (conflict vs not-found vs RBAC)

failed to migrate node %q: %w

Error message

failed to migrate node %q: %w

What it means

During cluster-pool to multi-pool IPAM migration, each CiliumNode is processed by migrateNode(), which rewrites the node's IPAM pools to the default multi-pool. Any per-node failure is wrapped as 'failed to migrate node %q' and joined into the migration controller's error list.

Source

Thrown at operator/pkg/ipam/allocator/multipool/multipool.go:133

							if err != nil {
								return fmt.Errorf("failed to get CiliumNode store, migration to multi-pool IPAM failed: %w", err)
							}

							wp := workerpool.NewWithContext(ctx, max(p.MultiPoolCfg.MigrationWorkers, 1))
							defer wp.Close()

							iter := store.IterKeys()
							for iter.Next() {
								key := iter.Key()
								wp.Submit(
									key.Name,
									func(ctx context.Context) error {
										var errs []error
										if err := migrateNode(
											ctx, store, p.Clientset.CiliumV2().CiliumNodes(),
											key, p.MultiPoolCfg.IPAMDefaultPool,
										); err != nil {
											errs = append(errs, fmt.Errorf("failed to migrate node %q: %w", key.Name, err))

											if err := updateStatusForFailure(
												ctx, store, p.Clientset.CiliumV2().CiliumNodes(),
												key, err,
											); err != nil {
												errs = append(errs, fmt.Errorf("failed to update CiliumNode status for node %q after migration failure: %w", key.Name, err))
											}
										}
										return errors.Join(errs...)
									},
								)
							}

							tasks, err := wp.Drain()
							if err != nil {
								p.Logger.ErrorContext(
									ctx, "Failed to drain worker pool for multi-pool migration",
									logfields.Error, err,

View on GitHub (pinned to ac7b90affa)

Solutions

Inspect the wrapped inner error for the specific node to find the real cause (conflict vs not-found vs RBAC)
Re-run/restart migration: the controller retries nodes whose migration did not complete
If conflicts dominate, reduce concurrent churn on CiliumNodes during migration and ensure the operator can update CiliumNode objects/status
Check node status — updateStatusForFailure annotates the failing node with the reason

Example fix

// before
// operator RBAC: ciliumnodes [get, list]
// after
// ciliumnodes [get, list, watch, update, patch] so migrateNode can rewrite pools

Defensive patterns

Strategy: retry

Validate before calling

// check permissions before migration
if err := auth.Can(ctx, clientset, "update", "cilium.io", "ciliumnodes"); err != nil {
    return fmt.Errorf("operator cannot update CiliumNodes: %w", err)
}

Try / catch

if err := migrateNode(ctx, store, nodes, key, pool); err != nil {
    if apierrors.IsConflict(err) || apierrors.IsTooManyRequests(err) {
        return retryWithBackoff(err) // transient; requeue this node
    }
    logger.Error("node migration failed permanently", "node", key.Name, "err", err)
    return err
}

Prevention

Schedule migration during low cluster churn to reduce CiliumNode update conflicts
Ensure RBAC allows update/patch on ciliumnodes and ciliumnodes/status
Monitor the per-node failure status annotations after migration to catch stragglers

When it happens

Trigger: The workerpool runs migrateNode(ctx, store, ciliumNodeClient, key, IPAMDefaultPool) for a node key and it returns an error (e.g. CiliumNode update conflicts, object not in store, API update rejected); the error is wrapped with the node name.

Common situations: Frequent CiliumNode updates from cilium-agent causing optimistic-concurrency conflicts; CiliumNode missing from the local store (deleted mid-migration); API server throttling or admission webhooks rejecting the update; insufficient RBAC to update CiliumNode status.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/5bd598e81c66ee08. Report an issue: GitHub.