kubernetes/kops · error

error adding needs-update label: %v

Error message

error adding needs-update label: %v

What it means

updateAddon in channels/pkg/channels/addon.go wraps any failure from AddNeedsUpdateLabel after the addon manifest was successfully applied. This happens when patching the 'kops.k8s.io/needs-update' annotation onto cluster nodes fails (node List or node Patch API error). The addon resources are applied but the rolling-update signal was not recorded, so node upgrades may not be triggered.

Source

Thrown at channels/pkg/channels/addon.go:226

		merr = multierr.Append(merr, fmt.Errorf("error pruning manifest: %w", pruneError))
	}

	if applyError != nil && pruneError == nil {
		// If we failed to apply, but not prune, we should try to apply again
		if err := applier.Apply(ctx, data); err != nil {
			merr = multierr.Append(merr, fmt.Errorf("error applying update after prune: %w", err))
		} else {
			// If we succeeded to apply after prune, clear the errors
			merr = nil
		}
	}

	if merr != nil {
		return fmt.Errorf("error updating addon from %q: %w", manifestURL, merr)
	}

	if err := a.AddNeedsUpdateLabel(ctx, k8sClient, required); err != nil {
		return fmt.Errorf("error adding needs-update label: %v", err)
	}

	channel := a.buildChannel()
	err = channel.SetInstalledVersion(ctx, k8sClient, a.ChannelVersion())
	if err != nil {
		return fmt.Errorf("error applying annotation to record addon installation: %v", err)
	}
	return nil
}

func (a *Addon) AddNeedsUpdateLabel(ctx context.Context, k8sClient kubernetes.Interface, required *AddonUpdate) error {
	if required.ExistingVersion != nil {
		if a.Spec.NeedsRollingUpdate != "" {
			err := a.patchNeedsUpdateLabel(ctx, k8sClient)
			if err != nil {
				return fmt.Errorf("error patching needs-update label: %v", err)
			}
		}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify the caller has RBAC rights to list and patch core/v1 Nodes.
2. Re-run 'kops update cluster' / apply: the addon manifest already applied and the label patch is idempotent.
3. Check API server connectivity and that target nodes still exist; reconcile node groups.
4. If the addon does not actually require a rolling update, remove needsRollingUpdate from its channel spec.

Example fix

// before
if err := a.AddNeedsUpdateLabel(ctx, k8sClient, required); err != nil {
	return fmt.Errorf("error adding needs-update label: %v", err)
}
// after
if err := a.AddNeedsUpdateLabel(ctx, k8sClient, required); err != nil {
	klog.Warningf("needs-update label not applied (addon manifest already applied): %v", err)
}

// pre-check node access before running the update
nodes, err := k8sClient.CoreV1().Nodes().List(ctx, metav1.ListOptions{LabelSelector: "node-role.kubernetes.io/node="})
if err != nil {
	return fmt.Errorf("cannot list nodes, needs-update patching will fail: %w", err)
}
_ = nodes

err := addon.EnsureUpdated(ctx, k8sClient, vfsContext, pruner, applier)
if err != nil && strings.Contains(err.Error(), "error adding needs-update label") {
	// manifest already applied; degrade gracefully and retry the label step
	klog.Warningf("addon applied but rolling-update label not set: %v", err)
}

When it happens

Trigger: EnsureUpdated -> updateAddon with an addon whose Spec.NeedsRollingUpdate is set and whose ExistingVersion is non-nil (an upgrade, not a fresh install), and patchNeedsUpdateLabel fails: node List with a role label selector fails, or a node strategic-merge Patch fails (RBAC denial, node deleted mid-loop, API server unreachable).

Common situations: Running 'kops update cluster' with an identity lacking patch permission on nodes; an addon (e.g. CNI) declares needsRollingUpdate while nodes are being deleted/resized concurrently; temporary API server outage during the upgrade.

Related errors

• bootstrapping node labels: %w
• error cordoning node: %v
• error applying annotation to record addon installation: %v
• error patching needs-update label: %v
• error querying namespace %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/780d3fac038314ac. Report an issue: GitHub.