kubernetes/kops · critical

control-plane node not healthy after update, stopping rollin

Error message

control-plane node not healthy after update, stopping rolling-update: %q

What it means

After bastions, RollingUpdate updates control-plane (master) groups strictly in series and stops immediately if any group fails, because an unhealthy control-plane node puts the whole cluster at risk (etcd quorum, API server availability). The underlying group error is wrapped as 'control-plane node not healthy after update'.

Source

Thrown at pkg/instancegroups/rollingupdate.go:179

	// Do not continue update if bastion(s) failed
	for _, err := range results {
		if err != nil {
			return fmt.Errorf("bastion not healthy after update, stopping rolling-update: %q", err)
		}
	}

	// Upgrade control plane next.
	{
		// We run control-plane nodes in series, even if they are in separate instance groups
		// typically they will be in separate instance groups, so we can force the zones,
		// and we don't want to roll all the control-plane nodes at the same time.  See issue #284

		for _, k := range sortGroups(masterGroups) {
			err := c.rollingUpdateInstanceGroup(ctx, masterGroups[k], c.MasterInterval)
			// Do not continue update if control-plane node(s) failed; cluster is potentially in an unhealthy state.
			if err != nil {
				return fmt.Errorf("control-plane node not healthy after update, stopping rolling-update: %q", err)
			}
		}
	}

	// Upgrade API servers
	{
		for k := range apiServerGroups {
			results[k] = fmt.Errorf("function panic apiservers")
		}

		for _, k := range sortGroups(apiServerGroups) {
			err := c.rollingUpdateInstanceGroup(ctx, apiServerGroups[k], c.NodeInterval)
			results[k] = err
			if err != nil {
				klog.Errorf("failed to roll InstanceGroup %q: %v", k, err)
			}

			if isExitableError(err) {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check control-plane node health: `kubectl get nodes -l node-role.kubernetes.io/control-plane` and inspect etcd member status (`etcdctl member list`)
2. Verify the replaced master joined etcd and the API server pod is Running; check nodeup/cluster-setup logs on the instance
3. Fix the underlying issue (etcd volume, networking, image pulls) and re-run `kops rolling-update cluster` to continue with remaining masters
4. Ensure at least 3 control-plane nodes so quorum survives a single-node update, and allow extra validation timeout

Example fix

// before: 2 control-plane nodes (quorum lost during update)
_etcdMemberCount: 2
// after
_etcdMemberCount: 3

nodes, _ := client.CoreV1().Nodes().List(ctx, metav1.ListOptions{LabelSelector: "node-role.kubernetes.io/control-plane"})
if len(nodes.Items) < 3 {
    return errors.New("fewer than 3 control-plane nodes; rolling update risks quorum loss")
}

if err := c.RollingUpdate(ctx, cluster, clusterState, igs, options); err != nil {
    if strings.Contains(err.Error(), "control-plane node not healthy") {
        // stop all further groups; inspect etcd & API server before resuming
        klog.Errorf("control-plane update failed: %v", err)
        return err
    }
}

When it happens

Trigger: rollingUpdateInstanceGroup failed for a master group: the replacement control-plane instance never became ready (etcd member failed to join, API server pods not starting, kubelet registration failure) or post-update validation timed out.

Common situations: etcd cluster losing quorum during sequential master replacement; new master unable to pull images or mount etcd volumes (EBS attachment failure); kubelet node registration failing due to networking; validation timeouts on slow clouds.

Related errors

• cannot create cluster validator: %v
• must configure at least one ControlPlane InstanceGroup
• cluster did not validate within a duration of %q
• bastion not healthy after update, stopping rolling-update: %
• EtcdMember #%d of etcd-cluster %s did not specify a Name

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/28fbd7a0975bf106. Report an issue: GitHub.