{"record":{"id":"28fbd7a0975bf106","repo":"kubernetes/kops","slug":"control-plane-node-not-healthy-after-update-stopp","errorCode":null,"errorMessage":"control-plane node not healthy after update, stopping rolling-update: %q","messagePattern":"control-plane node not healthy after update, stopping rolling-update: %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/instancegroups/rollingupdate.go","lineNumber":179,"sourceCode":"\n\t// Do not continue update if bastion(s) failed\n\tfor _, err := range results {\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"bastion not healthy after update, stopping rolling-update: %q\", err)\n\t\t}\n\t}\n\n\t// Upgrade control plane next.\n\t{\n\t\t// We run control-plane nodes in series, even if they are in separate instance groups\n\t\t// typically they will be in separate instance groups, so we can force the zones,\n\t\t// and we don't want to roll all the control-plane nodes at the same time.  See issue #284\n\n\t\tfor _, k := range sortGroups(masterGroups) {\n\t\t\terr := c.rollingUpdateInstanceGroup(ctx, masterGroups[k], c.MasterInterval)\n\t\t\t// Do not continue update if control-plane node(s) failed; cluster is potentially in an unhealthy state.\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"control-plane node not healthy after update, stopping rolling-update: %q\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\t// Upgrade API servers\n\t{\n\t\tfor k := range apiServerGroups {\n\t\t\tresults[k] = fmt.Errorf(\"function panic apiservers\")\n\t\t}\n\n\t\tfor _, k := range sortGroups(apiServerGroups) {\n\t\t\terr := c.rollingUpdateInstanceGroup(ctx, apiServerGroups[k], c.NodeInterval)\n\t\t\tresults[k] = err\n\t\t\tif err != nil {\n\t\t\t\tklog.Errorf(\"failed to roll InstanceGroup %q: %v\", k, err)\n\t\t\t}\n\n\t\t\tif isExitableError(err) {","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/rollingupdate.go#L161-L197","documentation":"After bastions, RollingUpdate updates control-plane (master) groups strictly in series and stops immediately if any group fails, because an unhealthy control-plane node puts the whole cluster at risk (etcd quorum, API server availability). The underlying group error is wrapped as 'control-plane node not healthy after update'.","triggerScenarios":"rollingUpdateInstanceGroup failed for a master group: the replacement control-plane instance never became ready (etcd member failed to join, API server pods not starting, kubelet registration failure) or post-update validation timed out.","commonSituations":"etcd cluster losing quorum during sequential master replacement; new master unable to pull images or mount etcd volumes (EBS attachment failure); kubelet node registration failing due to networking; validation timeouts on slow clouds.","solutions":["Check control-plane node health: `kubectl get nodes -l node-role.kubernetes.io/control-plane` and inspect etcd member status (`etcdctl member list`)","Verify the replaced master joined etcd and the API server pod is Running; check nodeup/cluster-setup logs on the instance","Fix the underlying issue (etcd volume, networking, image pulls) and re-run `kops rolling-update cluster` to continue with remaining masters","Ensure at least 3 control-plane nodes so quorum survives a single-node update, and allow extra validation timeout"],"exampleFix":"// before: 2 control-plane nodes (quorum lost during update)\n_etcdMemberCount: 2\n// after\n_etcdMemberCount: 3","handlingStrategy":"try-catch","validationCode":"nodes, _ := client.CoreV1().Nodes().List(ctx, metav1.ListOptions{LabelSelector: \"node-role.kubernetes.io/control-plane\"})\nif len(nodes.Items) < 3 {\n    return errors.New(\"fewer than 3 control-plane nodes; rolling update risks quorum loss\")\n}","typeGuard":null,"tryCatchPattern":"if err := c.RollingUpdate(ctx, cluster, clusterState, igs, options); err != nil {\n    if strings.Contains(err.Error(), \"control-plane node not healthy\") {\n        // stop all further groups; inspect etcd & API server before resuming\n        klog.Errorf(\"control-plane update failed: %v\", err)\n        return err\n    }\n}","preventionTips":["Run at least 3 control-plane nodes for quorum","Verify etcd member health (etcdctl member list) before and after each master","Watch API server pods and nodeup logs on replaced masters","Allow generous validation timeouts on slow clouds"],"tags":["control-plane","etcd","rolling-update","validation"],"backgroundTag":"control-plane-node-unhealthy","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}