{"record":{"id":"3b2da9aeb64a0873","repo":"kubernetes/kops","slug":"bastion-not-healthy-after-update-stopping-rolling","errorCode":null,"errorMessage":"bastion not healthy after update, stopping rolling-update: %q","messagePattern":"bastion not healthy after update, stopping rolling-update: %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/rollingupdate.go","lineNumber":165,"sourceCode":"\t\t\t\tresultsMutex.Unlock()\n\n\t\t\t\tdefer wg.Done()\n\n\t\t\t\terr := c.rollingUpdateInstanceGroup(ctx, bastionGroups[k], c.BastionInterval)\n\n\t\t\t\tresultsMutex.Lock()\n\t\t\t\tresults[k] = err\n\t\t\t\tresultsMutex.Unlock()\n\t\t\t}(k)\n\t\t}\n\n\t\twg.Wait()\n\t}\n\n\t// Do not continue update if bastion(s) failed\n\tfor _, err := range results {\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"bastion not healthy after update, stopping rolling-update: %q\", err)\n\t\t}\n\t}\n\n\t// Upgrade control plane next.\n\t{\n\t\t// We run control-plane nodes in series, even if they are in separate instance groups\n\t\t// typically they will be in separate instance groups, so we can force the zones,\n\t\t// and we don't want to roll all the control-plane nodes at the same time.  See issue #284\n\n\t\tfor _, k := range sortGroups(masterGroups) {\n\t\t\terr := c.rollingUpdateInstanceGroup(ctx, masterGroups[k], c.MasterInterval)\n\t\t\t// Do not continue update if control-plane node(s) failed; cluster is potentially in an unhealthy state.\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"control-plane node not healthy after update, stopping rolling-update: %q\", err)\n\t\t\t}\n\t\t}\n\t}\n","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/rollingupdate.go#L147-L183","documentation":"RollingUpdate rolls bastion groups first and, because all other traffic depends on bastions for SSH access, aborts the entire rolling update if any bastion group failed its post-update validation, wrapping the underlying error with 'bastion not healthy after update'.","triggerScenarios":"rollingUpdateInstanceGroup returned an error for a bastion group — bastion instance replaced but validation (validate cluster / readiness checks) failed: new bastion failed to register, security group/SSH rules lost, or the validate command timed out.","commonSituations":"Bastion replaced during `kops rolling-update cluster` but cloud LB not re-registered; validateFailing because bastion SSH port 22 blocked; DNS/ELB misconfiguration for the bastion; insufficient validation rollout timeout.","solutions":["Check bastion health: verify the bastion instance is running and registered with its LB/security groups allow TCP 22 from your CIDR","Run `kops validate cluster` to see the specific failing validation and address it","Re-run the rolling update once the bastion passes validation — the update is aborted, not partially applied to other groups","Increase validation timeout (`--validation-timeout`) if bastion readiness is slow"],"exampleFix":"// before: rolling update with no validation timeout headroom\nkops rolling-update cluster mycluster.k8s.local --yes\n// after\nkops rolling-update cluster mycluster.k8s.local --validation-timeout 30m --yes","handlingStrategy":"try-catch","validationCode":"err := kops.ValidateCluster(clusterName)\nif err != nil {\n    return fmt.Errorf(\"bastion validation already failing before update: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := c.RollingUpdate(ctx, cluster, clusterState, instanceGroups, options); err != nil {\n    var bastionErr string\n    if strings.Contains(err.Error(), \"bastion not healthy\") {\n        bastionErr = err.Error() // fix bastion LB/SG, then re-run; other groups untouched\n    }\n    return err\n}","preventionTips":["Keep bastion security groups open on TCP 22 from admin CIDRs","Run kops validate cluster before and after updates","Increase --validation-timeout for slow bastion registration"],"tags":["bastion","validation","rolling-update","ssh"],"backgroundTag":"rolling-update-validation-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}