kubernetes/kops · error

bastion not healthy after update, stopping rolling-update: %

Error message

bastion not healthy after update, stopping rolling-update: %q

What it means

RollingUpdate rolls bastion groups first and, because all other traffic depends on bastions for SSH access, aborts the entire rolling update if any bastion group failed its post-update validation, wrapping the underlying error with 'bastion not healthy after update'.

Source

Thrown at pkg/instancegroups/rollingupdate.go:165

				resultsMutex.Unlock()

				defer wg.Done()

				err := c.rollingUpdateInstanceGroup(ctx, bastionGroups[k], c.BastionInterval)

				resultsMutex.Lock()
				results[k] = err
				resultsMutex.Unlock()
			}(k)
		}

		wg.Wait()
	}

	// Do not continue update if bastion(s) failed
	for _, err := range results {
		if err != nil {
			return fmt.Errorf("bastion not healthy after update, stopping rolling-update: %q", err)
		}
	}

	// Upgrade control plane next.
	{
		// We run control-plane nodes in series, even if they are in separate instance groups
		// typically they will be in separate instance groups, so we can force the zones,
		// and we don't want to roll all the control-plane nodes at the same time.  See issue #284

		for _, k := range sortGroups(masterGroups) {
			err := c.rollingUpdateInstanceGroup(ctx, masterGroups[k], c.MasterInterval)
			// Do not continue update if control-plane node(s) failed; cluster is potentially in an unhealthy state.
			if err != nil {
				return fmt.Errorf("control-plane node not healthy after update, stopping rolling-update: %q", err)
			}
		}
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check bastion health: verify the bastion instance is running and registered with its LB/security groups allow TCP 22 from your CIDR
2. Run `kops validate cluster` to see the specific failing validation and address it
3. Re-run the rolling update once the bastion passes validation — the update is aborted, not partially applied to other groups
4. Increase validation timeout (`--validation-timeout`) if bastion readiness is slow

Example fix

// before: rolling update with no validation timeout headroom
kops rolling-update cluster mycluster.k8s.local --yes
// after
kops rolling-update cluster mycluster.k8s.local --validation-timeout 30m --yes

err := kops.ValidateCluster(clusterName)
if err != nil {
    return fmt.Errorf("bastion validation already failing before update: %w", err)
}

if err := c.RollingUpdate(ctx, cluster, clusterState, instanceGroups, options); err != nil {
    var bastionErr string
    if strings.Contains(err.Error(), "bastion not healthy") {
        bastionErr = err.Error() // fix bastion LB/SG, then re-run; other groups untouched
    }
    return err
}

When it happens

Trigger: rollingUpdateInstanceGroup returned an error for a bastion group — bastion instance replaced but validation (validate cluster / readiness checks) failed: new bastion failed to register, security group/SSH rules lost, or the validate command timed out.

Common situations: Bastion replaced during `kops rolling-update cluster` but cloud LB not re-registered; validateFailing because bastion SSH port 22 blocked; DNS/ELB misconfiguration for the bastion; insufficient validation rollout timeout.

Related errors

• spec.PublicKey is required
• error adding SSH public key: %v
• cannot create cluster validator: %v
• host is required
• hostname was empty

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/3b2da9aeb64a0873. Report an issue: GitHub.