kubernetes/kops · error

cluster not yet healthy

Error message

cluster not yet healthy

What it means

`kops validate cluster` polls the cluster's validation endpoint and only fails with this error when validation keeps reporting the cluster unhealthy after the retry loop finishes without a `--wait` window, or immediately when no wait was requested. It is thrown in RunValidateCluster when the validation result does not pass the health check. It signals that control-plane nodes/pods are not all ready yet, not a CLI bug.

Source

Thrown at cmd/kops/validate_cluster.go:250

			if consecutive < options.count {
				klog.Infof("(will retry): cluster passed validation %d consecutive times", consecutive)
				if options.wait > 0 {
					time.Sleep(options.interval)
					continue
				} else {
					return nil, fmt.Errorf("cluster passed validation %d consecutive times", consecutive)
				}
			} else {
				return result, nil
			}
		} else {
			if options.wait > 0 {
				klog.Warningf("(will retry): cluster not yet healthy")
				consecutive = 0
				time.Sleep(options.interval)
				continue
			} else {
				return nil, fmt.Errorf("cluster not yet healthy")
			}
		}
	}
}

func validateClusterOutputTable(result *validation.ValidationCluster, cluster *kopsapi.Cluster, instanceGroups []kopsapi.InstanceGroup, out io.Writer) error {
	t := &tables.Table{}
	t.AddColumn("NAME", func(c kopsapi.InstanceGroup) string {
		return c.ObjectMeta.Name
	})
	t.AddColumn("ROLE", func(c kopsapi.InstanceGroup) string {
		return string(c.Spec.Role)
	})
	t.AddColumn("MACHINETYPE", func(c kopsapi.InstanceGroup) string {
		return c.Spec.MachineType
	})
	t.AddColumn("SUBNETS", func(c kopsapi.InstanceGroup) string {
		return strings.Join(c.Spec.Subnets, ",")

View on GitHub (pinned to 4c8573c808)

Solutions

1. Re-run with a wait window: `kops validate cluster --wait 15m` and `--interval` to poll until healthy.
2. Check node/pod status: `kubectl get nodes` and `kubectl get pods -n kube-system` to find the unhealthy component.
3. Verify the API endpoint DNS and load balancer resolve and respond (curl the /healthz endpoint).
4. Inspect node bootstrap: `kops get instances`, cloud console logs, or SSH to a master and check kubelet logs.
5. Re-run `kops update cluster --yes` and `kops rolling-update cluster --yes` if configuration is stale.

Example fix

// before (fails immediately on transient unhealthiness)
kops validate cluster mycluster.k8s.local
// after
kops validate cluster mycluster.k8s.local --wait 15m --interval 30s

// pre-check before polling
out, _ := exec.Command("kubectl", "get", "nodes").Output()
if !strings.Contains(string(out), "NotReady") { /* proceed to validate */ }

// Go: tolerate transient unhealthiness
result, err := RunValidateCluster(ctx, cluster, options)
if err != nil && strings.Contains(err.Error(), "cluster not yet healthy") {
    time.Sleep(30 * time.Second)
    result, err = RunValidateCluster(ctx, cluster, options) // bounded retries
}

When it happens

Trigger: Running `kops validate cluster` (or `kops get cluster` flows that call it via RunReconcileCluster) while validation returns failures: nodes not Ready, kube-apiserver not reachable, or required pods not healthy. With `--wait 0` (default), the first unhealthy validation returns this error immediately; with `--wait N`, it is returned only after all retries in the window fail.

Common situations: Running validation right after `kops update cluster --yes` before the cluster finishes bootstrapping; a broken node healthcheck (kubelet not registered); API server load balancer not yet forwarding; DNS not propagated for the API endpoint; cloud resources still provisioning.

Related errors

• invalid InstanceGroup name: %v
• unexpected error creating validatior: %v
• cannot get pod health for %q: %v
• failed to parse apiVersion %q
• failed to find kind in object

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/c5f94fb53909136c. Report an issue: GitHub.