{"record":{"id":"c5f94fb53909136c","repo":"kubernetes/kops","slug":"cluster-not-yet-healthy","errorCode":null,"errorMessage":"cluster not yet healthy","messagePattern":"cluster not yet healthy","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/kops/validate_cluster.go","lineNumber":250,"sourceCode":"\t\t\tif consecutive < options.count {\n\t\t\t\tklog.Infof(\"(will retry): cluster passed validation %d consecutive times\", consecutive)\n\t\t\t\tif options.wait > 0 {\n\t\t\t\t\ttime.Sleep(options.interval)\n\t\t\t\t\tcontinue\n\t\t\t\t} else {\n\t\t\t\t\treturn nil, fmt.Errorf(\"cluster passed validation %d consecutive times\", consecutive)\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\treturn result, nil\n\t\t\t}\n\t\t} else {\n\t\t\tif options.wait > 0 {\n\t\t\t\tklog.Warningf(\"(will retry): cluster not yet healthy\")\n\t\t\t\tconsecutive = 0\n\t\t\t\ttime.Sleep(options.interval)\n\t\t\t\tcontinue\n\t\t\t} else {\n\t\t\t\treturn nil, fmt.Errorf(\"cluster not yet healthy\")\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc validateClusterOutputTable(result *validation.ValidationCluster, cluster *kopsapi.Cluster, instanceGroups []kopsapi.InstanceGroup, out io.Writer) error {\n\tt := &tables.Table{}\n\tt.AddColumn(\"NAME\", func(c kopsapi.InstanceGroup) string {\n\t\treturn c.ObjectMeta.Name\n\t})\n\tt.AddColumn(\"ROLE\", func(c kopsapi.InstanceGroup) string {\n\t\treturn string(c.Spec.Role)\n\t})\n\tt.AddColumn(\"MACHINETYPE\", func(c kopsapi.InstanceGroup) string {\n\t\treturn c.Spec.MachineType\n\t})\n\tt.AddColumn(\"SUBNETS\", func(c kopsapi.InstanceGroup) string {\n\t\treturn strings.Join(c.Spec.Subnets, \",\")","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/cmd/kops/validate_cluster.go#L232-L268","documentation":"`kops validate cluster` polls the cluster's validation endpoint and only fails with this error when validation keeps reporting the cluster unhealthy after the retry loop finishes without a `--wait` window, or immediately when no wait was requested. It is thrown in RunValidateCluster when the validation result does not pass the health check. It signals that control-plane nodes/pods are not all ready yet, not a CLI bug.","triggerScenarios":"Running `kops validate cluster` (or `kops get cluster` flows that call it via RunReconcileCluster) while validation returns failures: nodes not Ready, kube-apiserver not reachable, or required pods not healthy. With `--wait 0` (default), the first unhealthy validation returns this error immediately; with `--wait N`, it is returned only after all retries in the window fail.","commonSituations":"Running validation right after `kops update cluster --yes` before the cluster finishes bootstrapping; a broken node healthcheck (kubelet not registered); API server load balancer not yet forwarding; DNS not propagated for the API endpoint; cloud resources still provisioning.","solutions":["Re-run with a wait window: `kops validate cluster --wait 15m` and `--interval` to poll until healthy.","Check node/pod status: `kubectl get nodes` and `kubectl get pods -n kube-system` to find the unhealthy component.","Verify the API endpoint DNS and load balancer resolve and respond (curl the /healthz endpoint).","Inspect node bootstrap: `kops get instances`, cloud console logs, or SSH to a master and check kubelet logs.","Re-run `kops update cluster --yes` and `kops rolling-update cluster --yes` if configuration is stale."],"exampleFix":"// before (fails immediately on transient unhealthiness)\nkops validate cluster mycluster.k8s.local\n// after\nkops validate cluster mycluster.k8s.local --wait 15m --interval 30s","handlingStrategy":"retry","validationCode":"// pre-check before polling\nout, _ := exec.Command(\"kubectl\", \"get\", \"nodes\").Output()\nif !strings.Contains(string(out), \"NotReady\") { /* proceed to validate */ }","typeGuard":null,"tryCatchPattern":"// Go: tolerate transient unhealthiness\nresult, err := RunValidateCluster(ctx, cluster, options)\nif err != nil && strings.Contains(err.Error(), \"cluster not yet healthy\") {\n    time.Sleep(30 * time.Second)\n    result, err = RunValidateCluster(ctx, cluster, options) // bounded retries\n}","preventionTips":["Always pass --wait with a realistic window right after cluster creation or updates.","Check `kubectl get nodes` and kube-system pod health before validating.","Ensure the API endpoint DNS/LB is provisioned before running validation.","Automate polling with --interval instead of one-shot validation in CI."],"tags":["kubernetes","kops","validation","cluster-health"],"backgroundTag":"cluster-not-yet-healthy","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}