kubernetes/kops · error

cannot get pod health for %q: %v

Error message

cannot get pod health for %q: %v

What it means

After nodes validate, Validate calls collectPodFailures to check required pods (including master static pods) are healthy on the right nodes. Any error from that step is wrapped with the cluster name so you know which cluster's pod health check failed. It is a wrapper — the actionable detail is the inner error from collectPodFailures.

Source

Thrown at pkg/validation/validate_cluster.go:234

					}
				}
			}
		}

		if len(notReadyWorkerNodes) > 0 && len(notReadyWorkerNodes) <= v.maxUnreadyNodes {
			toleratedNodes = make(map[string]bool)
			for _, n := range notReadyWorkerNodes {
				toleratedNodes[n] = true
			}
			sort.Strings(notReadyWorkerNodes)
			klog.Warningf("Tolerating %d non-ready worker node(s): %s", len(notReadyWorkerNodes), strings.Join(notReadyWorkerNodes, ", "))
		}
	}

	readyNodes, nodeInstanceGroupMapping := validation.validateNodes(cloudGroups, v.allInstanceGroups, v.filterInstanceGroups, toleratedNodes)

	if err := validation.collectPodFailures(ctx, v.k8sClient, readyNodes, nodeInstanceGroupMapping, v.filterPodsForValidation, toleratedNodes); err != nil {
		return nil, fmt.Errorf("cannot get pod health for %q: %v", v.cluster.Name, err)
	}

	return validation, nil
}

var masterStaticPods = []string{
	"kube-apiserver",
	"kube-controller-manager",
	"kube-scheduler",
}

func (v *ValidationCluster) collectPodFailures(ctx context.Context, client kubernetes.Interface, readyNodes []v1.Node, nodeInstanceGroupMapping map[string]*kops.InstanceGroup, podValidationFilter func(pod *v1.Pod) bool, toleratedNodes map[string]bool) error {
	log := klog.FromContext(ctx)

	masterWithoutPod := map[string]map[string]bool{}

	for _, node := range readyNodes {
		labels := node.GetLabels()

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the inner %v error to distinguish auth/RBAC/network causes
2. Run kubectl get pods -A to confirm pod listing works with the same credentials
3. Retry if the cause was a transient API/timeout issue; increase the command timeout
4. Ensure the kubeconfig user can list pods in the namespaces being validated

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
validateCluster(ctx) // cannot get pod health for "prod": context deadline exceeded
// after
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
validateCluster(ctx)

_, err := k8sClient.CoreV1().Pods("kube-system").List(ctx, metav1.ListOptions{})
if err != nil {
    return fmt.Errorf("cannot list pods before validation: %v", err)
}

if _, err := v.Validate(ctx); err != nil {
    if strings.HasPrefix(err.Error(), "cannot get pod health") {
        // inspect inner error: auth/RBAC/network/timeouts
        return fmt.Errorf("pod health check failed for cluster: %w", err)
    }
    return err
}

When it happens

Trigger: validation.collectPodFailures returns a non-nil error, typically because an underlying Pod list call failed (see its own wrapping), or the context passed in is cancelled/timed out mid-check.

Common situations: API server connectivity drops while listing pods; RBAC lacks pod list permission in kube-system; validation runs with a short context deadline on a large cluster; cluster name reported helps when validating several clusters in CI.

Related errors

• error listing Pods: %v
• invalid InstanceGroup name: %v
• unexpected error creating validatior: %v
• cluster not yet healthy
• unable to parse Kubernetes cluster API URL: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/a4e2dc7cc370f7d3. Report an issue: GitHub.