kubernetes/kops · error

error listing nodes in cluster: %v

Error message

error listing nodes in cluster: %v

What it means

After building the k8s client, RunRollingUpdateCluster lists all Nodes via k8sClient.CoreV1().Nodes().List(ctx, ...) to correlate cloud instances with cluster members. If that API call fails, the command prints an explicit hint about --cloudonly to stderr and returns "error listing nodes in cluster: %v". The k8s API server was unreachable or rejected the request.

Source

Thrown at cmd/kops/rolling-update_cluster.go:270

		if err != nil {
			return fmt.Errorf("getting rest config: %w", err)
		}

		httpClient, err := f.HTTPClient(restConfig)
		if err != nil {
			return fmt.Errorf("getting http client: %w", err)
		}

		k8sClient, err = kubernetes.NewForConfigAndClient(restConfig, httpClient)
		if err != nil {
			return fmt.Errorf("getting kubernetes client: %w", err)
		}

		nodeList, err := k8sClient.CoreV1().Nodes().List(ctx, metav1.ListOptions{})
		if err != nil {
			fmt.Fprintf(os.Stderr, "Unable to reach the kubernetes API.\n")
			fmt.Fprintf(os.Stderr, "Use --cloudonly to do a rolling-update without confirming progress with the k8s API\n\n")
			return fmt.Errorf("error listing nodes in cluster: %v", err)
		}

		if nodeList != nil {
			nodes = nodeList.Items
		}
	}

	list, err := clientset.InstanceGroupsFor(cluster).List(ctx, metav1.ListOptions{})
	if err != nil {
		return err
	}

	countByRole := make(map[kopsapi.InstanceGroupRole]int32)
	var instanceGroups []*kopsapi.InstanceGroup
	for i := range list.Items {
		instanceGroup := &list.Items[i]
		instanceGroups = append(instanceGroups, instanceGroup)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Restore connectivity to the API server (VPN/bastion/security groups) and confirm with kubectl get nodes
2. Regenerate admin credentials: kops export kubeconfig <cluster> --admin
3. Re-run with --cloudonly to perform the rolling update purely from cloud state without k8s API confirmation (nodes may be unreachable anyway)
4. Check API server health on the master instance groups / load balancer

Example fix

// before
kops rolling-update cluster mycluster.k8s.local
// error listing nodes in cluster: ... connection refused
// after (no k8s access from this network)
kops rolling-update cluster mycluster.k8s.local --cloudonly

// precheck API reachability before rolling-update
client, err := kubernetes.NewForConfig(restConfig)
if err != nil { return err }
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := client.Discovery().ServerVersion(); err != nil {
    return fmt.Errorf("API server unreachable: %w", err)
}

if _, err := client.CoreV1().Nodes().List(ctx, metav1.ListOptions{}); err != nil {
    if apierrors.IsUnauthorized(err) || apierrors.IsForbidden(err) {
        return fmt.Errorf("re-authenticate: kops export kubeconfig <cluster> --admin: %w", err)
    }
    return fmt.Errorf("API unreachable, use --cloudonly or fix network: %w", err)
}

When it happens

Trigger: k8sClient.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) fails: API server unreachable (network/VPN down, wrong endpoint), authentication/authorization failure (expired token or cert), or the API server itself is down — and --cloudonly was not passed.

Common situations: Running rolling-update from a machine without network access to the cluster VPC; expired admin credentials after certificate rotation; API server load balancer deleted or cluster actually down mid-migration; RBAC changes removing nodes/list permission.

Related errors

• listing nodes in cluster: %v
• failed to taint node %q: %v
• error adding needs-update label: %v
• error patching needs-update label: %v
• error querying namespace %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/90042fc3dde1de23. Report an issue: GitHub.