kubernetes/kops · error

error listing nodes: %v

Error message

error listing nodes: %v

What it means

The dns-controller's nodeController.runUpdater wraps any failure from the Kubernetes Nodes().List API call into "error listing nodes: %v". The controller must watch all nodes to set up alias DNS targets, so if the initial list fails it cannot sync records and returns this error to the retry loop. The underlying cause (the %v suffix) is the client-go error, typically a network/timeout, RBAC denial, or invalid ResourceVersion issue.

Source

Thrown at dns-controller/pkg/watchers/node.go:87

	stopCh := c.StopChannel()
	go c.runWatcher(stopCh)

	<-stopCh
	klog.Infof("shutting down node controller")
}

func (c *NodeController) runWatcher(stopCh <-chan struct{}) {
	runOnce := func() (bool, error) {
		ctx := context.TODO()

		var listOpts metav1.ListOptions
		klog.V(4).Infof("querying without field filter")

		// Note we need to watch all the nodes, to set up alias targets
		allKeys := c.scope.AllKeys()
		nodeList, err := c.client.CoreV1().Nodes().List(ctx, listOpts)
		if err != nil {
			return false, fmt.Errorf("error listing nodes: %v", err)
		}
		foundKeys := make(map[string]bool)
		for i := range nodeList.Items {
			node := &nodeList.Items[i]
			klog.V(4).Infof("found node: %v", node.Name)
			key := c.updateNodeRecords(node)
			foundKeys[key] = true
		}
		for _, key := range allKeys {
			if !foundKeys[key] {
				// The node previously existed, but no longer exists; delete it from the scope
				klog.V(2).Infof("removing node not found in list: %s", key)
				c.scope.Replace(key, nil)
			}
		}
		c.scope.MarkReady()

		listOpts.Watch = true

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check RBAC: grant the controller's ServiceAccount cluster-wide list/watch on nodes (the bundled dns-controller RBAC manifest includes this).
2. Verify apiserver connectivity from the controller pod: kubectl exec into it and curl the apiserver health endpoint.
3. Inspect the wrapped error after '%v' in the log to identify the exact cause (403 vs timeout vs DNS failure).
4. If it happens once at startup, note the controller retries automatically; only intervene if it loops continuously.

Example fix

// before: clusterrole without node access
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["list","watch"]
// after: include nodes
rules:
- apiGroups: [""]
  resources: ["services","nodes"]
  verbs: ["list","watch"]

// Go: preflight the API before starting the controller
_, err := client.CoreV1().Nodes().List(ctx, metav1.ListOptions{Limit: 1})
if err != nil {
    return fmt.Errorf("preflight node list failed: %w", err)
}

nodes, err := client.CoreV1().Nodes().List(ctx, listOpts)
if err != nil {
    klog.Errorf("node list failed (rbac? connectivity?): %v", err)
    // backoff and retry the whole runUpdater cycle
    time.Sleep(backoff)
    return false, nil
}

When it happens

Trigger: Occurs when client.CoreV1().Nodes().List(ctx, listOpts) inside nodeController.runUpdater returns an error: API server unreachable, 403 on nodes list (missing RBAC permission on nodes resource), context deadline exceeded, or an expired/invalid ResourceVersion in listOpts when resuming.

Common situations: Running dns-controller with a ServiceAccount lacking 'get/list nodes' cluster rights; network partition or firewall blocking the apiserver; apiserver restarting during controller startup; misconfigured --kubeconfig or in-cluster credentials.

Related errors

• error querying namespace %q: %v
• error watching nodes: %v
• error listing pods: %v
• error listing services: %v
• error getting host %v: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/91d7c580cff2fecc. Report an issue: GitHub.