kubernetes/kops · error

error watching nodes: %v

Error message

error watching nodes: %v

What it means

nodeController.runUpdater wraps failures from client.CoreV1().Nodes().Watch(ctx, listOpts) in "error watching nodes: %v". After the initial list, the controller opens a watch from nodeList.ResourceVersion to receive incremental node changes; failure to establish this watch aborts the sync cycle (it retries). The wrapped error is usually a client-go watch-establishment failure such as an expired ResourceVersion or a network error.

Source

Thrown at dns-controller/pkg/watchers/node.go:109

			node := &nodeList.Items[i]
			klog.V(4).Infof("found node: %v", node.Name)
			key := c.updateNodeRecords(node)
			foundKeys[key] = true
		}
		for _, key := range allKeys {
			if !foundKeys[key] {
				// The node previously existed, but no longer exists; delete it from the scope
				klog.V(2).Infof("removing node not found in list: %s", key)
				c.scope.Replace(key, nil)
			}
		}
		c.scope.MarkReady()

		listOpts.Watch = true
		listOpts.ResourceVersion = nodeList.ResourceVersion
		watcher, err := c.client.CoreV1().Nodes().Watch(ctx, listOpts)
		if err != nil {
			return false, fmt.Errorf("error watching nodes: %v", err)
		}
		ch := watcher.ResultChan()
		for {
			select {
			case <-stopCh:
				klog.Infof("Got stop signal")
				return true, nil
			case event, ok := <-ch:
				if !ok {
					klog.Infof("node watch channel closed")
					return false, nil
				}

				node := event.Object.(*v1.Node)
				klog.V(4).Infof("node changed: %s %v", event.Type, node.Name)

				switch event.Type {
				case watch.Added, watch.Modified:

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify RBAC grants the 'watch' verb on nodes for the controller's ServiceAccount.
2. If the cause is 'too old resource version' (410), simply let the retry loop rerun: runUpdater re-lists and gets a fresh ResourceVersion.
3. Check apiserver/network health; fix persistent connectivity issues (firewall, DNS, proxy timeouts).
4. Upgrade client-go/kops if watch re-establishment repeatedly fails after compaction.

Example fix

// before: watch verb missing for nodes
resources: ["nodes"]
verbs: ["list"]
// after
resources: ["nodes"]
verbs: ["list","watch"]

// Go: verify watch permission before opening the watch
err := client.AuthorizationV1().SelfSubjectAccessReviews().Create(ctx, &authorizationv1.SelfSubjectAccessReview{
    Spec: authorizationv1.SelfSubjectAccessReviewSpec{
        ResourceAttributes: &authorizationv1.ResourceAttributes{Verb: "watch", Resource: "nodes"}}},)
// inspect err / allowed before proceeding

watcher, err := client.CoreV1().Nodes().Watch(ctx, listOpts)
if err != nil {
    if apierrors.IsResourceExpired(err) || apierrors.IsGone(err) {
        return false, nil // trigger re-list for fresh ResourceVersion
    }
    return false, fmt.Errorf("error watching nodes: %v", err)
}

When it happens

Trigger: client.CoreV1().Nodes().Watch(ctx, listOpts) fails because the ResourceVersion taken from the previous List is too old / has been compacted (HTTP 410 Gone), the watch connection cannot be established, RBAC forbids 'watch' on nodes, or ctx is cancelled.

Common situations: Long GC pauses or apiserver etcd compaction invalidating the ResourceVersion; network instability between controller and apiserver; missing 'watch' verb in RBAC for nodes; apiserver restarts.

Related errors

• error watching ingresses: %v
• error listing nodes: %v
• error listing pods: %v
• error watching pods: %v
• error listing services: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/36c0e1e6df2c0965. Report an issue: GitHub.