kubernetes/kops · error

error watching pods: %v

Error message

error watching pods: %v

What it means

podController.runUpdater wraps failures from client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts) as "error watching pods: %v". After listing pods, the controller opens a watch at podList.ResourceVersion for incremental updates; failure to establish the watch aborts the current sync (retried later). The cause is in the wrapped client-go error.

Source

Thrown at dns-controller/pkg/watchers/pod.go:102

			pod := &podList.Items[i]
			klog.V(4).Infof("found pod: %v", pod.Name)
			key := c.updatePodRecords(pod)
			foundKeys[key] = true
		}
		for _, key := range allKeys {
			if !foundKeys[key] {
				// The pod previous existed, but no longer exists; delete it from the scope
				klog.V(2).Infof("removing pod not found in list: %s", key)
				c.scope.Replace(key, nil)
			}
		}
		c.scope.MarkReady()

		listOpts.Watch = true
		listOpts.ResourceVersion = podList.ResourceVersion
		watcher, err := c.client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts)
		if err != nil {
			return false, fmt.Errorf("error watching pods: %v", err)
		}
		ch := watcher.ResultChan()
		for {
			select {
			case <-stopCh:
				klog.Infof("Got stop signal")
				return true, nil
			case event, ok := <-ch:
				if !ok {
					klog.Infof("pod watch channel closed")
					return false, nil
				}

				pod := event.Object.(*v1.Pod)
				klog.V(4).Infof("pod changed: %s %v", event.Type, pod.Name)

				switch event.Type {
				case watch.Added, watch.Modified:

View on GitHub (pinned to 4c8573c808)

Solutions

1. Ensure RBAC includes the 'watch' verb on pods in the target namespace.
2. For 'too old resource version' errors, allow the retry loop to re-list for a fresh ResourceVersion.
3. Recreate/verify the watched namespace if it was deleted.
4. Fix persistent connectivity problems between the controller and apiserver.

Example fix

// before
resources: ["pods"]
verbs: ["list"]
// after
resources: ["pods"]
verbs: ["list","watch"]

// Go: check watch RBAC on pods via SelfSubjectAccessReview before starting
spec.ResourceAttributes = &authorizationv1.ResourceAttributes{Verb: "watch", Resource: "pods", Namespace: ns}

watcher, err := client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts)
if err != nil {
    if apierrors.IsResourceExpired(err) || apierrors.IsGone(err) {
        return false, nil // re-list to obtain fresh ResourceVersion
    }
    return false, fmt.Errorf("error watching pods: %v", err)
}

When it happens

Trigger: client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts) fails: ResourceVersion compacted/expired (410 Gone), RBAC lacking 'watch' on pods, namespace deleted mid-sync, ctx cancelled, or network interruption.

Common situations: etcd compaction invalidating the list's ResourceVersion; namespace removed while the controller watched it; missing watch permission in namespace-scoped RBAC; apiserver restarts.

Related errors

• error watching nodes: %v
• error listing pods: %v
• error watching services: %v
• InstanceGroups::Watch not supported for server-side client
• failed to initialize the node controller, error: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/51afe93d10984db4. Report an issue: GitHub.