kubernetes/kops · error

error watching pods

Error message

error watching pods: %v

What it means

podController.runUpdater wraps failures from client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts) as "error watching pods: %v". After listing pods, the controller opens a watch at podList.ResourceVersion for incremental updates; failure to establish the watch aborts the current sync (retried later). The cause is in the wrapped client-go error.

Solutions

1. Ensure RBAC includes the 'watch' verb on pods in the target namespace.
2. For 'too old resource version' errors, allow the retry loop to re-list for a fresh ResourceVersion.
3. Recreate/verify the watched namespace if it was deleted.
4. Fix persistent connectivity problems between the controller and apiserver.

Example fix

// before
resources: ["pods"]
verbs: ["list"]
// after
resources: ["pods"]
verbs: ["list","watch"]

// Go: check watch RBAC on pods via SelfSubjectAccessReview before starting
spec.ResourceAttributes = &authorizationv1.ResourceAttributes{Verb: "watch", Resource: "pods", Namespace: ns}

watcher, err := client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts)
if err != nil {
    if apierrors.IsResourceExpired(err) || apierrors.IsGone(err) {
        return false, nil // re-list to obtain fresh ResourceVersion
    }
    return false, fmt.Errorf("error watching pods: %v", err)
}

When it happens

Trigger: client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts) fails: ResourceVersion compacted/expired (410 Gone), RBAC lacking 'watch' on pods, namespace deleted mid-sync, ctx cancelled, or network interruption.

Common situations: etcd compaction invalidating the list's ResourceVersion; namespace removed while the controller watched it; missing watch permission in namespace-scoped RBAC; apiserver restarts.

Related errors

• error listing pods
• error watching nodes
• error watching services
• cannot get pod health for
• error listing nodes

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/51afe93d10984db4. Report an issue: GitHub.

Appendix: source

Thrown at dns-controller/pkg/watchers/pod.go:102

			pod := &podList.Items[i]
			klog.V(4).Infof("found pod: %v", pod.Name)
			key := c.updatePodRecords(pod)
			foundKeys[key] = true
		}
		for _, key := range allKeys {
			if !foundKeys[key] {
				// The pod previous existed, but no longer exists; delete it from the scope
				klog.V(2).Infof("removing pod not found in list: %s", key)
				c.scope.Replace(key, nil)
			}
		}
		c.scope.MarkReady()

		listOpts.Watch = true
		listOpts.ResourceVersion = podList.ResourceVersion
		watcher, err := c.client.CoreV1().Pods(c.namespace).Watch(ctx, listOpts)
		if err != nil {
			return false, fmt.Errorf("error watching pods: %v", err)
		}
		ch := watcher.ResultChan()
		for {
			select {
			case <-stopCh:
				klog.Infof("Got stop signal")
				return true, nil
			case event, ok := <-ch:
				if !ok {
					klog.Infof("pod watch channel closed")
					return false, nil
				}

				pod := event.Object.(*v1.Pod)
				klog.V(4).Infof("pod changed: %s %v", event.Type, pod.Name)

				switch event.Type {
				case watch.Added, watch.Modified:

View on GitHub (pinned to 4c8573c808)