kubernetes/kops · error

error watching services: %v

Error message

error watching services: %v

What it means

serviceController.runUpdater wraps failures from client.CoreV1().Services(c.namespace).Watch(ctx, listOpts) as "error watching services: %v". The watch, anchored at serviceList.ResourceVersion, delivers incremental service changes for DNS record updates; establishment failure aborts the current pass (retried). The real cause is the wrapped client-go error.

Source

Thrown at dns-controller/pkg/watchers/service.go:101

			service := &serviceList.Items[i]
			klog.V(4).Infof("found service: %v", service.Name)
			key := c.updateServiceRecords(service)
			foundKeys[key] = true
		}
		for _, key := range allKeys {
			if !foundKeys[key] {
				// The service previously existed, but no longer exists; delete it from the scope
				klog.V(2).Infof("removing service not found in list: %s", key)
				c.scope.Replace(key, nil)
			}
		}
		c.scope.MarkReady()

		listOpts.Watch = true
		listOpts.ResourceVersion = serviceList.ResourceVersion
		watcher, err := c.client.CoreV1().Services(c.namespace).Watch(ctx, listOpts)
		if err != nil {
			return false, fmt.Errorf("error watching services: %v", err)
		}
		ch := watcher.ResultChan()
		for {
			select {
			case <-stopCh:
				klog.Infof("Got stop signal")
				return true, nil
			case event, ok := <-ch:
				if !ok {
					klog.Infof("service watch channel closed")
					return false, nil
				}

				service := event.Object.(*v1.Service)
				klog.V(4).Infof("service changed: %s %v", event.Type, service.Name)

				switch event.Type {
				case watch.Added, watch.Modified:

View on GitHub (pinned to 4c8573c808)

Solutions

1. Ensure the 'watch' verb is granted on services in the relevant RBAC rules.
2. Let the retry loop re-list after a 410 'too old resource version' error.
3. Check apiserver/etcd health if watches fail repeatedly.
4. Fix underlying network issues (proxies, idle connection timeouts) if errors cluster around connection resets.

Example fix

// before
resources: ["services"]
verbs: ["list"]
// after
resources: ["services"]
verbs: ["list","watch"]

// Go: SelfSubjectAccessReview for services watch before startup
spec.ResourceAttributes = &authorizationv1.ResourceAttributes{Verb: "watch", Resource: "services", Namespace: ns}

watcher, err := client.CoreV1().Services(c.namespace).Watch(ctx, listOpts)
if err != nil {
    if apierrors.IsResourceExpired(err) || apierrors.IsGone(err) {
        return false, nil // force a fresh list next cycle
    }
    return false, fmt.Errorf("error watching services: %v", err)
}

When it happens

Trigger: client.CoreV1().Services(c.namespace).Watch(ctx, listOpts) fails: expired/compacted ResourceVersion (410 Gone), RBAC missing 'watch' on services, context cancellation, or network failure to the apiserver.

Common situations: Long controller stalls leading to stale ResourceVersion; namespace-scoped RBAC without watch verb; apiserver restart or etcd compaction event.

Related errors

• error watching nodes: %v
• error watching pods: %v
• error listing services: %v
• InstanceGroups::Watch not supported for server-side client
• failed to initialize the node controller, error: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/9d75911259f1ea56. Report an issue: GitHub.