{"record":{"id":"9d75911259f1ea56","repo":"kubernetes/kops","slug":"error-watching-services-v","errorCode":null,"errorMessage":"error watching services: %v","messagePattern":"error watching services: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dns-controller/pkg/watchers/service.go","lineNumber":101,"sourceCode":"\t\t\tservice := &serviceList.Items[i]\n\t\t\tklog.V(4).Infof(\"found service: %v\", service.Name)\n\t\t\tkey := c.updateServiceRecords(service)\n\t\t\tfoundKeys[key] = true\n\t\t}\n\t\tfor _, key := range allKeys {\n\t\t\tif !foundKeys[key] {\n\t\t\t\t// The service previously existed, but no longer exists; delete it from the scope\n\t\t\t\tklog.V(2).Infof(\"removing service not found in list: %s\", key)\n\t\t\t\tc.scope.Replace(key, nil)\n\t\t\t}\n\t\t}\n\t\tc.scope.MarkReady()\n\n\t\tlistOpts.Watch = true\n\t\tlistOpts.ResourceVersion = serviceList.ResourceVersion\n\t\twatcher, err := c.client.CoreV1().Services(c.namespace).Watch(ctx, listOpts)\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"error watching services: %v\", err)\n\t\t}\n\t\tch := watcher.ResultChan()\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-stopCh:\n\t\t\t\tklog.Infof(\"Got stop signal\")\n\t\t\t\treturn true, nil\n\t\t\tcase event, ok := <-ch:\n\t\t\t\tif !ok {\n\t\t\t\t\tklog.Infof(\"service watch channel closed\")\n\t\t\t\t\treturn false, nil\n\t\t\t\t}\n\n\t\t\t\tservice := event.Object.(*v1.Service)\n\t\t\t\tklog.V(4).Infof(\"service changed: %s %v\", event.Type, service.Name)\n\n\t\t\t\tswitch event.Type {\n\t\t\t\tcase watch.Added, watch.Modified:","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/dns-controller/pkg/watchers/service.go#L83-L119","documentation":"serviceController.runUpdater wraps failures from client.CoreV1().Services(c.namespace).Watch(ctx, listOpts) as \"error watching services: %v\". The watch, anchored at serviceList.ResourceVersion, delivers incremental service changes for DNS record updates; establishment failure aborts the current pass (retried). The real cause is the wrapped client-go error.","triggerScenarios":"client.CoreV1().Services(c.namespace).Watch(ctx, listOpts) fails: expired/compacted ResourceVersion (410 Gone), RBAC missing 'watch' on services, context cancellation, or network failure to the apiserver.","commonSituations":"Long controller stalls leading to stale ResourceVersion; namespace-scoped RBAC without watch verb; apiserver restart or etcd compaction event.","solutions":["Ensure the 'watch' verb is granted on services in the relevant RBAC rules.","Let the retry loop re-list after a 410 'too old resource version' error.","Check apiserver/etcd health if watches fail repeatedly.","Fix underlying network issues (proxies, idle connection timeouts) if errors cluster around connection resets."],"exampleFix":"// before\nresources: [\"services\"]\nverbs: [\"list\"]\n// after\nresources: [\"services\"]\nverbs: [\"list\",\"watch\"]","handlingStrategy":"retry","validationCode":"// Go: SelfSubjectAccessReview for services watch before startup\nspec.ResourceAttributes = &authorizationv1.ResourceAttributes{Verb: \"watch\", Resource: \"services\", Namespace: ns}","typeGuard":null,"tryCatchPattern":"watcher, err := client.CoreV1().Services(c.namespace).Watch(ctx, listOpts)\nif err != nil {\n    if apierrors.IsResourceExpired(err) || apierrors.IsGone(err) {\n        return false, nil // force a fresh list next cycle\n    }\n    return false, fmt.Errorf(\"error watching services: %v\", err)\n}","preventionTips":["Pair 'list' and 'watch' verbs in RBAC for services.","Monitor for repeated watch failures as a signal of etcd compaction or proxy idle timeouts.","Restart the controller cleanly after apiserver upgrades."],"tags":["kubernetes","dns-controller","watch","services"],"backgroundTag":"kubernetes-watch-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}