{"record":{"id":"36c0e1e6df2c0965","repo":"kubernetes/kops","slug":"error-watching-nodes-v","errorCode":null,"errorMessage":"error watching nodes: %v","messagePattern":"error watching nodes: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dns-controller/pkg/watchers/node.go","lineNumber":109,"sourceCode":"\t\t\tnode := &nodeList.Items[i]\n\t\t\tklog.V(4).Infof(\"found node: %v\", node.Name)\n\t\t\tkey := c.updateNodeRecords(node)\n\t\t\tfoundKeys[key] = true\n\t\t}\n\t\tfor _, key := range allKeys {\n\t\t\tif !foundKeys[key] {\n\t\t\t\t// The node previously existed, but no longer exists; delete it from the scope\n\t\t\t\tklog.V(2).Infof(\"removing node not found in list: %s\", key)\n\t\t\t\tc.scope.Replace(key, nil)\n\t\t\t}\n\t\t}\n\t\tc.scope.MarkReady()\n\n\t\tlistOpts.Watch = true\n\t\tlistOpts.ResourceVersion = nodeList.ResourceVersion\n\t\twatcher, err := c.client.CoreV1().Nodes().Watch(ctx, listOpts)\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"error watching nodes: %v\", err)\n\t\t}\n\t\tch := watcher.ResultChan()\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-stopCh:\n\t\t\t\tklog.Infof(\"Got stop signal\")\n\t\t\t\treturn true, nil\n\t\t\tcase event, ok := <-ch:\n\t\t\t\tif !ok {\n\t\t\t\t\tklog.Infof(\"node watch channel closed\")\n\t\t\t\t\treturn false, nil\n\t\t\t\t}\n\n\t\t\t\tnode := event.Object.(*v1.Node)\n\t\t\t\tklog.V(4).Infof(\"node changed: %s %v\", event.Type, node.Name)\n\n\t\t\t\tswitch event.Type {\n\t\t\t\tcase watch.Added, watch.Modified:","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/dns-controller/pkg/watchers/node.go#L91-L127","documentation":"nodeController.runUpdater wraps failures from client.CoreV1().Nodes().Watch(ctx, listOpts) in \"error watching nodes: %v\". After the initial list, the controller opens a watch from nodeList.ResourceVersion to receive incremental node changes; failure to establish this watch aborts the sync cycle (it retries). The wrapped error is usually a client-go watch-establishment failure such as an expired ResourceVersion or a network error.","triggerScenarios":"client.CoreV1().Nodes().Watch(ctx, listOpts) fails because the ResourceVersion taken from the previous List is too old / has been compacted (HTTP 410 Gone), the watch connection cannot be established, RBAC forbids 'watch' on nodes, or ctx is cancelled.","commonSituations":"Long GC pauses or apiserver etcd compaction invalidating the ResourceVersion; network instability between controller and apiserver; missing 'watch' verb in RBAC for nodes; apiserver restarts.","solutions":["Verify RBAC grants the 'watch' verb on nodes for the controller's ServiceAccount.","If the cause is 'too old resource version' (410), simply let the retry loop rerun: runUpdater re-lists and gets a fresh ResourceVersion.","Check apiserver/network health; fix persistent connectivity issues (firewall, DNS, proxy timeouts).","Upgrade client-go/kops if watch re-establishment repeatedly fails after compaction."],"exampleFix":"// before: watch verb missing for nodes\nresources: [\"nodes\"]\nverbs: [\"list\"]\n// after\nresources: [\"nodes\"]\nverbs: [\"list\",\"watch\"]","handlingStrategy":"retry","validationCode":"// Go: verify watch permission before opening the watch\nerr := client.AuthorizationV1().SelfSubjectAccessReviews().Create(ctx, &authorizationv1.SelfSubjectAccessReview{\n    Spec: authorizationv1.SelfSubjectAccessReviewSpec{\n        ResourceAttributes: &authorizationv1.ResourceAttributes{Verb: \"watch\", Resource: \"nodes\"}}},)\n// inspect err / allowed before proceeding","typeGuard":null,"tryCatchPattern":"watcher, err := client.CoreV1().Nodes().Watch(ctx, listOpts)\nif err != nil {\n    if apierrors.IsResourceExpired(err) || apierrors.IsGone(err) {\n        return false, nil // trigger re-list for fresh ResourceVersion\n    }\n    return false, fmt.Errorf(\"error watching nodes: %v\", err)\n}","preventionTips":["Always grant both 'list' and 'watch' verbs together in RBAC for watched resources.","Expect periodic 410 Gone after etcd compaction and design for re-list.","Keep controller latency low so watches are refreshed before ResourceVersions expire."],"tags":["kubernetes","dns-controller","watch","rbac"],"backgroundTag":"kubernetes-watch-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}