{"record":{"id":"208dd34fc170f14c","repo":"tailscale/tailscale","slug":"watcher-error-w","errorCode":null,"errorMessage":"watcher error: %w","messagePattern":"watcher error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/containerboot/tailscaled.go","lineNumber":241,"sourceCode":"\t}\n\tb, err := os.ReadFile(path)\n\tif err != nil {\n\t\terrCh <- fmt.Errorf(\"error reading configfile: %w\", err)\n\t\treturn\n\t}\n\tprevTailscaledCfg = b\n\t// kubelet mounts Secrets to Pods using a series of symlinks, one of\n\t// which is <mount-dir>/..data that Kubernetes recommends consumers to\n\t// use if they need to monitor changes\n\t// https://github.com/kubernetes/kubernetes/blob/v1.28.1/pkg/volume/util/atomic_writer.go#L39-L61\n\tconst kubeletMountedCfg = \"..data\"\n\ttoWatch := filepath.Join(tailscaledCfgDir, kubeletMountedCfg)\n\tfor {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn\n\t\tcase err := <-errChan:\n\t\t\terrCh <- fmt.Errorf(\"watcher error: %w\", err)\n\t\t\treturn\n\t\tcase <-tickChan:\n\t\tcase event := <-eventChan:\n\t\t\tif event.Name != toWatch {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t}\n\t\tb, err := os.ReadFile(path)\n\t\tif err != nil {\n\t\t\terrCh <- fmt.Errorf(\"error reading configfile: %w\", err)\n\t\t\treturn\n\t\t}\n\t\t// For some proxy types the mounted volume also contains tailscaled state and other files. We\n\t\t// don't want to reload config unnecessarily on unrelated changes to these files.\n\t\tif reflect.DeepEqual(b, prevTailscaledCfg) {\n\t\t\tcontinue\n\t\t}\n\t\tprevTailscaledCfg = b","sourceCodeStart":223,"sourceCodeEnd":259,"githubUrl":"https://github.com/tailscale/tailscale/blob/cfe32b8be6a33f8e24fbc369cbfbf7c729d9e042/cmd/containerboot/tailscaled.go#L223-L259","documentation":"The fsnotify watcher delivered an error on its Errors channel and the watch loop propagates it via errCh and returns. fsnotify Errors carries low-level inotify failures, most commonly fsnotify.ErrEventOverflow (kernel event queue overflow) or errors from a watch that was removed because the watched directory was deleted or replaced.","triggerScenarios":"Very high churn of file events in the watched config directory overflowing fs.inotify.max_queued_events; the watched directory being deleted and recreated (breaking the watch); hitting per-user/system inotify instance or watch limits.","commonSituations":"Kubelet atomic-writer secret rotations combined with other busy files (tailscaled state) in the same directory; nodes with low fs.inotify.max_queued_events/max_user_watches; processes that rewrite the config dir instead of the files in it.","solutions":["Reduce unrelated file churn in the config directory (move state files elsewhere)","Raise kernel limits: sysctl fs.inotify.max_queued_events=65536 and fs.inotify.max_user_watches","Restart the container to re-establish the watch","Check whether the directory itself is being replaced by your tooling; update files in place or recreate the watch"],"exampleFix":"$ sysctl -w fs.inotify.max_queued_events=65536\n$ sysctl -w fs.inotify.max_user_watches=1048576","handlingStrategy":"retry","validationCode":"// No pre-check reliably predicts inotify overflow; monitor instead:\n// fs.file-max, fs.inotify.max_user_watches, fs.inotify.max_queued_events","typeGuard":null,"tryCatchPattern":"// In your own watchers, degrade instead of dying on fsnotify errors:\nif err := <-watcher.Errors; err != nil {\n\tif errors.Is(err, fsnotify.ErrEventOverflow) {\n\t\t// rescan state once and keep watching\n\t} else {\n\t\t// re-add watch / restart watcher\n\t}\n}","preventionTips":["Raise fs.inotify.max_queued_events and max_user_watches on nodes running config watches","Keep high-churn files (state DBs) out of the watched config directory","Alert on containerboot restarts so silent watch loss is noticed","Re-create watchers after directory replacement rather than trusting stale handles"],"tags":["fsnotify","inotify","containerboot","linux"],"backgroundTag":null,"analyzedSha":"cfe32b8be6a33f8e24fbc369cbfbf7c729d9e042","analyzedAt":"2026-08-15T19:58:31.583Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}