{"record":{"id":"17ef9de59fd9506e","repo":"thanos-io/thanos","slug":"creating-file-watcher","errorCode":null,"errorMessage":"creating file watcher","messagePattern":"creating file watcher","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/config.go","lineNumber":190,"sourceCode":"\tchangesCounter       prometheus.Counter\n\terrorCounter         prometheus.Counter\n\trefreshCounter       prometheus.Counter\n\thashringNodesGauge   *prometheus.GaugeVec\n\thashringTenantsGauge *prometheus.GaugeVec\n\n\t// lastLoadedConfigHash is the hash of the last successfully loaded configuration.\n\tlastLoadedConfigHash float64\n}\n\n// NewConfigWatcher creates a new ConfigWatcher.\nfunc NewConfigWatcher(logger log.Logger, reg prometheus.Registerer, path string, interval model.Duration) (*ConfigWatcher, error) {\n\tif logger == nil {\n\t\tlogger = log.NewNopLogger()\n\t}\n\n\twatcher, err := fsnotify.NewWatcher()\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"creating file watcher\")\n\t}\n\tif err := watcher.Add(path); err != nil {\n\t\treturn nil, errors.Wrapf(err, \"adding path %s to file watcher\", path)\n\t}\n\n\tc := &ConfigWatcher{\n\t\tch:       make(chan []HashringConfig),\n\t\tpath:     path,\n\t\tinterval: time.Duration(interval),\n\t\tlogger:   logger,\n\t\twatcher:  watcher,\n\t\thashGauge: promauto.With(reg).NewGauge(\n\t\t\tprometheus.GaugeOpts{\n\t\t\t\tName: \"thanos_receive_config_hash\",\n\t\t\t\tHelp: \"Hash of the currently loaded hashring configuration file.\",\n\t\t\t}),\n\t\tsuccessGauge: promauto.With(reg).NewGauge(\n\t\t\tprometheus.GaugeOpts{","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/config.go#L172-L208","documentation":"NewConfigWatcher wraps a failure from fsnotify.NewWatcher(), which allocates the OS-level file-watch facility (inotify on Linux). If the watcher cannot be created, Receive cannot observe hashring config changes and setup fails immediately. This is almost always an OS resource-limit problem, not a config-content problem.","triggerScenarios":"Calling NewConfigWatcher when inotify instances are exhausted (fs.inotify.max_user_instances) or the process hit its file-descriptor limit.","commonSituations":"Kubernetes nodes with many watching pods exhausting max_user_instances; low RLIMIT_NOFILE in the container; running in restricted sandboxes without inotify support.","solutions":["Raise the limit: sysctl fs.inotify.max_user_instances=<higher> and fs.inotify.max_user_watches","Increase the container's file-descriptor limit (ulimit -n / securityContext)","Reduce other inotify consumers on the node, or restart noisy workloads","If inotify is unavailable, poll the config file on an interval as a fallback"],"exampleFix":"// before (host defaults too low)\nfs.inotify.max_user_instances = 128\n// after\nsysctl -w fs.inotify.max_user_instances=1024","handlingStrategy":"retry","validationCode":"// preflight: check fd headroom\nfis, err := os.ReadDir(\"/proc/self/fd\")\nif err == nil && len(fis) > rlimNofile*90/100 {\n\tlog.Warn(\"close to fd limit; watcher creation may fail\")\n}","typeGuard":null,"tryCatchPattern":"w, err := receive.NewConfigWatcher(logger, path, interval)\nif err != nil && strings.Contains(err.Error(), \"creating file watcher\") {\n\t// raise inotify/ulimit on the host, then retry with backoff\n}","preventionTips":["Raise fs.inotify.max_user_instances/max_user_watches on all nodes","Set a generous RLIMIT_NOFILE in the container spec","Avoid running many fsnotify consumers per pod"],"tags":["fsnotify","inotify","resource-limits"],"backgroundTag":"module-init-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}