thanos-io/thanos · error

creating file watcher

Error message

creating file watcher

What it means

NewConfigWatcher wraps a failure from fsnotify.NewWatcher(), which allocates the OS-level file-watch facility (inotify on Linux). If the watcher cannot be created, Receive cannot observe hashring config changes and setup fails immediately. This is almost always an OS resource-limit problem, not a config-content problem.

Solutions

1. Raise the limit: sysctl fs.inotify.max_user_instances=<higher> and fs.inotify.max_user_watches
2. Increase the container's file-descriptor limit (ulimit -n / securityContext)
3. Reduce other inotify consumers on the node, or restart noisy workloads
4. If inotify is unavailable, poll the config file on an interval as a fallback

Example fix

// before (host defaults too low)
fs.inotify.max_user_instances = 128
// after
sysctl -w fs.inotify.max_user_instances=1024

// preflight: check fd headroom
fis, err := os.ReadDir("/proc/self/fd")
if err == nil && len(fis) > rlimNofile*90/100 {
	log.Warn("close to fd limit; watcher creation may fail")
}

w, err := receive.NewConfigWatcher(logger, path, interval)
if err != nil && strings.Contains(err.Error(), "creating file watcher") {
	// raise inotify/ulimit on the host, then retry with backoff
}

When it happens

Trigger: Calling NewConfigWatcher when inotify instances are exhausted (fs.inotify.max_user_instances) or the process hit its file-descriptor limit.

Common situations: Kubernetes nodes with many watching pods exhausting max_user_instances; low RLIMIT_NOFILE in the container; running in restricted sandboxes without inotify support.

Related errors

• adding path to file watcher

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/17ef9de59fd9506e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/config.go:190

	changesCounter       prometheus.Counter
	errorCounter         prometheus.Counter
	refreshCounter       prometheus.Counter
	hashringNodesGauge   *prometheus.GaugeVec
	hashringTenantsGauge *prometheus.GaugeVec

	// lastLoadedConfigHash is the hash of the last successfully loaded configuration.
	lastLoadedConfigHash float64
}

// NewConfigWatcher creates a new ConfigWatcher.
func NewConfigWatcher(logger log.Logger, reg prometheus.Registerer, path string, interval model.Duration) (*ConfigWatcher, error) {
	if logger == nil {
		logger = log.NewNopLogger()
	}

	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		return nil, errors.Wrap(err, "creating file watcher")
	}
	if err := watcher.Add(path); err != nil {
		return nil, errors.Wrapf(err, "adding path %s to file watcher", path)
	}

	c := &ConfigWatcher{
		ch:       make(chan []HashringConfig),
		path:     path,
		interval: time.Duration(interval),
		logger:   logger,
		watcher:  watcher,
		hashGauge: promauto.With(reg).NewGauge(
			prometheus.GaugeOpts{
				Name: "thanos_receive_config_hash",
				Help: "Hash of the currently loaded hashring configuration file.",
			}),
		successGauge: promauto.With(reg).NewGauge(
			prometheus.GaugeOpts{

View on GitHub (pinned to 35b8b99117)