What causes "add directory %s to watcher" in thanos-io/thanos?

Verify the directory exists and is a directory: ls -ld inside the container.

add directory to watcher

Error message

add directory %s to watcher

What it means

Thanos Reloader's Watch also registers each configured directory (r.cfgDirs) with the fsnotify watcher so new/changed files in those directories trigger reloads. This error is returned when watcher.addDirectory(dir) fails — the directory does not exist, is a file, or is not readable/watchable — aborting Watch.

Solutions

Verify the directory exists and is a directory: ls -ld <dir> inside the container.
Fix the --rule-dir (or equivalent) flag value; pre-create the directory if empty dirs are valid.
Ensure the volume/ConfigMap is mounted before the component starts (init order, subPath mounts).
Check filesystem permissions and use a locally-mounted filesystem that supports inotify.

Example fix

// before: dir never created in the image
--rule-dir=/etc/thanos/rules/
// after: create it first
RUN mkdir -p /etc/thanos/rules && --rule-dir=/etc/thanos/rules/

Defensive patterns

Strategy: validation

Validate before calling

for _, d := range cfgDirs {
    info, err := os.Stat(d.Dir)
    if err != nil || !info.IsDir() {
        return fmt.Errorf("config dir %s missing or not a directory", d.Dir)
    }
}

Try / catch

if err := reloader.Watch(ctx); err != nil {
    if strings.Contains(err.Error(), "add directory") {
        logger.Error("rule directory path invalid; check mount and flag value", "err", err)
        os.Exit(1)
    }
}

Prevention

Pre-create rule directories in the container image or via initContainers.
Ensure volumes are mounted before the reloader starts.
Keep watched directories on local, inotify-capable filesystems.

When it happens

Trigger: Watch (pkg/reloader/reloader.go:302) iterates r.cfgDirs; for a dir whose path is missing, is not a directory, or cannot be opened for watching, addDirectory returns an error wrapped as 'add directory %s to watcher'.

Common situations: --rule-dir pointing to a non-existent path; directory mounted as a file or not yet mounted at container start; permission issues after securityContext changes; network filesystems lacking inotify support.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fb6278281282828e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/reloader/reloader.go:302

	defer runutil.CloseWithLogOnErr(r.logger, r.watcher, "config watcher close")

	if r.cfgFile != "" {
		if err := r.watcher.addFile(r.cfgFile); err != nil {
			return errors.Wrapf(err, "add config file %s to watcher", r.cfgFile)
		}
		initialSyncCtx, initialSyncCancel := context.WithTimeout(ctx, r.watchInterval)
		err := r.apply(initialSyncCtx)
		initialSyncCancel()
		if err != nil {
			return err
		}
	}

	for _, cfgDir := range r.cfgDirs {
		dir := cfgDir.Dir
		if err := r.watcher.addDirectory(dir); err != nil {
			return errors.Wrapf(err, "add directory %s to watcher", dir)
		}
	}

	if r.watchInterval == 0 {
		// Skip watching the file-system.
		return nil
	}

	for _, dir := range r.watchedDirs {
		if err := r.watcher.addDirectory(dir); err != nil {
			return errors.Wrapf(err, "add directory %s to watcher", dir)
		}
	}

	// Start watching the file-system.
	var wg sync.WaitGroup
	wg.Go(func() {
		r.watcher.run(ctx)

View on GitHub (pinned to 35b8b99117)