thanos-io/thanos · error

remove

Error message

remove %s

What it means

Manager.Update rewrites the rule files directory: it first removes the entire workDir with os.RemoveAll. If removal fails (permissions, busy files, or being inside the directory), the error is wrapped as "remove %s" and the update aborts.

Solutions

1. Check and fix ownership/permissions of workDir (chown/chmod) for the process user
2. Ensure workDir is not a mount point or on a read-only filesystem
3. Set --rules-dir (workDir) to a dedicated writable path like /var/lib/thanos/rules
4. Stop processes holding files open in workDir, then retry the update

Example fix

// before
thanos rule --rules-dir /etc/thanos   # root-owned, process runs as nobody
// after
mkdir -p /var/lib/thanos/rules && chown nobody: /var/lib/thanos/rules
thanos rule --rules-dir /var/lib/thanos/rules

info, err := os.Stat(workDir)
if err == nil {
    test := filepath.Join(workDir, ".writable")
    if err := os.WriteFile(test, nil, 0o644); err != nil {
        return fmt.Errorf("workDir %s not writable: %w", workDir, err)
    }
    os.Remove(test)
}

func canRemoveAll(dir string) bool {
    parent := filepath.Dir(dir)
    f, err := os.Open(parent)
    if err != nil { return false }
    defer f.Close()
    _, err = f.Readdirnames(1)
    return err == nil
}

if err := os.RemoveAll(m.workDir); err != nil {
    var perr *fs.PathError
    if errors.As(err, &perr) && errors.Is(perr.Err, syscall.EACCES) {
        // fix permissions or fall back to per-file deletion
    }
    return errors.Wrapf(err, "remove %s", m.workDir)
}

When it happens

Trigger: os.RemoveAll(m.workDir) returns an error: workDir or a child is not writable/deletable by the process user, workDir is a mount point, or a file is held open on Windows.

Common situations: Running Thanos as non-root while workDir was created by root; workDir on a read-only volume or immutable mount; stale permissions after container restart; workDir nested under a bind mount.

Related errors

• create working compact directory
• create working downsample directory
• create dir
• create default tenant data dir
• remove storage lock files

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/dae09117d8636efe. Report an issue: GitHub.

Appendix: source

Thrown at pkg/rules/manager.go:336

}

// Update updates rules from given files to all managers we hold. We decide which groups should go where, based on
// special field in configGroups.configRuleAdapter struct.
func (m *Manager) Update(evalInterval time.Duration, files []string) error {
	var (
		errs            errutil.MultiError
		filesByStrategy = map[storepb.PartialResponseStrategy][]string{}
		ruleFiles       = map[string]string{}
	)

	// Initialize filesByStrategy for existing managers' strategies to make
	// sure that managers are updated when they have no rules configured.
	for strategy := range m.mgrs {
		filesByStrategy[strategy] = make([]string, 0)
	}

	if err := os.RemoveAll(m.workDir); err != nil {
		return errors.Wrapf(err, "remove %s", m.workDir)
	}
	if err := os.MkdirAll(m.workDir, os.ModePerm); err != nil {
		return errors.Wrapf(err, "create %s", m.workDir)
	}

	for _, fn := range files {
		b, err := os.ReadFile(filepath.Clean(fn))
		if err != nil {
			errs.Add(err)
			continue
		}

		var rg configGroups
		if err := yaml.Unmarshal(b, &rg); err != nil {
			errs.Add(errors.Wrap(err, fn))
			continue
		}

View on GitHub (pinned to 35b8b99117)