thanos-io/thanos · error

create

Error message

create %s

What it means

After removing workDir, Update recreates it with os.MkdirAll(m.workDir, os.ModePerm). Failure here (typically permission denied or a read-only parent) is wrapped as "create %s" and blocks loading the new rule files.

Solutions

1. Point the rules dir at a writable location (emptyDir volume in K8s, or /var/lib/thanos/rules with correct ownership)
2. Verify the parent of workDir exists and is writable by the process user
3. Ensure workDir is not an existing regular file (rm or rename it)
4. Add the path to the read-only-rootfs container's writable volumes

Example fix

// before
workDir: /rules   # read-only rootfs, path missing
// after (K8s)
volumes:
  - name: rules-dir
    emptyDir: {}
# container arg: --rules-dir=/var/lib/thanos/rules

if info, err := os.Stat(workDir); err == nil && !info.IsDir() {
    return fmt.Errorf("workDir %s exists and is not a directory", workDir)
}
if err := os.MkdirAll(filepath.Dir(workDir), 0o755); err != nil {
    return fmt.Errorf("parent of workDir not creatable: %w", err)
}

func ensureDirWritable(p string) error {
    if info, err := os.Stat(p); err == nil && !info.IsDir() {
        return fmt.Errorf("%s is not a directory", p)
    }
    return os.MkdirAll(p, 0o755)
}

if err := os.MkdirAll(m.workDir, os.ModePerm); err != nil {
    if errors.Is(err, fs.ErrPermission) {
        // fallback: use a temp dir
        tmp, terr := os.MkdirTemp("", "thanos-rules")
        if terr == nil { m.workDir = tmp }
    }
    return errors.Wrapf(err, "create %s", m.workDir)
}

When it happens

Trigger: os.MkdirAll fails because the parent directory does not exist and cannot be created, permissions deny creation, the path exists as a file, or the filesystem is read-only (e.g. read-only rootfs).

Common situations: Container running with read-only root filesystem and workDir on it; default workDir path not writable by the runtime user; path collision where workDir is an existing regular file; missing parent after volume remount.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• create dir
• create upload dir
• create dir
• create working compact directory
• create working downsample directory

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/aa285699551de858. Report an issue: GitHub.

Appendix: source

Thrown at pkg/rules/manager.go:339

// special field in configGroups.configRuleAdapter struct.
func (m *Manager) Update(evalInterval time.Duration, files []string) error {
	var (
		errs            errutil.MultiError
		filesByStrategy = map[storepb.PartialResponseStrategy][]string{}
		ruleFiles       = map[string]string{}
	)

	// Initialize filesByStrategy for existing managers' strategies to make
	// sure that managers are updated when they have no rules configured.
	for strategy := range m.mgrs {
		filesByStrategy[strategy] = make([]string, 0)
	}

	if err := os.RemoveAll(m.workDir); err != nil {
		return errors.Wrapf(err, "remove %s", m.workDir)
	}
	if err := os.MkdirAll(m.workDir, os.ModePerm); err != nil {
		return errors.Wrapf(err, "create %s", m.workDir)
	}

	for _, fn := range files {
		b, err := os.ReadFile(filepath.Clean(fn))
		if err != nil {
			errs.Add(err)
			continue
		}

		var rg configGroups
		if err := yaml.Unmarshal(b, &rg); err != nil {
			errs.Add(errors.Wrap(err, fn))
			continue
		}

		// NOTE: This is very ugly, but we need to write those yaml into tmp dir without the partial partial response field
		// which is not supported, to be able to reuse rules.Manager. The problem is that it uses yaml.UnmarshalStrict.
		groupsByStrategy := map[storepb.PartialResponseStrategy][]configRuleAdapter{}

View on GitHub (pinned to 35b8b99117)