thanos-io/thanos · critical

create dir

Error message

create dir

What it means

downsampleBucket wraps the error from os.MkdirAll(dir, 0750) with the context "create dir". The tool cannot create the local working directory where blocks are downloaded and downsampled before upload. This is an environment-level failure (permissions, path, disk) rather than a logic bug.

Solutions

1. Check permissions/ownership on the --data-dir path and chown/chmod it for the thanos process user.
2. Verify the path does not already exist as a regular file; remove or change --data-dir.
3. Ensure the parent directory chain exists and the filesystem is writable (not mounted ro, disk not full).
4. If in a container, mount an emptyDir/PV at the data dir with correct fsGroup/securityContext.

Example fix

// before (systemd unit)
ReadWritePaths=/var/lib/thanos
// after
ReadWritePaths=/var/lib/thanos
cd /var/lib && chown thanos:thanos thanos && chmod 750 thanos

info, err := os.Stat(dataDir)
if err == nil && !info.IsDir() {
    return fmt.Errorf("--data-dir %s exists and is not a directory", dataDir)
}
if err := os.MkdirAll(dataDir, 0750); err != nil {
    return fmt.Errorf("data-dir not usable: %w", err)
}

When it happens

Trigger: os.MkdirAll fails because the parent path is missing and cannot be created, the process lacks write permission on the path (0750 mode), the path exists as a regular file, or the filesystem is read-only/full.

Common situations: Running `thanos downsample` with --data-dir pointing at a read-only volume or a path owned by another user; container running as non-root with a host-mounted dir not writable; --data-dir accidentally pointing at a file; NFS/EFS permission issues.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• create
• create upload dir
• create dir
• create working compact directory
• create working downsample directory

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/0769d89f360fe317. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:192

	level.Info(logger).Log("msg", "starting downsample node")
	return nil
}

func downsampleBucket(
	ctx context.Context,
	logger log.Logger,
	metrics *DownsampleMetrics,
	bkt objstore.Bucket,
	metas map[ulid.ULID]*metadata.Meta,
	dir string,
	downsampleConcurrency int,
	blockFilesConcurrency int,
	hashFunc metadata.HashFunc,
	acceptMalformedIndex bool,
) (rerr error) {
	if err := os.MkdirAll(dir, 0750); err != nil {
		return errors.Wrap(err, "create dir")
	}

	defer func() {
		// Leave the downsample directory for inspection if it is a halt error
		// or if it is not then so that possibly we would not have to download everything again.
		if rerr != nil {
			return
		}
		if err := os.RemoveAll(dir); err != nil {
			level.Error(logger).Log("msg", "failed to remove downsample cache directory", "path", dir, "err", err)
		}
	}()

	// mapping from a hash over all source IDs to blocks. We don't need to downsample a block
	// if a downsampled version with the same hash already exists.
	sources5m := map[ulid.ULID]struct{}{}
	sources1h := map[ulid.ULID]struct{}{}

View on GitHub (pinned to 35b8b99117)