thanos-io/thanos · error

create working downsample directory

Error message

create working downsample directory

What it means

os.MkdirAll failed to create (or verify) the downsample working directory under --data-dir. Fires when the filesystem is unwritable, full, or the path conflicts with a non-directory. This aborts runCompact startup before any compaction runs.

Solutions

1. Ensure <dataDir> is writable by the process user (chown/chmod).
2. Remove any regular file occupying <dataDir>/downsample.
3. Check volume free space and read-write mount status.

Example fix

// before
/data/downsample exists as a file (from a bad restore)
// after
rm /data/downsample && mkdir -p /data/downsample && chown -R thanos:thanos /data

if err := os.MkdirAll(path.Join(dataDir, "downsample"), 0o755); err != nil {
    return fmt.Errorf("cannot prepare downsample dir under %q: %w", dataDir, err)
}

if err := runCompact(...); err != nil {
    if strings.Contains(err.Error(), "create working downsample directory") {
        logger.Error(err, "downsample dir unusable; check permissions and free space")
    }
    return err
}

When it happens

Trigger: os.MkdirAll(downsamplingDir, os.ModePerm) fails due to permissions, a file at that path, read-only filesystem, or ENOSPC.

Common situations: Same as the compact dir error: PV permission issues, non-root process, full disk, or path collision with an existing file.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• create working compact directory
• create default tenant data dir
• remove storage lock files
• create meta fetcher
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5de9437be999404c. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:360

	// Instantiate the compactor with different time slices. Timestamps in TSDB
	// are in milliseconds.
	comp, err := tsdb.NewLeveledCompactor(ctx, reg, logutil.GoKitLogToSlog(logger), levels, downsample.NewPool(), mergeFunc)
	if err != nil {
		return errors.Wrap(err, "create compactor")
	}

	var (
		compactDir      = path.Join(conf.dataDir, "compact")
		downsamplingDir = path.Join(conf.dataDir, "downsample")
	)

	if err := os.MkdirAll(compactDir, os.ModePerm); err != nil {
		return errors.Wrap(err, "create working compact directory")
	}

	if err := os.MkdirAll(downsamplingDir, os.ModePerm); err != nil {
		return errors.Wrap(err, "create working downsample directory")
	}

	grouper := compact.NewDefaultGrouper(
		logger,
		insBkt,
		conf.acceptMalformedIndex,
		enableVerticalCompaction,
		reg,
		compactMetrics.blocksMarked.WithLabelValues(metadata.DeletionMarkFilename, ""),
		compactMetrics.garbageCollectedBlocks,
		compactMetrics.blocksMarked.WithLabelValues(metadata.NoCompactMarkFilename, metadata.OutOfOrderChunksNoCompactReason),
		metadata.HashFunc(conf.hashFunc),
		conf.blockFilesConcurrency,
		conf.compactBlocksFetchConcurrency,
	)
	var planner compact.Planner

	tsdbPlanner := compact.NewPlanner(logger, levels, noCompactMarkerFilter)

View on GitHub (pinned to 35b8b99117)