thanos-io/thanos · error

replace chunks dir

Error message

replace chunks dir

What it means

Thanos compactor's block writer (pkg/block/writer.go Flush) finalizes a block by atomically moving the index file and chunks directory from a temporary directory (bTmp) into the final block directory (bDir) with fileutil.Replace. This error wraps a failure to move the 'chunks' directory into place after the index was already moved. It means the block was mostly written but the final rename/exchange failed, typically due to filesystem-level problems or leftover destination state.

Solutions

1. Delete the stale/partial destination block directory (bDir) so the replace target no longer exists, then rerun compaction
2. Ensure the tmp dir and block dir are on the same filesystem (configure --data-dir and tmp on the same mount)
3. Check disk space and write permissions on the block store directory
4. If on NFS/fuse without RENAME_EXCHANGE, move to a local filesystem or a fileutil version with a rename fallback
5. Verify no concurrent compactor process is writing the same block ( Thanos compactor must run as a singleton)

Example fix

// before: tmp on different FS
# thanos compact --data-dir=/data/thanos --tmp.dir=/tmp
// after: same filesystem
# thanos compact --data-dir=/data/thanos --tmp.dir=/data/thanos-tmp

// before triggering compaction flush
if _, err := os.Stat(bDir); err == nil {
    return fmt.Errorf("destination block dir %s already exists; clean it first", bDir)
}
if st, st2, err := statFS(bTmp), statFS(bDir), error(nil); sameMount(bTmp, bDir) == false {
    return fmt.Errorf("tmp dir and block dir must be on the same filesystem")
}

// Go
if err := Flush(...); err != nil {
    if strings.Contains(err.Error(), "replace chunks dir") {
        os.RemoveAll(blockDir) // clear stale destination
        err = retryFlush()
    }
    return err
}

When it happens

Trigger: fileutil.Replace (renameat2 RENAME_EXCHANGE or rename fallback) fails when the destination bDir/chunks already exists on a filesystem without RENAME_EXCHANGE support, when source and destination are on different mounts/filesystems, when the tmp dir was removed concurrently, or on EACCES/EXDEV/ENOSPC during the rename.

Common situations: Block directory left behind by a previously crashed compaction; object-storage-backed local cache on a different filesystem than the tmp dir; read-only or full disk on the compactor; parallel compaction processes writing the same block ID; network filesystems (NFS) lacking renameat2 support.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• replace index file
• level is bigger then default set of
• create meta fetcher
• get compaction levels
• create compactor

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/31999342671685d0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/writer.go:143

	// Close temp dir before rename block dir (for windows platform).
	if err = df.Close(); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "close temporary dir")
	}
	df = nil

	if err := tsdb_errors.CloseAll(d.closers); err != nil {
		d.closers = nil
		return tsdb.BlockStats{}, err
	}
	d.closers = nil

	// Block files successfully written, make them visible by moving files from tmp dir.
	if err := fileutil.Replace(filepath.Join(d.bTmp, IndexFilename), filepath.Join(d.bDir, IndexFilename)); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "replace index file")
	}
	if err := fileutil.Replace(filepath.Join(d.bTmp, ChunksDirname), filepath.Join(d.bDir, ChunksDirname)); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "replace chunks dir")
	}
	return d.stats, nil
}

type statsGatheringSeriesWriter struct {
	iw tsdb.IndexWriter
	cw tsdb.ChunkWriter

	stats   tsdb.BlockStats
	symbols int64
}

func (s *statsGatheringSeriesWriter) AddSymbol(sym string) error {
	if err := s.iw.AddSymbol(sym); err != nil {
		return err
	}
	s.symbols++
	return nil

View on GitHub (pinned to 35b8b99117)