thanos-io/thanos · critical

replace index file

Error message

replace index file

What it means

Flush fails when fileutil.Replace cannot move the finished index file from the tmp dir to the final block directory, making the block visible. This is the finalization step of block publication; a failure here leaves the block unpublished with files stranded in the tmp dir.

Solutions

1. Ensure bTmp and bDir are on the same filesystem so rename/replace works.
2. Check write permissions on the final block directory (d.bDir).
3. Remove conflicting stale index files in the destination block dir.
4. Prevent concurrent Flush calls on the same DiskWriter/block dir and retry.

Example fix

// before
d, err := NewDiskWriter(ctx, logger, "/mnt/nfs/blocks/01BK...") // tmp on tmpfs
// after: keep bdir and its tmp dir on the same volume
d, err := NewDiskWriter(ctx, logger, "/data/blocks/01BK...")

tmpDev, err := deviceID(filepath.Join(bdir, "tmp-for-creation"))
if err != nil { return err }
destDev, err := deviceID(bdir)
if err != nil { return err }
if tmpDev != destDev {
    return fmt.Errorf("tmp dir and block dir are on different filesystems (%d vs %d)", tmpDev, destDev)
}

if _, err := d.Flush(ctx); err != nil {
    if strings.Contains(err.Error(), "replace index file") {
        return fmt.Errorf("block finalization failed (fs/permission conflict): %w", err)
    }
    return err
}

When it happens

Trigger: fileutil.Replace(<bTmp>/index, <bDir>/index) failing due to cross-device rename (EXDEV), destination permission denied, destination being a non-empty directory, or source file missing (earlier step failed silently).

Common situations: bTmp and bDir accidentally on different filesystems/mount points; bDir read-only or owned by another user; an existing corrupt index in bDir conflicting with replacement; concurrent flushes racing on the same block.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• open chunk writer
• open index writer
• open temporary block dir
• sync temporary dir file
• close temporary dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6b5daea41b8774d1. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/writer.go:140

	if err := df.Sync(); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "sync temporary dir file")
	}

	// Close temp dir before rename block dir (for windows platform).
	if err = df.Close(); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "close temporary dir")
	}
	df = nil

	if err := tsdb_errors.CloseAll(d.closers); err != nil {
		d.closers = nil
		return tsdb.BlockStats{}, err
	}
	d.closers = nil

	// Block files successfully written, make them visible by moving files from tmp dir.
	if err := fileutil.Replace(filepath.Join(d.bTmp, IndexFilename), filepath.Join(d.bDir, IndexFilename)); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "replace index file")
	}
	if err := fileutil.Replace(filepath.Join(d.bTmp, ChunksDirname), filepath.Join(d.bDir, ChunksDirname)); err != nil {
		return tsdb.BlockStats{}, errors.Wrap(err, "replace chunks dir")
	}
	return d.stats, nil
}

type statsGatheringSeriesWriter struct {
	iw tsdb.IndexWriter
	cw tsdb.ChunkWriter

	stats   tsdb.BlockStats
	symbols int64
}

func (s *statsGatheringSeriesWriter) AddSymbol(sym string) error {
	if err := s.iw.AddSymbol(sym); err != nil {
		return err

View on GitHub (pinned to 35b8b99117)