thanos-io/thanos · error

remove any existing index at path

Error message

remove any existing index at path

What it means

newBinaryWriter builds the binary index-header file for a TSDB block. When a cache filename is given, it first deletes any pre-existing file at that path with os.RemoveAll. This error wraps the underlying filesystem error from that removal, so it means the old file (or directory) at the cache path could not be deleted.

Solutions

1. Check permissions on the parent directory of cacheFilename and ensure the process user can write/delete there (ls -ld on the dir).
2. If cacheFilename is a directory, point it to a file path or clear the directory manually.
3. Remount the volume read-write, or move the cache directory to a writable location.
4. Stop competing processes holding the file and retry.
5. If the file is disposable, remove it out-of-band and rerun WriteBinary.

Example fix

// before
if err := os.RemoveAll(cacheFilename); err != nil {
    return nil, errors.Wrap(err, "remove any existing index at path")
}
// after
if _, statErr := os.Stat(cacheFilename); statErr == nil {
    if err := os.RemoveAll(cacheFilename); err != nil {
        return nil, errors.Wrapf(err, "remove any existing index at path %s (check dir permissions)", cacheFilename)
    }
}

if info, err := os.Stat(cachePath); err == nil && !info.Mode().IsRegular() {
    return fmt.Errorf("cache path %s is not a regular file; cannot replace", cachePath)
}
if err := unix.Access(filepath.Dir(cachePath), unix.W_OK); err != nil {
    return fmt.Errorf("no write permission on %s", filepath.Dir(cachePath))
}

func cachePathWritable(path string) bool {
    dir := filepath.Dir(path)
    f, err := os.CreateTemp(dir, ".wtest")
    if err != nil { return false }
    f.Close(); os.Remove(f.Name())
    return true
}

hdr, err := NewBinaryReader(ctx, logger, bkt, dir, id, sampling, metrics)
if err != nil {
    var pathErr *os.PathError
    if errors.As(err, &pathErr) && (errors.Is(pathErr.Err, syscall.EACCES) || errors.Is(pathErr.Err, syscall.EPERM)) {
        // fix permissions or fall back to memory-only reader
        return NewBinaryReader(ctx, logger, bkt, "", id, sampling, metrics)
    }
    return err
}

When it happens

Trigger: os.RemoveAll(cacheFilename) fails inside newBinaryWriter (via WriteBinary with a non-empty cacheFilename): the path exists but the process lacks write permission on the parent directory, the path is a non-empty directory, the filesystem is read-only, or the path is locked by another process.

Common situations: Running Thanos/objects store bucket download as a non-root user without permissions on the cache directory; cacheFilename pointing at a directory instead of a file; the cache path residing on a read-only mounted volume; stale files owned by a different UID after a container/user change.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• open dir
• read dir
• create working compact directory
• create working downsample directory
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/35bf0cec2784423f. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/indexheader/binary_reader.go:311

	var binWriter PosWriter
	if cacheFilename != "" {
		dir := filepath.Dir(cacheFilename)

		df, err := fileutil.OpenDir(dir)
		if os.IsNotExist(err) {
			if err := os.MkdirAll(dir, os.ModePerm); err != nil {
				return nil, err
			}
			df, err = fileutil.OpenDir(dir)
		}
		if err != nil {
			return nil, err
		}

		defer runutil.CloseWithErrCapture(&err, df, "dir close")

		if err := os.RemoveAll(cacheFilename); err != nil {
			return nil, errors.Wrap(err, "remove any existing index at path")
		}

		var fileWriter *FileWriter
		fileWriter, err = NewFileWriter(cacheFilename, len(buf))
		if err != nil {
			return nil, err
		}
		if err := df.Sync(); err != nil {
			return nil, errors.Wrap(err, "sync dir")
		}
		binWriter = fileWriter
	} else {
		binWriter = NewMemoryWriter(id, len(buf))
	}

	w = &binaryWriter{
		writer: binWriter,

View on GitHub (pinned to 35b8b99117)