thanos-io/thanos · error

stat

Error message

stat %v

What it means

GatherFileStats stats the block's index file (<blockDir>/index) and wraps the os.Stat error with this message. The index file is a mandatory part of every TSDB block; its absence means the block is incomplete or corrupted on disk.

Solutions

1. Verify index exists: ls <blockDir>/index. If missing, the block is incomplete — delete it or re-create it from a healthy replica.
2. Check permissions on the block dir and index file.
3. Ensure the block is fully written (block ID dir contains meta.json, index, chunks/) before uploading.
4. Investigate why Prometheus did not finish writing the block (crash logs, OOM).

Example fix

// before
indexFile, err := os.Stat(filepath.Join(blockDir, IndexFilename))
if err != nil {
	return nil, errors.Wrapf(err, "stat %v", filepath.Join(blockDir, IndexFilename))
}
// after
indexPath := filepath.Join(blockDir, IndexFilename)
indexFile, err := os.Stat(indexPath)
if os.IsNotExist(err) {
	return nil, errors.Errorf("block %s is incomplete: index file %s missing", blockDir, indexPath)
} else if err != nil {
	return nil, errors.Wrapf(err, "stat %v", indexPath)
}

indexPath := filepath.Join(blockDir, "index")
if fi, err := os.Stat(indexPath); err != nil {
	return fmt.Errorf("block %s missing index: %w", blockDir, err)
} else if fi.Size() == 0 {
	return fmt.Errorf("block %s has empty index file", blockDir)
}

func hasIndexFile(blockDir string) bool {
	fi, err := os.Stat(filepath.Join(blockDir, "index"))
	return err == nil && !fi.IsDir() && fi.Size() > 0
}

files, err := block.GatherFileStats(blockDir, hashFunc, logger)
if err != nil && os.IsNotExist(errors.Unwrap(err)) {
	return skipIncompleteBlock(blockDir) // quarantine/delete, don't retry
}

When it happens

Trigger: upload() calls GatherFileStats on a block dir where <blockDir>/index does not exist (never written or deleted) or cannot be stat'ed due to permissions/I-O errors.

Common situations: Incomplete block left by a crashed Prometheus (head compaction interrupted); retention/cleanup removed the index prematurely; wrong data dir; blocks partially restored from backup missing index.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• read dir
• open block
• start remote write agent db
• open TSDB
• remove storage lock files

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/bd1b12357c2a71b5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:354

		}

		mf := metadata.File{
			RelPath:   filepath.Join(ChunksDirname, f.Name()),
			SizeBytes: fi.Size(),
		}
		if hf != metadata.NoneFunc && !f.IsDir() {
			h, err := metadata.CalculateHash(filepath.Join(blockDir, ChunksDirname, f.Name()), hf, logger)
			if err != nil {
				return nil, errors.Wrapf(err, "calculate hash %v", filepath.Join(ChunksDirname, f.Name()))
			}
			mf.Hash = &h
		}
		res = append(res, mf)
	}

	indexFile, err := os.Stat(filepath.Join(blockDir, IndexFilename))
	if err != nil {
		return nil, errors.Wrapf(err, "stat %v", filepath.Join(blockDir, IndexFilename))
	}
	mf := metadata.File{
		RelPath:   indexFile.Name(),
		SizeBytes: indexFile.Size(),
	}
	if hf != metadata.NoneFunc {
		h, err := metadata.CalculateHash(filepath.Join(blockDir, IndexFilename), hf, logger)
		if err != nil {
			return nil, errors.Wrapf(err, "calculate hash %v", indexFile.Name())
		}
		mf.Hash = &h
	}
	res = append(res, mf)

	metaFile, err := os.Stat(filepath.Join(blockDir, MetaFilename))
	if err != nil {
		return nil, errors.Wrapf(err, "stat %v", filepath.Join(blockDir, MetaFilename))
	}

View on GitHub (pinned to 35b8b99117)