thanos-io/thanos · error
getting file info
Error message
getting file info %v
What it means
While iterating entries of the chunks/ directory, GatherFileStats calls f.Info() to get os.FileInfo and wraps its error with this message. It fails when a directory entry cannot be stat'ed — typically because the file was removed between ReadDir and Info (lstat on a dangling entry) or an I/O error occurred.
Solutions
- Stop concurrent processes from deleting the block while GatherFileStats runs (e.g., don't upload blocks still owned by Prometheus retention).
- Re-scan the directory; if the block is being actively compacted, retry after compaction completes.
- Check filesystem health (dmesg, fsck) if lstat errors persist.
- Verify permissions allow lstat on all chunk files.
Example fix
// before
fi, err := f.Info()
if err != nil {
return nil, errors.Wrapf(err, "getting file info %v", filepath.Join(ChunksDirname, f.Name()))
}
// after
fi, err := f.Info()
if err != nil {
if os.IsNotExist(err) {
continue // file vanished mid-scan; skip it
}
return nil, errors.Wrapf(err, "getting file info %v", filepath.Join(ChunksDirname, f.Name()))
} Defensive patterns
Strategy: retry
Validate before calling
entries, err := os.ReadDir(filepath.Join(blockDir, "chunks"))
if err != nil { return err }
for _, e := range entries {
if _, err := e.Info(); err != nil {
return fmt.Errorf("chunk entry %s unreadable pre-upload: %w", e.Name(), err)
}
} Try / catch
files, err := block.GatherFileStats(blockDir, hashFunc, logger)
if err != nil {
// transient race with deletion: retry once after short delay
err = retryWithBackoff(ctx, 3, 500*time.Millisecond, func() error {
files, err = block.GatherFileStats(blockDir, hashFunc, logger)
return err
})
} Prevention
- Avoid uploading blocks that may be concurrently compacted or retention-deleted.
- Run uploads and data cleanup from the same lifecycle owner (sidecar watches Prometheus).
- Skip retries when the block dir itself is gone (OS.IsNotExist on the parent).
- Monitor for duplicated cleanup jobs racing on shared volumes.
When it happens
Trigger: A chunk file inside <blockDir>/chunks is deleted (e.g., concurrent cleanup/compaction) after ReadDir listed it, or the entry can't be stat'ed due to filesystem/I/O errors.
Common situations: Concurrent deletion of the block while uploading; node with flaky disks; sticky/permission issues preventing lstat on a chunk file.
Understand the failure class
Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.
Related errors
AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07).
Data as JSON: /api/errors/9575cf8c1c939552.
Report an issue: GitHub.
Appendix: source
Thrown at pkg/block/block.go:335
// ReadDir returns files in sorted order already.
var result []string
for _, f := range files {
result = append(result, f.Name())
}
return result
}
// GatherFileStats returns metadata.File entry for files inside TSDB block (index, chunks, meta.json).
func GatherFileStats(blockDir string, hf metadata.HashFunc, logger log.Logger) (res []metadata.File, _ error) {
files, err := os.ReadDir(filepath.Join(blockDir, ChunksDirname))
if err != nil {
return nil, errors.Wrapf(err, "read dir %v", filepath.Join(blockDir, ChunksDirname))
}
for _, f := range files {
fi, err := f.Info()
if err != nil {
return nil, errors.Wrapf(err, "getting file info %v", filepath.Join(ChunksDirname, f.Name()))
}
mf := metadata.File{
RelPath: filepath.Join(ChunksDirname, f.Name()),
SizeBytes: fi.Size(),
}
if hf != metadata.NoneFunc && !f.IsDir() {
h, err := metadata.CalculateHash(filepath.Join(blockDir, ChunksDirname, f.Name()), hf, logger)
if err != nil {
return nil, errors.Wrapf(err, "calculate hash %v", filepath.Join(ChunksDirname, f.Name()))
}
mf.Hash = &h
}
res = append(res, mf)
}
indexFile, err := os.Stat(filepath.Join(blockDir, IndexFilename))
if err != nil {View on GitHub (pinned to 35b8b99117)