thanos-io/thanos · error

stat block

Error message

stat block %v

What it means

This error wraps os.Stat failures on an individual block directory path during blockMetasFromOldest. When s.skipCorruptedBlocks is false (default), any entry in the shipper dir whose stat fails aborts the whole Sync. When skipping is enabled, the block is logged and added to failedBlocks instead.

Solutions

1. Remove the broken entry (dangling symlink/corrupt dir) from the shipper dir, then re-run Sync.
2. Enable skipCorruptedBlocks (if using the Shipper API directly) so bad blocks are logged and skipped instead of failing the sync.
3. Fix permissions on the offending subdirectory so stat succeeds.
4. Run fsck / verify the volume if entries are corrupted after a crash.

Example fix

// before: one dangling symlink fails the whole sync
return nil, nil, errors.Wrapf(err, "stat block %v", dir)

// after: construct shipper with skipping enabled
shipper.New(..., shipper.WithSkipCorruptedBlocks())

entries, err := os.ReadDir(shipperDir)
if err != nil {
    return err
}
for _, e := range entries {
    p := filepath.Join(shipperDir, e.Name())
    if _, err := os.Stat(p); err != nil {
        log.Printf("removing dangling entry %s: %v", p, err)
        os.RemoveAll(p)
    }
}

func entryAccessible(base, name string) bool {
    _, err := os.Stat(filepath.Join(base, name))
    return err == nil
}

if err != nil {
    if s.skipCorruptedBlocks {
        level.Error(s.logger).Log("msg", "stat block", "err", err, "block", dir)
        failedBlocks = append(failedBlocks, n)
        continue
    }
    return nil, nil, errors.Wrapf(err, "stat block %v", dir)
}

When it happens

Trigger: Calling Shipper.Sync or AreAllBlocksUploaded when an entry inside the shipper dir cannot be stat'ed: a dangling symlink, an entry removed concurrently between ReadDir and Stat, a broken filesystem entry, or permission denial on a subdirectory.

Common situations: Partial block deletion by retention cleanup leaving dangling entries; permission mismatch between Prometheus (creator) and Thanos sidecar (reader) users; corrupted directory entries after unclean shutdown/power loss.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• hard link block
• write meta file
• open dir
• read dir
• open data dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/539d19769862870a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:553

	names := make([]string, 0, len(fis))
	for _, fi := range fis {
		names = append(names, fi.Name())
	}
	for _, n := range names {
		if _, ok := block.IsBlockDir(n); !ok {
			continue
		}
		dir := filepath.Join(s.dir.Name(), n)

		fi, err := s.dir.Stat(n)
		if err != nil {
			if s.skipCorruptedBlocks {
				level.Error(s.logger).Log("msg", "stat block", "err", err, "block", dir)
				failedBlocks = append(failedBlocks, n)
				continue
			}
			return nil, nil, errors.Wrapf(err, "stat block %v", dir)
		}
		if !fi.IsDir() {
			continue
		}
		m, err := metadata.ReadFromDir(dir)
		if err != nil {
			if s.skipCorruptedBlocks {
				level.Error(s.logger).Log("msg", "read metadata for block", "err", err, "block", dir)
				failedBlocks = append(failedBlocks, n)
				continue
			}
			return nil, nil, errors.Wrapf(err, "read metadata for block %v", dir)
		}
		metas = append(metas, m)
	}
	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime
	})

View on GitHub (pinned to 35b8b99117)