thanos-io/thanos · error

read metadata for block

Error message

read metadata for block %v

What it means

blockMetasFromOldest iterates the shipper's local data directory and calls metadata.ReadFromDir on every block directory to load its meta.json. When reading or parsing a block's metadata fails and skipCorruptedBlocks is false, the whole sync/upload iteration is aborted with this wrapped error naming the block directory. It signals that a local TSDB block is unreadable or its meta.json is missing/corrupt, so its upload state cannot be determined.

Solutions

1. Set Shipper.WithSkipCorruptedBlocks(true) so bad blocks are logged and skipped (Sync then returns ErrorSyncBlockCorrupted) instead of failing outright.
2. Inspect the named block directory: verify <dir>/meta.json exists and is valid JSON with the expected fields (ulid, minTime, maxTime, version).
3. Delete the corrupted block directory from the data dir and let Prometheus/Thanos re-sync it, or restore it from a healthy replica.
4. Check filesystem health (dmesg, fsck) if multiple blocks fail to read; a failing volume usually corrupts several blocks at once.

Example fix

// before: shipper aborts on any unreadable block meta
shipper.NewPrometheusShipper(...)
// after: tolerate corrupted blocks and let sync report them
shipper.NewPrometheusShipper(...).WithSkipCorruptedBlocks(true)

meta, err := metadata.ReadFromDir(blockDir)
if err != nil {
	level.Error(logger).Log("msg", "skipping corrupted block", "block", blockDir, "err", err)
	return nil // skip instead of failing the whole sync
}

func blockReadable(dir string) bool {
	_, err := os.Stat(filepath.Join(dir, "meta.json"))
	return err == nil
}

metas, failed, err := sh.Sync(ctx)
if errors.Is(err, shipper.ErrorSyncBlockCorrupted) {
	level.Warn(logger).Log("msg", "some blocks corrupted, skipped", "failed", failed)
	return nil
} else if err != nil {
	return errors.Wrap(err, "sync")
}

When it happens

Trigger: Calling Sync() or AreAllBlocksUploaded() on a Shipper whose local dir contains a block directory whose meta.json is missing, truncated, fails JSON validation, or cannot be stat/read, with skipCorruptedBlocks=false.

Common situations: Crash or OOM-kill mid-compaction leaving a partially written block; disk corruption on the Prometheus/Thanos sidecar data volume; manually deleted or partially copied block directories; NFS/EFS volumes dropping files; running an older Prometheus producing meta files the current metadata parser rejects.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• unable to access path
• unable to open tbc directory
• read dir
• getting file info
• calculate hash

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e56d73b0f782e1c5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:565

		if err != nil {
			if s.skipCorruptedBlocks {
				level.Error(s.logger).Log("msg", "stat block", "err", err, "block", dir)
				failedBlocks = append(failedBlocks, n)
				continue
			}
			return nil, nil, errors.Wrapf(err, "stat block %v", dir)
		}
		if !fi.IsDir() {
			continue
		}
		m, err := metadata.ReadFromDir(dir)
		if err != nil {
			if s.skipCorruptedBlocks {
				level.Error(s.logger).Log("msg", "read metadata for block", "err", err, "block", dir)
				failedBlocks = append(failedBlocks, n)
				continue
			}
			return nil, nil, errors.Wrapf(err, "read metadata for block %v", dir)
		}
		metas = append(metas, m)
	}
	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime
	})

	if len(failedBlocks) > 0 {
		err = ErrorSyncBlockCorrupted
	}
	return metas, failedBlocks, err
}

func hardlinkBlock(src, dst string) error {
	chunkDir := filepath.Join(dst, block.ChunksDirname)

	if err := os.MkdirAll(chunkDir, 0750); err != nil {
		return errors.Wrap(err, "create chunks dir")

View on GitHub (pinned to 35b8b99117)