thanos-io/thanos · error

get block metas from oldest

Error message

get block metas from oldest

What it means

Wraps errors from blockMetasFromOldest() during AreAllBlocksUploaded, which walks the local directory and re-reads every block's meta.json sorted from oldest to newest. The shipper needs the full local block list to compare against the uploaded manifest, so any read/parse failure of a block meta file aborts the check.

Solutions

1. Check the wrapped cause; restore or remove the offending block's meta.json / block directory.
2. Retry the check — transient directory races with Prometheus retention usually resolve on the next call.
3. Verify the data directory is the actual TSDB dir (not a partially-mounted path).
4. Run bucket verification tooling to confirm the affected block is safe in object storage before deleting it locally.

info, err := os.Stat(dataDir)
if err != nil || !info.IsDir() {
    return fmt.Errorf("data dir unreadable: %w", err)
}

ok, err := shipper.AreAllBlocksUploaded()
if err != nil && strings.Contains(err.Error(), "get block metas from oldest") {
    time.Sleep(retryInterval)
    ok, err = shipper.AreAllBlocksUploaded() // transient dir-walk races often clear
}

When it happens

Trigger: Calling AreAllBlocksUploaded (used by sidecar readiness/gateway) when any block directory under s.dir has a corrupt, missing, or unreadable meta.json, or directory traversal fails with an I/O error.

Common situations: Corrupted block after unclean restart; NFS/对象 storage mount flakiness; blocks being deleted by Prometheus retention while the check walks the directory; wrong data directory permissions.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• get all block meta.
• create meta fetcher
• create working compact directory
• create working downsample directory
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/75a18a6d35019b65. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:312

	// TODO(bwplotka) so confusing! we need to sort it first. Add comment to TSDB code.
	metas := append([]tsdb.BlockMeta{newMeta}, c.metas...)
	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime
	})
	if o := tsdb.OverlappingBlocks(metas); len(o) > 0 {
		// TODO(bwplotka): Consider checking if overlaps relates to block in concern?
		return errors.Errorf("shipping compacted block %s is blocked; overlap spotted: %s", newMeta.ULID, o.String())
	}
	return nil
}

func (s *Shipper) AreAllBlocksUploaded() (bool, error) {
	s.mtx.RLock()
	defer s.mtx.RUnlock()

	metas, _, err := s.blockMetasFromOldest()
	if err != nil {
		return false, errors.Wrap(err, "get block metas from oldest")
	}

	if len(metas) == 0 {
		return true, nil
	}

	meta, err := ReadMetaFile(s.metadataFilePath)
	if err != nil {
		return false, errors.Wrap(err, "read meta file")
	}

	uploaded := make(map[ulid.ULID]struct{}, len(meta.Uploaded))
	for _, id := range meta.Uploaded {
		uploaded[id] = struct{}{}
	}

	for _, m := range metas {
		if _, ok := uploaded[m.ULID]; !ok {

View on GitHub (pinned to 35b8b99117)