What causes "get all block meta." in thanos-io/thanos?

Inspect the wrapped cause in the error message and fix/restore the failing block's meta.json (re-copy the block or re-ingest it).

thanos-io/thanos · error

get all block meta.

Error message

get all block meta.

What it means

Wraps any error encountered while the Syncer's meta sync scans the local TSDB directory and appends each block's metadata (via a bblock reader callback) into the in-memory meta list. It fires when reading/parsing a block meta.json under the shipper's directory fails (I/O error, corrupt/missing meta.json, invalid ULID directory name). The shipper needs a complete meta snapshot before it can decide overlaps or uploads, so a partial read aborts the whole sync setup.

Solutions

Inspect the wrapped cause in the error message and fix/restore the failing block's meta.json (re-copy the block or re-ingest it).
Run `thanos tools bucket verify` against the bucket and re-upload the affected block.
Remove the corrupt block directory from the local TSDB dir if its data is safely in object storage, then restart.
Check filesystem permissions and disk health (dmesg, mount rw) on the data directory.

Example fix

// before
// shipper fails at startup: get all block meta.: open /data/01ARZ3.../meta.json: no such file or directory
// after
// restore meta.json for the listed block or remove the corrupt block dir:
// rm -rf /data/01ARZ3NDEKTSV4RRFFQ69G5FAV   # only if block already uploaded
// then restart thanos sidecar

Defensive patterns

Strategy: validation

Validate before calling

for _, b := range blockDirs {
    if _, err := os.Stat(filepath.Join(b, "meta.json")); err != nil {
        log.Printf("skipping block with missing meta.json: %s: %v", b, err)
    }
}

Try / catch

if _, err := shipper.Sync(ctx); err != nil {
    if strings.Contains(err.Error(), "get all block meta.") {
        log.Printf("corrupt local block meta, needs repair: %v", err)
    }
}

Prevention

Use graceful shutdown for Prometheus to avoid truncated meta.json files
Monitor sidecar logs for block-scan warnings instead of waiting for sync failure
Never manually copy block directories without their meta.json
Run bucket verification periodically to detect corrupt blocks early

When it happens

Trigger: Calling Sync (or anything triggering checker sync, e.g. IsOverlapping on a non-synced checker) when a block directory under s.dir contains a corrupt or unreadable meta.json, or a directory walker I/O error occurs.

Common situations: Disk corruption or truncated meta.json after an unclean shutdown; manually copied block directories missing meta.json; permission problems on the TSDB dir; partially downloaded/deleted blocks while Prometheus is running.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a0f2fa404e6a78d8. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:279

		if !ok {
			return nil
		}

		m, err := block.DownloadMeta(ctx, c.logger, c.bucket, id)
		if err != nil {
			return err
		}

		if !labels.Equal(labels.FromMap(m.Thanos.Labels), c.labels()) {
			return nil
		}

		c.metas = append(c.metas, m.BlockMeta)
		c.lookupMetas[m.ULID] = struct{}{}
		return nil

	}); err != nil {
		return errors.Wrap(err, "get all block meta.")
	}

	c.synced = true
	return nil
}

func (c *lazyOverlapChecker) IsOverlapping(ctx context.Context, newMeta tsdb.BlockMeta) error {
	if !c.synced {
		level.Info(c.logger).Log("msg", "gathering all existing blocks from the remote bucket for check", "id", newMeta.ULID.String())
		if err := c.sync(ctx); err != nil {
			return err
		}
	}

	// TODO(bwplotka) so confusing! we need to sort it first. Add comment to TSDB code.
	metas := append([]tsdb.BlockMeta{newMeta}, c.metas...)
	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime

View on GitHub (pinned to 35b8b99117)