thanos-io/thanos · error

read meta from

Error message

read meta from %s

What it means

This error is wrapped around a failure to read a TSDB block's meta.json from the directory the compactor downloaded it into (bdir), during repair of an overlapping/issue-347 block inside Thanos' compact.Group repair flow (compact.go repairBucketBlock). The underlying error comes from metadata.ReadFromDir, which parses and validates meta.json (ULID, time range, compaction level, Thanos version, labels). The wrap preserves the block directory path so operators can locate the corrupted download.

Solutions

1. Inspect the block directory in the bucket (thanos tools bucket verify or ls) and check meta.json contents for corruption or missing fields
2. Delete or move the broken block out of the bucket so the compactor no longer tries to repair it (only if it is not referenced/needed), then let the compactor resync
3. Re-download the block manually (thanos tools bucket download) to rule out transient download corruption and confirm the object in storage itself is intact
4. Check the producing side: ensure Prometheus sidecar/uploaded blocks have valid, complete meta.json (upgrade Thanos/Prometheus if very old format)
5. If object storage returned partial data consistently, verify bucket permissions and that the bucket is not being concurrently modified by another compactor

Example fix

// before (manual diagnosis of the failing dir)
// compactor logs: read meta from /var/thanos/compact/tmp/repair/01ARZ3N...

// after: validate the block in the bucket before compaction runs
$ thanos tools bucket verify --objstore.config-file=bucket.yaml
$ thanos tools bucket inspect --objstore.config-file=bucket.yaml
// if meta.json is truly corrupt, remove the broken block:
$ thanos tools bucket webhooks... # or manually delete the block ULID dir in the bucket

// before running the compactor, verify every block in the bucket parses
err := bucket.Iterate(ctx, "", func(id string) error {
    return nil // enumerate block ULIDs first
})
meta, err := metadata.ReadFromDir(blockDir)
if err != nil {
    logger.Warn("skipping unreadable block", "dir", blockDir, "err", err)
}
// preflight: thanos tools bucket verify --objstore.config-file=bucket.yaml

When it happens

Trigger: metadata.ReadFromDir(bdir) returns an error during repairBucketBlock after block.Download succeeds: meta.json missing from the downloaded block directory, meta.json unparseable JSON, meta.json failing validation (invalid ULID, minTime>maxTime, missing thanos.version field for Thanos-written blocks, mismatched block id between directory name and meta), or the download produced an empty/partial directory that still passed block.Download's checks.

Common situations: Object storage corruption or truncated uploads of meta.json; blocks uploaded by very old Prometheus/Thanos versions lacking required meta fields; partial block downloads from flaky object storage backends (S3 throttling, network interruption) where the index downloaded fine but meta.json did not; a bucket containing a directory that is not a real TSDB block (e.g. a marker file or partial upload from a crashed uploader).

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• could not sync metas
• read meta of
• meta.json corrupted
• invalid magic number
• repair failed for block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8d86223916a0bb48. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1136

	tmpdir, err := os.MkdirTemp("", fmt.Sprintf("repair-issue-347-id-%s-", ie.id))
	if err != nil {
		return err
	}

	defer func() {
		if err := os.RemoveAll(tmpdir); err != nil {
			level.Warn(logger).Log("msg", "failed to remote tmpdir", "err", err, "tmpdir", tmpdir)
		}
	}()

	bdir := filepath.Join(tmpdir, ie.id.String())
	if err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {
		return retry(errors.Wrapf(err, "download block %s", ie.id))
	}

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return errors.Wrapf(err, "read meta from %s", bdir)
	}

	resid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", ie.id)
	}

	// Verify repaired id before uploading it.
	if err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {
		return retry(errors.Wrapf(err, "upload of %s failed", resid))
	}

	level.Info(logger).Log("msg", "deleting broken block", "id", ie.id)

View on GitHub (pinned to 35b8b99117)