thanos-io/thanos · error

unmarshal meta.json for block

Error message

unmarshal meta.json for block %s

What it means

DownloadMeta successfully read meta.json bytes from object storage but json.Unmarshal could not decode them into metadata.Meta. This means the stored meta.json is corrupt, empty, or not valid JSON matching the metadata schema.

Solutions

1. Inspect the meta.json object in the bucket (cat it via mc/aws s3 cp) to see the corruption.
2. Delete the corrupted block from the bucket if it's not needed, or re-upload a valid meta.json.
3. Verify the uploader that wrote the block (network interruption during upload?).
4. Check for storage backend corruption and restore from backup if blocks are broadly affected.

Example fix

// before
if err = json.Unmarshal(obj, &m); err != nil {
	return metadata.Meta{}, errors.Wrapf(err, "unmarshal meta.json for block %s", id.String())
}
// after
if err = json.Unmarshal(obj, &m); err != nil {
	return metadata.Meta{}, errors.Wrapf(err, "unmarshal meta.json for block %s (len=%d, body=%q)", id.String(), len(obj), string(obj))
}

rc, err := bkt.Get(ctx, path.Join(id.String(), metadata.MetaFilename))
if err != nil { return err }
defer rc.Close()
obj, err := io.ReadAll(rc)
if err != nil { return err }
if !json.Valid(obj) {
	return fmt.Errorf("meta.json for block %s is not valid JSON (len=%d)", id, len(obj))
}

func isValidMetaJSON(b []byte) bool {
	if len(b) == 0 || !json.Valid(b) { return false }
	var m metadata.Meta
	return json.Unmarshal(b, &m) == nil
}

m, err := block.DownloadMeta(ctx, logger, bkt, id)
if err != nil && strings.Contains(err.Error(), "unmarshal meta.json") {
	// corrupted meta: quarantine or delete the block, do not retry
	return deleteCorruptBlock(ctx, bkt, id)
}

When it happens

Trigger: io.ReadAll succeeded but json.Unmarshal(obj, &m) returned an error — malformed JSON, truncated upload, empty object, or meta.json written by an incompatible producer.

Common situations: Partially written meta.json from a failed upload; a manually edited/deleted-then-recreated meta.json; object corrupted by a buggy uploader or storage backend; block written by a different Thanos/TSDB version with incompatible fields.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• retention failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8634acc201a3df40. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:295

// DownloadMeta downloads only meta file from bucket by block ID.
// TODO(bwplotka): Differentiate between network error & partial upload.
func DownloadMeta(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) (metadata.Meta, error) {
	rc, err := bkt.Get(ctx, path.Join(id.String(), MetaFilename))
	if err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "meta.json bkt get for %s", id.String())
	}
	defer runutil.CloseWithLogOnErr(logger, rc, "download meta bucket client")

	var m metadata.Meta

	obj, err := io.ReadAll(rc)
	if err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "read meta.json for block %s", id.String())
	}

	if err = json.Unmarshal(obj, &m); err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "unmarshal meta.json for block %s", id.String())
	}

	return m, nil
}

func IsBlockMetaFile(path string) bool {
	return filepath.Base(path) == MetaFilename
}

func IsBlockDir(path string) (id ulid.ULID, ok bool) {
	id, err := ulid.Parse(filepath.Base(path))
	return id, err == nil
}

// GetSegmentFiles returns list of segment files for given block. Paths are relative to the chunks directory.
// In case of errors, nil is returned.
func GetSegmentFiles(blockDir string) []string {
	files, err := os.ReadDir(filepath.Join(blockDir, ChunksDirname))

View on GitHub (pinned to 35b8b99117)