thanos-io/thanos · error · ErrorSyncMetaCorrupted

meta.json unmarshal

Error message

meta.json %v unmarshal: %v

What it means

loadMeta in BaseFetcher reads a block's meta.json from the object store and unmarshals it into metadata.Meta. When json.Unmarshal fails, the error is wrapped with ErrorSyncMetaCorrupted and this message. It means the block's meta.json exists but is not valid JSON (or not valid for the Meta struct), so the block is treated as corrupted during bucket sync.

Solutions

1. Check the raw meta.json of the failing block in the bucket (download it and run it through a JSON validator); re-upload or re-truncate the block if corrupted.
2. Delete the corrupted block's meta.json or the partial block so sync treats it as partial rather than corrupted, then re-sync.
3. Verify the object store configuration/gateway isn't returning truncated objects (checksums, network).
4. Re-create the block by re-uploading from the source Prometheus/Compactor instead of hand-repairing meta.json.

Example fix

// before: block meta.json truncated in bucket
{"ulid": "01ARZ3NDEKTSV4RRFFQ69G5FAV", "minTime": 1577836800000
// after: complete valid meta.json (re-upload block)
{"ulid":"01ARZ3NDEKTSV4RRFFQ69G5FAV","minTime":1577836800000,"maxTime":1577923200000,"stats":{"numSamples":0},"compaction":{"level":1,"sources":["01ARZ3NDEKTSV4RRFFQ69G5FAV"]},"version":1}

var raw map[string]any
if err := json.Unmarshal(metaContent, &raw); err != nil {
    // meta.json is not valid JSON before syncing: repair or re-upload block
}

func isCorruptedMetaError(err error) bool {
    return errors.Is(err, block.ErrorSyncMetaCorrupted) || strings.Contains(err.Error(), "unmarshal")
}

metas, err := fetcher.FetchMetadata(ctx, ...)
if err != nil {
    var corrupted []ulid.ULID
    if errors.Is(err, block.ErrorSyncMetaCorrupted) || strings.Contains(err.Error(), "meta.json") {
        // re-download meta.json, validate JSON, re-upload or exclude block, then retry sync
    } else {
        return err
    }
}

When it happens

Trigger: Calling fetchMeta/fetchMetadata (bucket sync) on a block whose meta.json is truncated, empty, non-JSON bytes, or has fields with wrong JSON types (e.g. version as string) so json.Unmarshal into *metadata.Meta fails.

Common situations: Interrupted block upload leaving a partial meta.json; manual editing of meta.json in the bucket; an older/other Thanos or Prometheus version writing a schema json.Unmarshal cannot accept; corrupted object-store objects (misconfigured storage returning garbage).

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• block has no meta file
• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• create meta fetcher

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e591e63afd779fc1. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:496

	r, err := f.bkt.ReaderWithExpectedErrs(f.bkt.IsObjNotFoundErr).Get(ctx, metaFile)
	if f.bkt.IsObjNotFoundErr(err) {
		// Meta.json was deleted between bkt.Exists and here.
		return nil, errors.Wrapf(ErrorSyncMetaNotFound, "%v", err)
	}
	if err != nil {
		return nil, errors.Wrapf(err, "get meta file: %v", metaFile)
	}

	defer runutil.CloseWithLogOnErr(f.logger, r, "close bkt meta get")

	metaContent, err := io.ReadAll(r)
	if err != nil {
		return nil, errors.Wrapf(err, "read meta file: %v", metaFile)
	}

	m := &metadata.Meta{}
	if err := json.Unmarshal(metaContent, m); err != nil {
		return nil, errors.Wrapf(ErrorSyncMetaCorrupted, "meta.json %v unmarshal: %v", metaFile, err)
	}

	if m.Version != metadata.TSDBVersion1 {
		return nil, errors.Errorf("unexpected meta file: %s version: %d", metaFile, m.Version)
	}

	// Best effort cache in local dir.
	if f.cacheDir != "" {
		if err := os.MkdirAll(cachedBlockDir, os.ModePerm); err != nil {
			level.Warn(f.logger).Log("msg", "best effort mkdir of the meta.json block dir failed; ignoring", "dir", cachedBlockDir, "err", err)
		}

		if err := m.WriteToDir(f.logger, cachedBlockDir); err != nil {
			level.Warn(f.logger).Log("msg", "best effort save of the meta.json to local dir failed; ignoring", "dir", cachedBlockDir, "err", err)
		}
	}
	return m, nil
}

View on GitHub (pinned to 35b8b99117)