thanos-io/thanos · error · ErrorSyncMetaCorrupted

meta.json corrupted

Error message

meta.json corrupted

What it means

ErrorSyncMetaCorrupted is returned by BaseFetcher.loadMeta when meta.json was read successfully but json.Unmarshal could not parse it into metadata.Meta (or filtered fields failed validation earlier). The sentinel wraps the unmarshal error so callers can distinguish corrupt metadata from transient storage errors. Corrupt blocks are skipped and counted in resp.corruptedMetas.

Solutions

1. Re-upload or repair the block: download meta.json, fix its JSON (correct types for ULID, minTime/maxTime as int64, version field), and re-upload.
2. Delete the corrupt block from the bucket (or use thanos tools bucket verify/repair) so syncers stop flagging it.
3. Check the uploader that wrote the block: ensure meta.json is uploaded last and atomically after all block files.
4. Inspect resp.corruptedMetas logs to identify the exact block IDs and inspect those meta.json files.

Example fix

// before: hand-edited meta.json with wrong types
// {"ulid": "01ARZ3...", "minTime": "1600000000000"}

// after: correct types per metadata.Meta schema
// {"ulid": "01ARZ3...", "minTime": 1600000000000, "maxTime": 1600003600000, "version": 1}

content, err := bkt.Get(ctx, metaPath)
if err != nil { return err }
var m metadata.Meta
if err := json.Unmarshal(content, &m); err != nil {
    // corrupt: repair or delete the block before syncing
}
if m.ULID.Compare(id) != 0 || m.ThanOSVersion == 0 {
    // schema mismatch: flag the block
}

func isMetaCorrupted(err error) bool {
    return errors.Is(err, block.ErrorSyncMetaCorrupted)
}

if errors.Is(err, block.ErrorSyncMetaCorrupted) {
    var wrapped *errors.wrapError // extract path via errors.Wrapf message
    log.Warn("corrupt meta.json; excluding block", "err", err)
    // report the block ID to an operator queue for repair
}

When it happens

Trigger: loadMeta: json.Unmarshal(metaContent, m) fails after io.ReadAll succeeded, wrapping ErrorSyncMetaCorrupted with the metaFile path and unmarshal error. FetchBlocks matches errors.Is(err, ErrorSyncMetaCorrupted) and increments resp.corruptedMetas.

Common situations: Partial/truncated uploads to object storage (network interruption during block upload); manually edited or hand-crafted meta.json with wrong JSON types (e.g. string where int expected); empty meta.json files from failed uploads; byte corruption in storage.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• read meta of
• read meta from
• first pass of downsampling failed
• could not sync metas
• read meta

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d18dad81da57e8c4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:432

	if err != nil {
		return nil, err
	}
	return b.NewMetaFetcherWithMetrics(fetcherMetrics, filters), nil
}

// NewMetaFetcher transforms BaseFetcher into actually usable *MetaFetcher.
func (f *BaseFetcher) NewMetaFetcher(reg prometheus.Registerer, filters []MetadataFilter, logTags ...any) *MetaFetcher {
	return f.NewMetaFetcherWithMetrics(NewFetcherMetrics(reg, nil, nil), filters, logTags...)
}

// NewMetaFetcherWithMetrics transforms BaseFetcher into actually usable *MetaFetcher.
func (f *BaseFetcher) NewMetaFetcherWithMetrics(fetcherMetrics *FetcherMetrics, filters []MetadataFilter, logTags ...any) *MetaFetcher {
	return &MetaFetcher{metrics: fetcherMetrics, wrapped: f, filters: filters, logger: log.With(f.logger, logTags...)}
}

var (
	ErrorSyncMetaNotFound  = errors.New("meta.json not found")
	ErrorSyncMetaCorrupted = errors.New("meta.json corrupted")
)

func (f *BaseFetcher) metaUpdated(id ulid.ULID, modified time.Time) bool {
	if f.modifiedTimestamps[id].IsZero() {
		return false
	}
	return !f.modifiedTimestamps[id].Equal(modified)
}

func (f *BaseFetcher) bustCacheForID(id ulid.ULID) {
	f.cacheBusts.Inc()

	f.cached.Delete(id)
	if err := os.RemoveAll(filepath.Join(f.cacheDir, id.String())); err != nil {
		level.Warn(f.logger).Log("msg", "failed to remove cached meta.json dir", "dir", filepath.Join(f.cacheDir, id.String()), "err", err)
	}
}

View on GitHub (pinned to 35b8b99117)