thanos-io/thanos · warning · ErrorSyncMetaNotFound

meta.json not found

Error message

meta.json not found

What it means

ErrorSyncMetaNotFound is returned by BaseFetcher.loadMeta when meta.json for a block disappears from object storage between the bucket Exists check and the read. The loadMeta code wraps the underlying bucket not-found error with this sentinel so callers can match it with errors.Is/As. It signals a benign race: the block was deleted concurrently (typically by a compactor/GC) while the syncer was fetching its metadata.

Solutions

1. Treat it as benign: noMetas is counted and the block is skipped; retry the sync later — the block is gone from the bucket view.
2. Increase block deletion delay (--store.grpc.series-sample-limit irrelevant; use --objstore.* / compactor deletion-delay) so concurrent syncers finish reading before GC deletes blocks.
3. Check whether another process (compactor, manual cleanup, S3 lifecycle policy) is deleting blocks; pause or reconfigure it.
4. If this occurs persistently for blocks that still exist, verify bucket credentials/permissions and that IsObjNotFoundErr is configured correctly for your object store.

Example fix

// before (reactive: block vanishes mid-sync)
// compactor deletion-delay too small

// after: give concurrent readers time
// thanos compact --delete-delay=48h
// (ensures blocks removed by compaction stay readable for in-flight syncs)

exists, err := bkt.Exists(ctx, metaPath)
if err == nil && !exists {
    // skip block before calling the fetcher; it's already gone
}

func isMetaNotFound(err error) bool {
    return errors.Is(err, block.ErrorSyncMetaNotFound)
}

m, err := metaFetcher.Fetch(ctx, metas)
switch {
case err == nil:
    // use metas
case errors.Is(err, block.ErrorSyncMetaNotFound):
    // benign: block deleted concurrently; log at debug, retry next sync
case errors.As(err, &e): // inspect wrapped cause
    // unexpected: handle genuine storage error
}

When it happens

Trigger: loadMeta: f.bkt.Exists returns true for a block's meta.json, then ReaderWithExpectedErrs(...).Get(ctx, metaFile) fails with an object-not-found error that satisfies f.bkt.IsObjNotFoundErr; the fetcher wraps it in ErrorSyncMetaNotFound. In fetchBlocks this causes the block to be counted in resp.noMetas and skipped.

Common situations: Compactor/GC deletes blocks concurrently with a Store/Querier syncing blocks; two Thanos instances racing where one removes blocks (deletion delay too short); manual block deletion or lifecycle rules in S3/GCS removing objects mid-sync; stale bucket view from eventually-consistent object stores.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• %v
• get meta file
• read meta file
• upload of failed
• compaction

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f7bdc98d4fab9925. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:431

	b, err := NewBaseFetcherWithMetrics(logger, concurrency, bkt, blockIDsFetcher, dir, baseFetcherMetrics)
	if err != nil {
		return nil, err
	}
	return b.NewMetaFetcherWithMetrics(fetcherMetrics, filters), nil
}

// NewMetaFetcher transforms BaseFetcher into actually usable *MetaFetcher.
func (f *BaseFetcher) NewMetaFetcher(reg prometheus.Registerer, filters []MetadataFilter, logTags ...any) *MetaFetcher {
	return f.NewMetaFetcherWithMetrics(NewFetcherMetrics(reg, nil, nil), filters, logTags...)
}

// NewMetaFetcherWithMetrics transforms BaseFetcher into actually usable *MetaFetcher.
func (f *BaseFetcher) NewMetaFetcherWithMetrics(fetcherMetrics *FetcherMetrics, filters []MetadataFilter, logTags ...any) *MetaFetcher {
	return &MetaFetcher{metrics: fetcherMetrics, wrapped: f, filters: filters, logger: log.With(f.logger, logTags...)}
}

var (
	ErrorSyncMetaNotFound  = errors.New("meta.json not found")
	ErrorSyncMetaCorrupted = errors.New("meta.json corrupted")
)

func (f *BaseFetcher) metaUpdated(id ulid.ULID, modified time.Time) bool {
	if f.modifiedTimestamps[id].IsZero() {
		return false
	}
	return !f.modifiedTimestamps[id].Equal(modified)
}

func (f *BaseFetcher) bustCacheForID(id ulid.ULID) {
	f.cacheBusts.Inc()

	f.cached.Delete(id)
	if err := os.RemoveAll(filepath.Join(f.cacheDir, id.String())); err != nil {
		level.Warn(f.logger).Log("msg", "failed to remove cached meta.json dir", "dir", filepath.Join(f.cacheDir, id.String()), "err", err)
	}
}

View on GitHub (pinned to 35b8b99117)