{"record":{"id":"f7bdc98d4fab9925","repo":"thanos-io/thanos","slug":"meta-json-not-found","errorCode":null,"errorMessage":"meta.json not found","messagePattern":"meta\\.json not found","errorType":"error_code","errorClass":"ErrorSyncMetaNotFound","httpStatus":null,"severity":"warning","filePath":"pkg/block/fetcher.go","lineNumber":431,"sourceCode":"\tb, err := NewBaseFetcherWithMetrics(logger, concurrency, bkt, blockIDsFetcher, dir, baseFetcherMetrics)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn b.NewMetaFetcherWithMetrics(fetcherMetrics, filters), nil\n}\n\n// NewMetaFetcher transforms BaseFetcher into actually usable *MetaFetcher.\nfunc (f *BaseFetcher) NewMetaFetcher(reg prometheus.Registerer, filters []MetadataFilter, logTags ...any) *MetaFetcher {\n\treturn f.NewMetaFetcherWithMetrics(NewFetcherMetrics(reg, nil, nil), filters, logTags...)\n}\n\n// NewMetaFetcherWithMetrics transforms BaseFetcher into actually usable *MetaFetcher.\nfunc (f *BaseFetcher) NewMetaFetcherWithMetrics(fetcherMetrics *FetcherMetrics, filters []MetadataFilter, logTags ...any) *MetaFetcher {\n\treturn &MetaFetcher{metrics: fetcherMetrics, wrapped: f, filters: filters, logger: log.With(f.logger, logTags...)}\n}\n\nvar (\n\tErrorSyncMetaNotFound  = errors.New(\"meta.json not found\")\n\tErrorSyncMetaCorrupted = errors.New(\"meta.json corrupted\")\n)\n\nfunc (f *BaseFetcher) metaUpdated(id ulid.ULID, modified time.Time) bool {\n\tif f.modifiedTimestamps[id].IsZero() {\n\t\treturn false\n\t}\n\treturn !f.modifiedTimestamps[id].Equal(modified)\n}\n\nfunc (f *BaseFetcher) bustCacheForID(id ulid.ULID) {\n\tf.cacheBusts.Inc()\n\n\tf.cached.Delete(id)\n\tif err := os.RemoveAll(filepath.Join(f.cacheDir, id.String())); err != nil {\n\t\tlevel.Warn(f.logger).Log(\"msg\", \"failed to remove cached meta.json dir\", \"dir\", filepath.Join(f.cacheDir, id.String()), \"err\", err)\n\t}\n}","sourceCodeStart":413,"sourceCodeEnd":449,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/fetcher.go#L413-L449","documentation":"ErrorSyncMetaNotFound is returned by BaseFetcher.loadMeta when meta.json for a block disappears from object storage between the bucket Exists check and the read. The loadMeta code wraps the underlying bucket not-found error with this sentinel so callers can match it with errors.Is/As. It signals a benign race: the block was deleted concurrently (typically by a compactor/GC) while the syncer was fetching its metadata.","triggerScenarios":"loadMeta: f.bkt.Exists returns true for a block's meta.json, then ReaderWithExpectedErrs(...).Get(ctx, metaFile) fails with an object-not-found error that satisfies f.bkt.IsObjNotFoundErr; the fetcher wraps it in ErrorSyncMetaNotFound. In fetchBlocks this causes the block to be counted in resp.noMetas and skipped.","commonSituations":"Compactor/GC deletes blocks concurrently with a Store/Querier syncing blocks; two Thanos instances racing where one removes blocks (deletion delay too short); manual block deletion or lifecycle rules in S3/GCS removing objects mid-sync; stale bucket view from eventually-consistent object stores.","solutions":["Treat it as benign: noMetas is counted and the block is skipped; retry the sync later — the block is gone from the bucket view.","Increase block deletion delay (--store.grpc.series-sample-limit irrelevant; use --objstore.* / compactor deletion-delay) so concurrent syncers finish reading before GC deletes blocks.","Check whether another process (compactor, manual cleanup, S3 lifecycle policy) is deleting blocks; pause or reconfigure it.","If this occurs persistently for blocks that still exist, verify bucket credentials/permissions and that IsObjNotFoundErr is configured correctly for your object store."],"exampleFix":"// before (reactive: block vanishes mid-sync)\n// compactor deletion-delay too small\n\n// after: give concurrent readers time\n// thanos compact --delete-delay=48h\n// (ensures blocks removed by compaction stay readable for in-flight syncs)","handlingStrategy":"try-catch","validationCode":"exists, err := bkt.Exists(ctx, metaPath)\nif err == nil && !exists {\n    // skip block before calling the fetcher; it's already gone\n}","typeGuard":"func isMetaNotFound(err error) bool {\n    return errors.Is(err, block.ErrorSyncMetaNotFound)\n}","tryCatchPattern":"m, err := metaFetcher.Fetch(ctx, metas)\nswitch {\ncase err == nil:\n    // use metas\ncase errors.Is(err, block.ErrorSyncMetaNotFound):\n    // benign: block deleted concurrently; log at debug, retry next sync\ncase errors.As(err, &e): // inspect wrapped cause\n    // unexpected: handle genuine storage error\n}","preventionTips":["Keep compactor deletion-delay larger than the longest expected sync duration","Avoid manual/lifecycle-based deletion while syncers are running","Monitor resp.noMetas rate; occasional occurrences are expected, spikes indicate aggressive GC","Use bucket indexes or consistent object stores to reduce stale Exists results"],"tags":["object-storage","race-condition","thanos","block-sync","retryable"],"backgroundTag":"resource-not-found","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}