{"record":{"id":"d5e720c24a7e76d7","repo":"thanos-io/thanos","slug":"filter-blocks-marked-for-deletion","errorCode":null,"errorMessage":"filter blocks marked for deletion","messagePattern":"filter blocks marked for deletion","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/fetcher.go","lineNumber":1371,"sourceCode":"\n\t// Workers scheduled, distribute blocks.\n\teg.Go(func() error {\n\t\tdefer close(ch)\n\n\t\tfor _, id := range blockIDs {\n\t\t\tselect {\n\t\t\tcase ch <- id:\n\t\t\t\t// Nothing to do.\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn ctx.Err()\n\t\t\t}\n\t\t}\n\n\t\treturn nil\n\t})\n\n\tif err := eg.Wait(); err != nil {\n\t\treturn errors.Wrap(err, \"filter blocks marked for deletion\")\n\t}\n\n\tf.mtx.Lock()\n\tif f.deletionMarkMap == nil {\n\t\tf.deletionMarkMap = make(map[ulid.ULID]*metadata.DeletionMark)\n\t}\n\tmaps.Copy(f.deletionMarkMap, deletionMarkMap)\n\n\tfor u := range f.deletionMarkMap {\n\t\tif _, exists := preFilterMetas[u]; exists {\n\t\t\tcontinue\n\t\t}\n\n\t\tdelete(f.deletionMarkMap, u)\n\t}\n\n\tf.mtx.Unlock()\n","sourceCodeStart":1353,"sourceCodeEnd":1389,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/fetcher.go#L1353-L1389","documentation":"This is a wrapped error from Thanos' block fetcher (FetchDeletionMarks). While fetching deletion marks from each storage provider concurrently via an errgroup, one of the workers failed, and the fetcher wraps the cause with 'filter blocks marked for deletion' via errors.Wrap. The message points at the filtering phase where blocks marked for deletion are collected; the real cause is in the wrapped chain.","triggerScenarios":"Calling the block fetcher's deletion-mark sync (e.g. Fetcher.Fetch / FetchDeletionMarks used by tools like thanos compact or bucket verify) when any per-provider request to load metadata.DeletionMark objects fails — e.g. object storage list/get failure, malformed deletion-mark.json, or network error inside the errgroup.","commonSituations":"Object storage (S3/GCS/Azure) outages or misconfigured credentials during 'thanos compact' or 'thanos bucket' runs; a corrupted or partially written deletion-mark.json in the bucket; transient network timeouts while listing metadata.","solutions":["Inspect the wrapped cause below this message in the error chain (errors.Unwrap / %+v) — fix that root error first (storage credentials, connectivity, corrupt deletion-mark.json).","Verify object storage connectivity and credentials: run 'thanos tools bucket verify' against the bucket.","If deletion-mark.json is corrupt, repair or remove it from the bucket (it lives under the deleted blocks / metadata prefix).","Retry the operation; the fetch is typically retriable after a transient storage failure."],"exampleFix":"// before: opaque log\nlogger.Error(err, \"failed to sync blocks\")\n// after: print full cause chain\nlogger.Error(fmt.Sprintf(\"%+v\", err), \"failed to sync blocks\") // reveals root storage error under \"filter blocks marked for deletion\"","handlingStrategy":"try-catch","validationCode":"// Go: check bucket reachability before syncing deletion marks\nif err := b.Iter(ctx, \"\", func(string) error { return nil }); err != nil {\n    return fmt.Errorf(\"object storage unreachable: %w\", err)\n}","typeGuard":"var derr *errors.Error\nif errors.As(err, &derr) && strings.Contains(derr.Error(), \"filter blocks marked for deletion\") {\n    // inspect derr.Unwrap() for the root storage error\n}","tryCatchPattern":"if err := fetcher.Fetch(ctx, metas); err != nil {\n    var root error\n    for errors.Unwrap(err) != nil { root = errors.Unwrap(err) }\n    log.Errorf(\"deletion-mark sync failed, root cause: %v\", root)\n    // retry with backoff for transient storage errors\n}","preventionTips":["Monitor object storage health and credentials before running compaction jobs.","Wrap fetcher calls with retriable-error detection and exponential backoff.","Log full error chains (%+v) so wrapped causes are visible.","Run 'thanos tools bucket verify' periodically to catch corrupt metadata early."],"tags":["object-storage","block-metadata","error-wrapping","thanos"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}