thanos-io/thanos · error

filter blocks marked for deletion

Error message

filter blocks marked for deletion

What it means

This is a wrapped error from Thanos' block fetcher (FetchDeletionMarks). While fetching deletion marks from each storage provider concurrently via an errgroup, one of the workers failed, and the fetcher wraps the cause with 'filter blocks marked for deletion' via errors.Wrap. The message points at the filtering phase where blocks marked for deletion are collected; the real cause is in the wrapped chain.

Solutions

1. Inspect the wrapped cause below this message in the error chain (errors.Unwrap / %+v) — fix that root error first (storage credentials, connectivity, corrupt deletion-mark.json).
2. Verify object storage connectivity and credentials: run 'thanos tools bucket verify' against the bucket.
3. If deletion-mark.json is corrupt, repair or remove it from the bucket (it lives under the deleted blocks / metadata prefix).
4. Retry the operation; the fetch is typically retriable after a transient storage failure.

Example fix

// before: opaque log
logger.Error(err, "failed to sync blocks")
// after: print full cause chain
logger.Error(fmt.Sprintf("%+v", err), "failed to sync blocks") // reveals root storage error under "filter blocks marked for deletion"

// Go: check bucket reachability before syncing deletion marks
if err := b.Iter(ctx, "", func(string) error { return nil }); err != nil {
    return fmt.Errorf("object storage unreachable: %w", err)
}

var derr *errors.Error
if errors.As(err, &derr) && strings.Contains(derr.Error(), "filter blocks marked for deletion") {
    // inspect derr.Unwrap() for the root storage error
}

if err := fetcher.Fetch(ctx, metas); err != nil {
    var root error
    for errors.Unwrap(err) != nil { root = errors.Unwrap(err) }
    log.Errorf("deletion-mark sync failed, root cause: %v", root)
    // retry with backoff for transient storage errors
}

When it happens

Trigger: Calling the block fetcher's deletion-mark sync (e.g. Fetcher.Fetch / FetchDeletionMarks used by tools like thanos compact or bucket verify) when any per-provider request to load metadata.DeletionMark objects fails — e.g. object storage list/get failure, malformed deletion-mark.json, or network error inside the errgroup.

Common situations: Object storage (S3/GCS/Azure) outages or misconfigured credentials during 'thanos compact' or 'thanos bucket' runs; a corrupted or partially written deletion-mark.json in the bucket; transient network timeouts while listing metadata.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• %v
• get meta file
• create bucket compactor
• compaction
• sync before first pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d5e720c24a7e76d7. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:1371

	// Workers scheduled, distribute blocks.
	eg.Go(func() error {
		defer close(ch)

		for _, id := range blockIDs {
			select {
			case ch <- id:
				// Nothing to do.
			case <-ctx.Done():
				return ctx.Err()
			}
		}

		return nil
	})

	if err := eg.Wait(); err != nil {
		return errors.Wrap(err, "filter blocks marked for deletion")
	}

	f.mtx.Lock()
	if f.deletionMarkMap == nil {
		f.deletionMarkMap = make(map[ulid.ULID]*metadata.DeletionMark)
	}
	maps.Copy(f.deletionMarkMap, deletionMarkMap)

	for u := range f.deletionMarkMap {
		if _, exists := preFilterMetas[u]; exists {
			continue
		}

		delete(f.deletionMarkMap, u)
	}

	f.mtx.Unlock()

View on GitHub (pinned to 35b8b99117)