thanos-io/thanos · error

overlaps found while gathering blocks.

Error message

overlaps found while gathering blocks. %s

What it means

After gathering block metadata from the bucket, the compactor sorts metas by MinTime and runs tsdb.OverlappingBlocks; any overlap between block time ranges is rejected because Prometheus TSDB compaction assumes non-overlapping blocks. Overlapping blocks make merges ambiguous and can produce duplicated samples, so the compactor refuses to proceed until they are resolved.

Solutions

1. Check for multiple compactor instances pointed at the same bucket and ensure only one runs (Thanos compactor is not HA)
2. Run `thanos tools bucket verify` to detect and repair overlaps automatically
3. If the error is tsdb issue 347, use the built-in RepairIssue347 path (or ensure the compactor's --repair mode handles it)
4. Identify the overlapping block IDs from the error text and mark the wrong one for deletion with a deletion mark via `thanos tools bucket mark`
5. Fix the ingestion topology (sidecar/receive) so the same time range is not uploaded twice

Example fix

// before: two compactors, same bucket
thanos compact --data-dir=/data-a --objstore=s3... &
thanos compact --data-dir=/data-b --objstore=s3... &
// after: single compactor (leader-election / one replica)
thanos compact --data-dir=/data --objstore=s3...

metas, err := GatherBlockMetas(ctx, logger, bkt, mint, maxt, resolution)
if overlaps := tsdb.OverlappingBlocks(metas); len(overlaps) > 0 {
    // resolve before compaction: verify / delete / repair
    return fmt.Errorf("bucket has overlapping blocks: %s", overlaps)
}

// Go
if _, err := compactBucket(...); err != nil {
    var ie compact.Issue347Error
    if errors.As(err, &ie) {
        err = compact.RepairIssue347(ctx, logger, bkt, deletionCtr, err)
    } else {
        logger.Error("non-347 overlap; ensure single compactor and verify bucket", "err", err)
    }
}

When it happens

Trigger: tsdb.OverlappingBlocks(metas) returns a non-empty list after GatherBlockMetas, i.e. two or more blocks in the bucket share overlapping [MinTime, MaxTime) ranges within the same resolution/compaction level.

Common situations: Historic data written by Prometheus/tsdb issue #347 (index corruption creating duplicated blocks); two compactor instances running concurrently against the same bucket; manually copied/uploaded blocks with wrong time ranges; blocks restored from backup in wrong order; receive+sidecar both uploading the same time range.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• shipping compacted block
• level is bigger then default set of
• get compaction levels
• create compactor
• create bucket compactor

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4ba304a480a07fd6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1104

		excludeMap[meta.ULID] = struct{}{}
	}

	for _, m := range cg.metasByMinTime {
		if _, ok := excludeMap[m.ULID]; ok {
			continue
		}
		metas = append(metas, m.BlockMeta)
	}

	if include != nil {
		metas = append(metas, include.BlockMeta)
	}

	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime
	})
	if overlaps := tsdb.OverlappingBlocks(metas); len(overlaps) > 0 {
		return errors.Errorf("overlaps found while gathering blocks. %s", overlaps)
	}
	return nil
}

// RepairIssue347 repairs the https://github.com/prometheus/tsdb/issues/347 issue when having issue347Error.
func RepairIssue347(ctx context.Context, logger log.Logger, bkt objstore.Bucket, blocksMarkedForDeletion prometheus.Counter, issue347Err error) error {
	ie, ok := errors.Cause(issue347Err).(Issue347Error)
	if !ok {
		return errors.Errorf("Given error is not an issue347 error: %v", issue347Err)
	}

	level.Info(logger).Log("msg", "Repairing block broken by https://github.com/prometheus/tsdb/issues/347", "id", ie.id, "err", issue347Err)

	tmpdir, err := os.MkdirTemp("", fmt.Sprintf("repair-issue-347-id-%s-", ie.id))
	if err != nil {
		return err
	}

View on GitHub (pinned to 35b8b99117)