shipping compacted block

Error message

shipping compacted block %s is blocked; overlap spotted: %s

What it means

Returned by IsOverlapping when tsdb.OverlappingBlocks detects that the newly compacted block's time range overlaps blocks already known to the shipper. Thanos refuses to upload compacted blocks that overlap existing ones to preserve the global block universe invariant (non-overlapping blocks per resolution).

Solutions

Identify the overlapping block ULIDs from the Overlap message and delete the redundant duplicate block (locally and/or from the bucket).
Enable --shipper.allow-out-of-order-uploads on the sidecar if your setup intentionally allows overlapping uploads.
Re-compact locally so the overlapping blocks merge into a single non-overlapping block, then re-sync.
Check for duplicated data directories (e.g. two sidecars pointing at the same TSDB or a stale snapshot) and remove the stale one.

Example fix

// before
// sidecar: shipping compacted block 01ARZ... is blocked; overlap spotted: [01ARZ... and 01BXY...: 1600000-1610000 vs 1605000-1615000]
// after
// thanos sidecar --shipper.allow-out-of-order-uploads   # or delete the duplicate block
// via: thanos tools bucket rm --objstore.config-file=bucket.yaml --id=01BXY...

Defensive patterns

Strategy: validation

Validate before calling

metas := append([]tsdb.BlockMeta{newMeta}, existingMetas...)
if o := tsdb.OverlappingBlocks(metas); len(o) > 0 {
    return fmt.Errorf("pre-flight overlap check failed: %s", o.String())
}

Try / catch

err := checker.IsOverlapping(ctx, meta)
if err != nil {
    if strings.Contains(err.Error(), "overlap spotted") {
        // inspect o.String() ULIDs and resolve duplicates before re-syncing
        log.Printf("blocked compacted block %s: %v", meta.ULID, err)
    }
}

Prevention

Avoid overlapping retention/compaction settings in Prometheus
Only run one sidecar per TSDB data directory
Validate backfilled data ranges against existing bucket blocks before import
Use `thanos tools bucket inspect` to audit block time ranges

When it happens

Trigger: Sync with uploadCompacted enabled calls checker.IsOverlapping for a block with Compaction.Level > 1, and the new block's MinTime/MaxTime range overlaps any existing local block meta.

Common situations: Running Prometheus with overlapping retention or a restored/backfilled block that duplicates existing data; out-of-order compaction after a sidecar crash; re-creating blocks from snapshots while old blocks still exist; using --shipper.allow-out-of-order-uploads=false with backfilled data.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/181ac0a93c4a776c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:301

	return nil
}

func (c *lazyOverlapChecker) IsOverlapping(ctx context.Context, newMeta tsdb.BlockMeta) error {
	if !c.synced {
		level.Info(c.logger).Log("msg", "gathering all existing blocks from the remote bucket for check", "id", newMeta.ULID.String())
		if err := c.sync(ctx); err != nil {
			return err
		}
	}

	// TODO(bwplotka) so confusing! we need to sort it first. Add comment to TSDB code.
	metas := append([]tsdb.BlockMeta{newMeta}, c.metas...)
	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime
	})
	if o := tsdb.OverlappingBlocks(metas); len(o) > 0 {
		// TODO(bwplotka): Consider checking if overlaps relates to block in concern?
		return errors.Errorf("shipping compacted block %s is blocked; overlap spotted: %s", newMeta.ULID, o.String())
	}
	return nil
}

func (s *Shipper) AreAllBlocksUploaded() (bool, error) {
	s.mtx.RLock()
	defer s.mtx.RUnlock()

	metas, _, err := s.blockMetasFromOldest()
	if err != nil {
		return false, errors.Wrap(err, "get block metas from oldest")
	}

	if len(metas) == 0 {
		return true, nil
	}

	meta, err := ReadMetaFile(s.metadataFilePath)

View on GitHub (pinned to 35b8b99117)