thanos-io/thanos · critical · halt

pre compaction overlap check

Error message

pre compaction overlap check

What it means

During compaction planning, the compaction group runs areBlocksOverlapping to detect blocks with overlapping time ranges. Overlapping blocks in a Prometheus HA setup mean duplicate data that would corrupt compaction output. By default (vertical compaction disabled) Thanos halts compaction entirely when overlaps are found, because it cannot safely pick which replica's data to keep.

Solutions

1. Enable vertical compaction (--compact.enable-vertical-compaction) so overlapping blocks are deduplicated instead of halting, and run bucket deduplication
2. Fix bucket layout: ensure each Prometheus replica writes blocks with unique external labels (prometheus label) or use separate bucket prefixes
3. Run 'thanos tools bucket verify' or the deduplication tooling to resolve overlapping blocks already in the bucket
4. Remove duplicate/leftover blocks manually after confirming which copy is authoritative

Example fix

// before
// thanos compact --data-dir /data --objstore.bucket prom --compact.enable-vertical-compaction=false
// after
// thanos compact --data-dir /data --objstore.bucket prom --compact.enable-vertical-compaction
// plus unique prometheus external labels per replica so dedup works

// Before enabling compaction on a bucket, check for overlaps:
thanos tools bucket inspect --objstore.bucket prom | grep -i overlap
// or programmatically: iterate metas and assert sorted-by-MinTime blocks have next.MinTime >= cur.MaxTime

When it happens

Trigger: cg.areBlocksOverlapping(nil) returns an error (at least two blocks in the group share overlapping minTime/maxTime ranges) while cg.enableVerticalCompaction is false.

Common situations: Running two Prometheus replicas writing to the same store bucket without distinct external labels or a sidecar dedup setup; mistakenly pointing multiple sidecars at the same bucket prefix; leftover blocks from a misconfigured ruler/receive; enabling vertical compaction was the intended fix but the flag is off.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• compaction
• error executing compaction
• create bucket compactor
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/91bbdecbfc9ceab0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1177

	// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).
	if err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, "source of repaired block", blocksMarkedForDeletion); err != nil {
		return errors.Wrapf(err, "marking old block %s for deletion has failed", ie.id)
	}
	return nil
}

func (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp Compactor, blockDeletableChecker BlockDeletableChecker, compactionLifecycleCallback CompactionLifecycleCallback, errChan chan error) (bool, []ulid.ULID, error) {
	cg.mtx.Lock()
	defer cg.mtx.Unlock()

	// Check for overlapped blocks.
	overlappingBlocks := false
	if err := cg.areBlocksOverlapping(nil); err != nil {
		// TODO(bwplotka): It would really nice if we could still check for other overlaps than replica. In fact this should be checked
		// in syncer itself. Otherwise with vertical compaction enabled we will sacrifice this important check.
		if !cg.enableVerticalCompaction {
			return false, nil, halt(errors.Wrap(err, "pre compaction overlap check"))
		}

		overlappingBlocks = true
	}

	var toCompact []*metadata.Meta
	if err := tracing.DoInSpanWithErr(ctx, "compaction_planning", func(ctx context.Context) (e error) {
		toCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.Extensions())
		return e
	}); err != nil {
		return false, nil, errors.Wrap(err, "plan compaction")
	}
	if len(toCompact) == 0 {
		// Nothing to do.
		return false, nil, nil
	}

	level.Info(cg.logger).Log("msg", "compaction available and planned", "plan", fmt.Sprintf("%v", toCompact))

View on GitHub (pinned to 35b8b99117)