{"record":{"id":"91bbdecbfc9ceab0","repo":"thanos-io/thanos","slug":"pre-compaction-overlap-check","errorCode":null,"errorMessage":"pre compaction overlap check","messagePattern":"pre compaction overlap check","errorType":"exception","errorClass":"halt","httpStatus":null,"severity":"critical","filePath":"pkg/compact/compact.go","lineNumber":1177,"sourceCode":"\n\t// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).\n\tif err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, \"source of repaired block\", blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrapf(err, \"marking old block %s for deletion has failed\", ie.id)\n\t}\n\treturn nil\n}\n\nfunc (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp Compactor, blockDeletableChecker BlockDeletableChecker, compactionLifecycleCallback CompactionLifecycleCallback, errChan chan error) (bool, []ulid.ULID, error) {\n\tcg.mtx.Lock()\n\tdefer cg.mtx.Unlock()\n\n\t// Check for overlapped blocks.\n\toverlappingBlocks := false\n\tif err := cg.areBlocksOverlapping(nil); err != nil {\n\t\t// TODO(bwplotka): It would really nice if we could still check for other overlaps than replica. In fact this should be checked\n\t\t// in syncer itself. Otherwise with vertical compaction enabled we will sacrifice this important check.\n\t\tif !cg.enableVerticalCompaction {\n\t\t\treturn false, nil, halt(errors.Wrap(err, \"pre compaction overlap check\"))\n\t\t}\n\n\t\toverlappingBlocks = true\n\t}\n\n\tvar toCompact []*metadata.Meta\n\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_planning\", func(ctx context.Context) (e error) {\n\t\ttoCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.Extensions())\n\t\treturn e\n\t}); err != nil {\n\t\treturn false, nil, errors.Wrap(err, \"plan compaction\")\n\t}\n\tif len(toCompact) == 0 {\n\t\t// Nothing to do.\n\t\treturn false, nil, nil\n\t}\n\n\tlevel.Info(cg.logger).Log(\"msg\", \"compaction available and planned\", \"plan\", fmt.Sprintf(\"%v\", toCompact))","sourceCodeStart":1159,"sourceCodeEnd":1195,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1159-L1195","documentation":"During compaction planning, the compaction group runs areBlocksOverlapping to detect blocks with overlapping time ranges. Overlapping blocks in a Prometheus HA setup mean duplicate data that would corrupt compaction output. By default (vertical compaction disabled) Thanos halts compaction entirely when overlaps are found, because it cannot safely pick which replica's data to keep.","triggerScenarios":"cg.areBlocksOverlapping(nil) returns an error (at least two blocks in the group share overlapping minTime/maxTime ranges) while cg.enableVerticalCompaction is false.","commonSituations":"Running two Prometheus replicas writing to the same store bucket without distinct external labels or a sidecar dedup setup; mistakenly pointing multiple sidecars at the same bucket prefix; leftover blocks from a misconfigured ruler/receive; enabling vertical compaction was the intended fix but the flag is off.","solutions":["Enable vertical compaction (--compact.enable-vertical-compaction) so overlapping blocks are deduplicated instead of halting, and run bucket deduplication","Fix bucket layout: ensure each Prometheus replica writes blocks with unique external labels (prometheus label) or use separate bucket prefixes","Run 'thanos tools bucket verify' or the deduplication tooling to resolve overlapping blocks already in the bucket","Remove duplicate/leftover blocks manually after confirming which copy is authoritative"],"exampleFix":"// before\n// thanos compact --data-dir /data --objstore.bucket prom --compact.enable-vertical-compaction=false\n// after\n// thanos compact --data-dir /data --objstore.bucket prom --compact.enable-vertical-compaction\n// plus unique prometheus external labels per replica so dedup works","handlingStrategy":"validation","validationCode":"// Before enabling compaction on a bucket, check for overlaps:\nthanos tools bucket inspect --objstore.bucket prom | grep -i overlap\n// or programmatically: iterate metas and assert sorted-by-MinTime blocks have next.MinTime >= cur.MaxTime","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Give every Prometheus replica a unique prometheus external label","Never point two sidecars/receivers at the same bucket prefix for the same tenant","Enable vertical compaction + deduplication deliberately if overlaps are expected"],"tags":["compaction","object-storage","block-overlap","thanos"],"backgroundTag":"invalid-state-transition","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}