thanos-io/thanos · error

could not group metadata for compaction

Error message

could not group metadata for compaction

What it means

After metas are synced, grouper.Groups(metas) groups block metadata by their compaction group (resolution + external labels). If grouping fails, the progress worker returns 'could not group metadata for compaction'. The default BaseGrouper errors when blocks in the same group would be incompatible — typically when duplicate blocks with identical labels and resolution but conflicting compaction ranges exist.

Solutions

1. Run 'thanos tools bucket verify' to identify duplicate/overlapping blocks in the affected group.
2. Mark the redundant blocks for deletion (deletion-mark.json) or repair via 'thanos tools bucket repair --mark-logical-deleted'.
3. Ensure exactly one compactor runs per bucket to prevent future duplicates.
4. Fix Prometheus external_labels so each source produces a unique label set.

Example fix

// before
// duplicate blocks with same labels cause Groups() to fail
// after
thanos tools bucket repair --objstore.bucket=thanos --mark-logical-deleted
thanos compact --data-dir=/var/thanos/compact --objstore.bucket=thanos

// Detect duplicate groups client-side before compaction
seen := map[string]struct{}{}
for _, m := range metas {
    key := m.Thanos.Labels.String() + fmt.Sprintf("/%d", m.Thanos.Resolution)
    if _, dup := seen[key]; dup {
        log.Printf("duplicate compaction group: %s", key)
    }
    seen[key] = struct{}{}
}

groups, err := grouper.Groups(metas)
if err != nil {
    if errors.Is(err, base.ErrGroupPermission) || errors.Is(err, base.ErrGroupNotFound) {
        // duplicate/overlapping blocks: run bucket verify + repair, then alert
        alertDuplicateBlocks(err)
    }
    return errors.Wrapf(err, "could not group metadata for compaction")
}

When it happens

Trigger: grouper.Groups(metas) returns ErrGroupPermission (duplicate compaction group) or ErrGroupNotFound; in practice: two blocks with identical Thanos external labels and resolution whose time ranges overlap or duplicate.

Common situations: Two compactors running concurrently on the same bucket produced duplicated/overlapping blocks; backfilled historical blocks were pushed with the same labels; Prometheus labels changed mid-stream so two blocks claim the same group.

Related errors

• create bucket compactor
• compaction
• critical error detected
• error executing compaction
• could not calculate compaction progress

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/0bacaead325d828e. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:666

				return runutil.Repeat(conf.progressCalculateInterval, ctx.Done(), func() error {

					if err := sy.SyncMetas(ctx); err != nil {
						// The RetryError signals that we hit an retriable error (transient error, no connection).
						// You should alert on this being triggered too frequently.
						if compact.IsRetryError(err) {
							level.Error(logger).Log("msg", "retriable error", "err", err)
							compactMetrics.retried.Inc()

							return nil
						}

						return errors.Wrapf(err, "could not sync metas")
					}

					metas := sy.Metas()
					groups, err := grouper.Groups(metas)
					if err != nil {
						return errors.Wrapf(err, "could not group metadata for compaction")
					}

					if err = ps.ProgressCalculate(ctx, groups); err != nil {
						return errors.Wrapf(err, "could not calculate compaction progress")
					}

					retGroups, err := grouper.Groups(metas)
					if err != nil {
						return errors.Wrapf(err, "could not group metadata for retention")
					}

					if err = rs.ProgressCalculate(ctx, retGroups); err != nil {
						return errors.Wrapf(err, "could not calculate retention progress")
					}

					if !conf.disableDownsampling {
						groups, err = grouper.Groups(metas)
						if err != nil {

View on GitHub (pinned to 35b8b99117)