thanos-io/thanos · error

could not group metadata into downsample groups

Error message

could not group metadata into downsample groups

What it means

When downsampling is enabled, grouper.Groups(metas) is invoked once more to build downsample groups; failure yields 'could not group metadata into downsample groups'. Same root causes as the other grouping errors: duplicate or incompatible blocks within a group (ErrGroupPermission / ErrGroupNotFound).

Solutions

1. Run 'thanos tools bucket verify' to find duplicates; repair with 'thanos tools bucket repair --mark-logical-deleted'.
2. Ensure a single compactor per bucket.
3. Fix Prometheus external_labels to make groups unique.
4. If downsampling metrics are not needed, run with --disable-downsampling to skip this path (compaction still groups once).

Example fix

// before
thanos compact --objstore.bucket=thanos --data-dir=/var/thanos/compact
// after
thanos compact --objstore.bucket=thanos --data-dir=/var/thanos/compact --disable-downsampling
# (only as mitigation; fix duplicate blocks first)

seen := map[string]struct{}{}
for _, m := range metas {
    key := m.Thanos.Labels.String()
    if _, dup := seen[key]; dup {
        log.Printf("duplicate downsample group candidate: %s", key)
    }
    seen[key] = struct{}{}
}

groups, err := grouper.Groups(metas)
if err != nil {
    if errors.Is(err, base.ErrGroupPermission) {
        alertDuplicateBlocks(err)
    }
    return errors.Wrapf(err, "could not group metadata into downsample groups")
}

When it happens

Trigger: grouper.Groups(metas) fails in the !conf.disableDownsampling branch of the progress loop — duplicate blocks sharing labels/resolution, or invalid resolution values in meta.json.

Common situations: Duplicate blocks produced by concurrent compactors, or backfilled/uploaded historical blocks with clashing external labels.

Related errors

• raw resolution must be higher than the minimum block size…
• 5m resolution retention must be higher than the minimum…
• sync before first pass of downsampling
• first pass of downsampling failed
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e249f951be0e8a93. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:685

					}

					if err = ps.ProgressCalculate(ctx, groups); err != nil {
						return errors.Wrapf(err, "could not calculate compaction progress")
					}

					retGroups, err := grouper.Groups(metas)
					if err != nil {
						return errors.Wrapf(err, "could not group metadata for retention")
					}

					if err = rs.ProgressCalculate(ctx, retGroups); err != nil {
						return errors.Wrapf(err, "could not calculate retention progress")
					}

					if !conf.disableDownsampling {
						groups, err = grouper.Groups(metas)
						if err != nil {
							return errors.Wrapf(err, "could not group metadata into downsample groups")
						}
						if err := ds.ProgressCalculate(ctx, groups); err != nil {
							return errors.Wrapf(err, "could not calculate downsampling progress")
						}
					}

					return nil
				})
			}, func(err error) {
				cancel()
			})
		}
	}

	level.Info(logger).Log("msg", "starting compact node")
	statusProber.Ready()
	return nil
}

View on GitHub (pinned to 35b8b99117)