{"record":{"id":"0bacaead325d828e","repo":"thanos-io/thanos","slug":"could-not-group-metadata-for-compaction","errorCode":null,"errorMessage":"could not group metadata for compaction","messagePattern":"could not group metadata for compaction","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/compact.go","lineNumber":666,"sourceCode":"\t\t\t\treturn runutil.Repeat(conf.progressCalculateInterval, ctx.Done(), func() error {\n\n\t\t\t\t\tif err := sy.SyncMetas(ctx); err != nil {\n\t\t\t\t\t\t// The RetryError signals that we hit an retriable error (transient error, no connection).\n\t\t\t\t\t\t// You should alert on this being triggered too frequently.\n\t\t\t\t\t\tif compact.IsRetryError(err) {\n\t\t\t\t\t\t\tlevel.Error(logger).Log(\"msg\", \"retriable error\", \"err\", err)\n\t\t\t\t\t\t\tcompactMetrics.retried.Inc()\n\n\t\t\t\t\t\t\treturn nil\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not sync metas\")\n\t\t\t\t\t}\n\n\t\t\t\t\tmetas := sy.Metas()\n\t\t\t\t\tgroups, err := grouper.Groups(metas)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not group metadata for compaction\")\n\t\t\t\t\t}\n\n\t\t\t\t\tif err = ps.ProgressCalculate(ctx, groups); err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not calculate compaction progress\")\n\t\t\t\t\t}\n\n\t\t\t\t\tretGroups, err := grouper.Groups(metas)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not group metadata for retention\")\n\t\t\t\t\t}\n\n\t\t\t\t\tif err = rs.ProgressCalculate(ctx, retGroups); err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not calculate retention progress\")\n\t\t\t\t\t}\n\n\t\t\t\t\tif !conf.disableDownsampling {\n\t\t\t\t\t\tgroups, err = grouper.Groups(metas)\n\t\t\t\t\t\tif err != nil {","sourceCodeStart":648,"sourceCodeEnd":684,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/compact.go#L648-L684","documentation":"After metas are synced, grouper.Groups(metas) groups block metadata by their compaction group (resolution + external labels). If grouping fails, the progress worker returns 'could not group metadata for compaction'. The default BaseGrouper errors when blocks in the same group would be incompatible — typically when duplicate blocks with identical labels and resolution but conflicting compaction ranges exist.","triggerScenarios":"grouper.Groups(metas) returns ErrGroupPermission (duplicate compaction group) or ErrGroupNotFound; in practice: two blocks with identical Thanos external labels and resolution whose time ranges overlap or duplicate.","commonSituations":"Two compactors running concurrently on the same bucket produced duplicated/overlapping blocks; backfilled historical blocks were pushed with the same labels; Prometheus labels changed mid-stream so two blocks claim the same group.","solutions":["Run 'thanos tools bucket verify' to identify duplicate/overlapping blocks in the affected group.","Mark the redundant blocks for deletion (deletion-mark.json) or repair via 'thanos tools bucket repair --mark-logical-deleted'.","Ensure exactly one compactor runs per bucket to prevent future duplicates.","Fix Prometheus external_labels so each source produces a unique label set."],"exampleFix":"// before\n// duplicate blocks with same labels cause Groups() to fail\n// after\nthanos tools bucket repair --objstore.bucket=thanos --mark-logical-deleted\nthanos compact --data-dir=/var/thanos/compact --objstore.bucket=thanos","handlingStrategy":"validation","validationCode":"// Detect duplicate groups client-side before compaction\nseen := map[string]struct{}{}\nfor _, m := range metas {\n    key := m.Thanos.Labels.String() + fmt.Sprintf(\"/%d\", m.Thanos.Resolution)\n    if _, dup := seen[key]; dup {\n        log.Printf(\"duplicate compaction group: %s\", key)\n    }\n    seen[key] = struct{}{}\n}","typeGuard":null,"tryCatchPattern":"groups, err := grouper.Groups(metas)\nif err != nil {\n    if errors.Is(err, base.ErrGroupPermission) || errors.Is(err, base.ErrGroupNotFound) {\n        // duplicate/overlapping blocks: run bucket verify + repair, then alert\n        alertDuplicateBlocks(err)\n    }\n    return errors.Wrapf(err, \"could not group metadata for compaction\")\n}","preventionTips":["Run exactly one compactor per bucket.","Backfill historical blocks with unique external labels and non-overlapping time ranges.","Keep Prometheus external_labels stable — changing them mid-stream creates duplicate groups.","Schedule 'thanos tools bucket verify' regularly."],"tags":["thanos","compaction","grouping","duplicate-blocks"],"backgroundTag":"duplicate-blocks-detected","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}