{"record":{"id":"c444211e11a74fc1","repo":"thanos-io/thanos","slug":"failed-to-run-pre-compaction-callback-for-plan-s","errorCode":null,"errorMessage":"failed to run pre compaction callback for plan: %s","messagePattern":"failed to run pre compaction callback for plan: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1202,"sourceCode":"\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_planning\", func(ctx context.Context) (e error) {\n\t\ttoCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.Extensions())\n\t\treturn e\n\t}); err != nil {\n\t\treturn false, nil, errors.Wrap(err, \"plan compaction\")\n\t}\n\tif len(toCompact) == 0 {\n\t\t// Nothing to do.\n\t\treturn false, nil, nil\n\t}\n\n\tlevel.Info(cg.logger).Log(\"msg\", \"compaction available and planned\", \"plan\", fmt.Sprintf(\"%v\", toCompact))\n\n\t// Once we have a plan we need to download the actual data.\n\tgroupCompactionBegin := time.Now()\n\tbegin := groupCompactionBegin\n\n\tif err := compactionLifecycleCallback.PreCompactionCallback(ctx, cg.logger, cg, toCompact); err != nil {\n\t\treturn false, nil, errors.Wrapf(err, \"failed to run pre compaction callback for plan: %s\", fmt.Sprintf(\"%v\", toCompact))\n\t}\n\tlevel.Info(cg.logger).Log(\"msg\", \"finished running pre compaction callback; downloading blocks\", \"duration\", time.Since(begin), \"duration_ms\", time.Since(begin).Milliseconds(), \"plan\", fmt.Sprintf(\"%v\", toCompact))\n\n\tbegin = time.Now()\n\tg, errCtx := errgroup.WithContext(ctx)\n\tg.SetLimit(cg.compactBlocksFetchConcurrency)\n\n\ttoCompactDirs := make([]string, 0, len(toCompact))\n\tfor _, m := range toCompact {\n\t\tbdir := filepath.Join(dir, m.ULID.String())\n\t\tfunc(ctx context.Context, meta *metadata.Meta) {\n\t\t\tg.Go(func() error {\n\t\t\t\tstart := time.Now()\n\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_download\", func(ctx context.Context) error {\n\t\t\t\t\treturn block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir, objstore.WithFetchConcurrency(cg.blockFilesConcurrency))\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn retry(errors.Wrapf(err, \"download block %s\", meta.ULID))\n\t\t\t\t}","sourceCodeStart":1184,"sourceCodeEnd":1220,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1184-L1220","documentation":"Thanos invokes the registered PreCompactionCallback (used e.g. by the store-gateway/sharding layer to react to a compaction plan) after a plan is computed but before blocks are downloaded. If the callback returns an error, the whole compaction pass is aborted with this message that embeds the plan (block ULIDs).","triggerScenarios":"compactionLifecycleCallback.PreCompactionCallback returns error for the given toCompact plan — e.g. the callback implementation (partitioner/sharding callback) fails validating or preparing the block set.","commonSituations":"Running compactor with sharding where a callback fails on blocks assigned to another shard; custom lifecycle callbacks (compact.CompactionLifecycleCallback implementations) throwing on unexpected block counts; transient backend issues inside the callback.","solutions":["Read the wrapped error and the printed plan (block ULIDs) to see which blocks the callback rejected","If using sharding, verify the sharding rings/config are consistent and blocks belong to this compactor instance","Fix or remove the custom PreCompactionCallback implementation; retry after transient failures","Re-run compaction — the callback runs again on the next loop with a fresh plan"],"exampleFix":"// before\ncallback := myCustomCallback{} // returns error on plans > N blocks\n// after\nfunc (c myCustomCallback) PreCompactionCallback(ctx context.Context, l log.Logger, g *compact.Group, plans []*metadata.Meta) error {\n    if len(plans) > maxBlocks { return nil } // skip instead of erroring\n    return c.do(ctx, plans)\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := compactor.Compact(ctx); err != nil {\n    var planErr error\n    if strings.Contains(err.Error(), \"failed to run pre compaction callback\") {\n        planErr = err // log plan ULIDs embedded in message, alert operator\n    }\n    log.Error(err, \"pre-compaction callback failed\")\n}","preventionTips":["Keep custom CompactionLifecycleCallback implementations simple and side-effect free","Ensure sharding config is consistent so callbacks only see blocks they own","Add idempotent, retriable logic inside PreCompactionCallback"],"tags":["compaction","callback","lifecycle","thanos"],"backgroundTag":"unexpected-response-shape","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}