thanos-io/thanos · error

failed to run pre compaction callback for plan

Error message

failed to run pre compaction callback for plan: %s

What it means

Thanos invokes the registered PreCompactionCallback (used e.g. by the store-gateway/sharding layer to react to a compaction plan) after a plan is computed but before blocks are downloaded. If the callback returns an error, the whole compaction pass is aborted with this message that embeds the plan (block ULIDs).

Solutions

1. Read the wrapped error and the printed plan (block ULIDs) to see which blocks the callback rejected
2. If using sharding, verify the sharding rings/config are consistent and blocks belong to this compactor instance
3. Fix or remove the custom PreCompactionCallback implementation; retry after transient failures
4. Re-run compaction — the callback runs again on the next loop with a fresh plan

Example fix

// before
callback := myCustomCallback{} // returns error on plans > N blocks
// after
func (c myCustomCallback) PreCompactionCallback(ctx context.Context, l log.Logger, g *compact.Group, plans []*metadata.Meta) error {
    if len(plans) > maxBlocks { return nil } // skip instead of erroring
    return c.do(ctx, plans)
}

if err := compactor.Compact(ctx); err != nil {
    var planErr error
    if strings.Contains(err.Error(), "failed to run pre compaction callback") {
        planErr = err // log plan ULIDs embedded in message, alert operator
    }
    log.Error(err, "pre-compaction callback failed")
}

When it happens

Trigger: compactionLifecycleCallback.PreCompactionCallback returns error for the given toCompact plan — e.g. the callback implementation (partitioner/sharding callback) fails validating or preparing the block set.

Common situations: Running compactor with sharding where a callback fails on blocks assigned to another shard; custom lifecycle callbacks (compact.CompactionLifecycleCallback implementations) throwing on unexpected block counts; transient backend issues inside the callback.

Related errors

• create bucket compactor
• compaction
• critical error detected
• error executing compaction
• could not group metadata for compaction

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c444211e11a74fc1. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1202

	if err := tracing.DoInSpanWithErr(ctx, "compaction_planning", func(ctx context.Context) (e error) {
		toCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.Extensions())
		return e
	}); err != nil {
		return false, nil, errors.Wrap(err, "plan compaction")
	}
	if len(toCompact) == 0 {
		// Nothing to do.
		return false, nil, nil
	}

	level.Info(cg.logger).Log("msg", "compaction available and planned", "plan", fmt.Sprintf("%v", toCompact))

	// Once we have a plan we need to download the actual data.
	groupCompactionBegin := time.Now()
	begin := groupCompactionBegin

	if err := compactionLifecycleCallback.PreCompactionCallback(ctx, cg.logger, cg, toCompact); err != nil {
		return false, nil, errors.Wrapf(err, "failed to run pre compaction callback for plan: %s", fmt.Sprintf("%v", toCompact))
	}
	level.Info(cg.logger).Log("msg", "finished running pre compaction callback; downloading blocks", "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds(), "plan", fmt.Sprintf("%v", toCompact))

	begin = time.Now()
	g, errCtx := errgroup.WithContext(ctx)
	g.SetLimit(cg.compactBlocksFetchConcurrency)

	toCompactDirs := make([]string, 0, len(toCompact))
	for _, m := range toCompact {
		bdir := filepath.Join(dir, m.ULID.String())
		func(ctx context.Context, meta *metadata.Meta) {
			g.Go(func() error {
				start := time.Now()
				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_download", func(ctx context.Context) error {
					return block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir, objstore.WithFetchConcurrency(cg.blockFilesConcurrency))
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return retry(errors.Wrapf(err, "download block %s", meta.ULID))
				}

View on GitHub (pinned to 35b8b99117)