thanos-io/thanos · error

failed to run post compaction callback for result block

Error message

failed to run post compaction callback for result block %s

What it means

Wraps compactionLifecycleCallback.PostCompactionCallback for an already-uploaded result block: the registered post-compaction hook (e.g. cache invalidation or downstream notification) failed after the block was committed to the bucket. The error is retryable — the callback may re-run for the same block.

Solutions

1. Inspect the callback's underlying error
2. Check bucket permissions for cleanup operations
3. Allow the compactor to retry

When it happens

Trigger: Thrown at pkg/compact/compact.go:1363 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/bb434479aaef4793. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1363

		// unless vertical compaction is enabled.
		if !cg.enableVerticalCompaction {
			if err := cg.areBlocksOverlapping(newMeta, toCompact...); err != nil {
				return false, nil, halt(errors.Wrapf(err, "resulted compacted block %s overlaps with something", bdir))
			}
		}

		begin = time.Now()

		err = tracing.DoInSpanWithErr(ctx, "compaction_block_upload", func(ctx context.Context) error {
			return block.Upload(ctx, cg.logger, cg.bkt, bdir, cg.hashFunc, objstore.WithUploadConcurrency(cg.blockFilesConcurrency))
		})
		if err != nil {
			return false, nil, retry(errors.Wrapf(err, "upload of %s failed", compID))
		}
		level.Info(cg.logger).Log("msg", "uploaded block", "result_block", compID, "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds())
		level.Info(cg.logger).Log("msg", "running post compaction callback", "result_block", compID)
		if err := compactionLifecycleCallback.PostCompactionCallback(ctx, cg.logger, cg, compID); err != nil {
			return false, nil, retry(errors.Wrapf(err, "failed to run post compaction callback for result block %s", compID))
		}
		level.Info(cg.logger).Log("msg", "finished running post compaction callback", "result_block", compID)
	}

	// Mark for deletion the blocks we just compacted from the group and bucket so they do not get included
	// into the next planning cycle.
	// Eventually the block we just uploaded should get synced into the group again (including sync-delay).
	for _, meta := range toCompact {
		if err := tracing.DoInSpanWithErr(ctx, "compaction_block_delete", func(ctx context.Context) error {
			return cg.deleteBlock(meta.ULID, filepath.Join(dir, meta.ULID.String()), blockDeletableChecker)
		}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
			return false, nil, retry(errors.Wrapf(err, "mark old block for deletion from bucket"))
		}
		cg.groupGarbageCollectedBlocks.Inc()
	}

	level.Info(cg.logger).Log("msg", "finished compacting blocks", "duration", time.Since(groupCompactionBegin),
		"duration_ms", time.Since(groupCompactionBegin).Milliseconds(), "result_blocks", compIDStrs, "source_blocks", sourceBlockStr)

View on GitHub (pinned to 35b8b99117)