thanos-io/thanos · error

compaction

Error message

compaction

What it means

Wrapper error from the compactMainFn closure in runCompact (cmd/thanos/compact.go:445) around compactor.Compact(ctx). It means a compaction iteration over bucket blocks failed; the real cause is nested inside (e.g. network errors to object storage, corrupt block index, meta.json issues).

Solutions

1. Read the wrapped inner error to find the failing block or bucket operation
2. Retry the command — compaction is resumable; transient object-store errors often resolve
3. Inspect the problematic block in the bucket (meta.json, index checksums) and remove/repair corrupt blocks if confirmed
4. Verify object store credentials/permissions and network connectivity
5. Consider --skip-block-with-out-of-order-chunks or fixing uploaders producing out-of-order chunks

// ensure bucket reachable before starting
if _, err := insBkt.Iter(ctx, "", func(string) error { return nil }); err != nil {
	return errors.Wrap(err, "bucket not reachable before compaction")
}

if err := compactor.Compact(ctx); err != nil {
	if errors.Is(err, context.Canceled) {
		return err // shutdown, do not retry
	}
	level.Error(logger).Log("msg", "compaction failed", "err", err)
	return errors.Wrap(err, "compaction") // outer loop/Halt will retry
}

When it happens

Trigger: Any error returned by compactor.Compact(ctx): object store unavailability, partial/corrupt blocks, out-of-order block conflicts, grouping/plan errors.

Common situations: S3/GCS transient failures or rate limits, blocks with checksum mismatches, blocks uploaded out of order (skipped or erroring depending on flags), permission errors on bucket.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• error executing compaction
• pre compaction overlap check
• create bucket compactor
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f5d35aac86ec0a92. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:445

	if retentionByResolution[compact.ResolutionLevel1h].Milliseconds() != 0 {
		level.Info(logger).Log("msg", "retention policy of 1 hour aggregated samples is enabled", "duration", retentionByResolution[compact.ResolutionLevel1h])
	}

	var cleanMtx sync.Mutex
	// TODO(GiedriusS): we could also apply retention policies here but the logic would be a bit more complex.
	cleanPartialMarked := func() error {
		cleanMtx.Lock()
		defer cleanMtx.Unlock()

		compact.BestEffortCleanAbortedPartialUploads(ctx, logger, sy.Partial(), insBkt, compactMetrics.partialUploadDeleteAttempts, compactMetrics.blocksCleaned, compactMetrics.blockCleanupFailures, ignoreDeletionMarkFilter.DeletionMarkBlocks())
		compactMetrics.cleanups.Inc()

		return nil
	}

	compactMainFn := func() error {
		if err := compactor.Compact(ctx); err != nil {
			return errors.Wrap(err, "compaction")
		}

		if !conf.disableDownsampling {
			// After all compactions are done, work down the downsampling backlog.
			// We run two passes of this to ensure that the 1h downsampling is generated
			// for 5m downsamplings created in the first run.
			level.Info(logger).Log("msg", "start first pass of downsampling")
			if err := sy.SyncMetas(ctx); err != nil {
				return errors.Wrap(err, "sync before first pass of downsampling")
			}

			filteredMetas := sy.Metas()
			noDownsampleBlocks := noDownsampleMarkerFilter.NoDownsampleMarkedBlocks()
			for ul := range noDownsampleBlocks {
				delete(filteredMetas, ul)
			}

			for _, meta := range filteredMetas {

View on GitHub (pinned to 35b8b99117)