thanos-io/thanos · error

error executing compaction

Error message

error executing compaction

What it means

When the compaction iteration returns an error that is neither a HaltError nor a RetryError, the compactor wraps it as 'error executing compaction' and terminates. It is the catch-all fatal path for errors that cannot be retried nor classified as halting bugs (often context cancellation or unexpected I/O errors from the bucket store).

Solutions

1. Read the wrapped inner error to see the real cause (credentials, 404, timeout, etc.) and fix the object store configuration accordingly.
2. Verify bucket credentials and permissions (s3 access keys, GCS service account) using 'thanos tools bucket ls'.
3. Check whether the process is being cancelled (SIGTERM, k8s liveness kill) mid-compaction; give compaction enough grace period.
4. If the error is genuinely transient but unclassified, upgrade Thanos — error classification (retry vs halt) improves between releases.

Example fix

// before
return errors.Wrap(err, "error executing compaction") // context canceled
// after
# ensure parent ctx is long-lived; check runGroup signal handling
ctx, cancel = context.WithCancel(context.Background()) // wire to term signals via g.Add(...)

// Pre-flight: check object store reachable and credentials valid before starting compaction
_, err := bkt.Iter(ctx, "", func(string) error { return nil })
if err != nil {
    return fmt.Errorf("bucket preflight failed: %w", err)
}

if err := runCompaction(ctx); err != nil {
    switch {
    case compact.IsRetryError(err):
        scheduleRetry() // transient
    case compact.IsHaltError(err):
        alertAndHalt(err)
    default:
        log.Fatalf("error executing compaction: %v", err) // fatal, inspect inner cause
    }
}

When it happens

Trigger: Inside the runutil.Repeat compaction loop, compact.Compact returns an error that fails both compact.IsHaltError and compact.IsRetryError — e.g. a non-retryable bucketGetObject error, an object-store access/permission error not classified as retryable, or ctx cancellation mid-compaction.

Common situations: Misconfigured object storage credentials (403 not classified as retry), deleting the data-dir mid-run, network policies causing non-transient storage API errors, or a stale context being cancelled by an sibling service shutting down.

Related errors

• compaction
• pre compaction overlap check
• create bucket compactor
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4e587a498b9009e8. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:565

				if conf.haltOnError {
					level.Error(logger).Log("msg", "critical error detected; halting", "err", err)
					compactMetrics.halted.Set(1)
					select {}
				} else {
					return errors.Wrap(err, "critical error detected")
				}
			}

			// The RetryError signals that we hit an retriable error (transient error, no connection).
			// You should alert on this being triggered too frequently.
			if compact.IsRetryError(err) {
				level.Error(logger).Log("msg", "retriable error", "err", err)
				compactMetrics.retried.Inc()
				// TODO(bplotka): use actual "retry()" here instead of waiting 5 minutes?
				return nil
			}

			return errors.Wrap(err, "error executing compaction")
		})
	}, func(error) {
		cancel()
	})

	if conf.wait {
		if !conf.disableWeb {
			r := route.New()

			ins := extpromhttp.NewInstrumentationMiddleware(reg, nil)

			global := ui.NewBucketUI(logger, conf.webConf.externalPrefix, conf.webConf.prefixHeaderName, component)
			global.Register(r, ins)

			// Configure Request Logging for HTTP calls.
			opts := []logging.Option{logging.WithDecider(func(_ string, _ error) logging.Decision {
				return logging.NoLogCall
			})}

View on GitHub (pinned to 35b8b99117)