argoproj/argo-workflows · error

failed to create emissary: %w

Error message

failed to create emissary: %w

What it means

When the running container is itself an emissary sidecar (container name appears in template.GetSidecarNames), the executor creates an emissary client via emissary.New() to wait on the main container(s) and terminate itself when they exit. If that client cannot be constructed (its underlying runtime probes/sockets are unavailable), the error is wrapped as "failed to create emissary" and the retryable start loop fails.

Source

Thrown at cmd/argoexec/commands/emissary.go:244

	}

	cmdErr := retry.OnError(backoff, func(error) bool { return true }, func() error {
		command, closer, err := startCommand(ctx, name, args, template, containerName, includeScriptOutput)
		if err != nil {
			return fmt.Errorf("failed to start command: %w", err)
		}
		defer closer()

		forwardSignals(ctx, signals, command.Process.Pid, false)
		pid := command.Process.Pid
		innerCtx, cancel := context.WithCancel(ctx)
		defer cancel()
		startFileSignalHandler(innerCtx, pid, containerName)
		for _, sidecarName := range template.GetSidecarNames() {
			if sidecarName == containerName {
				em, err := emissary.New()
				if err != nil {
					return fmt.Errorf("failed to create emissary: %w", err)
				}

				go func() {
					mainContainerNames := template.GetMainContainerNames()
					err = em.Wait(innerCtx, mainContainerNames)
					if err != nil {
						logger.WithError(err).WithFields(logging.Fields{
							"mainContainerNames": mainContainerNames,
						}).Error(innerCtx, "failed to wait for main container(s)")
					}

					logger.WithFields(logging.Fields{
						"mainContainerNames": mainContainerNames,
						"containerName":      containerName,
					}).Info(innerCtx, "main container(s) exited, terminating container")
					err = em.Kill(innerCtx, []string{containerName}, argoexecexecutor.TerminationGracePeriodDuration())
					if err != nil {
						logger.WithField("containerName", containerName).WithError(err).Error(innerCtx, "failed to terminate/kill container")

View on GitHub (pinned to 35bff19146)

Solutions

1. Check that the container-runtime socket volume (e.g. /run/containerd/containerd.sock) is mounted into executor pods.
2. Verify the workflow executor's containerRuntimeExecutor / emissary configuration matches the cluster runtime.
3. Read the wrapped inner error in the pod logs to identify socket-vs-runtime mismatch.
4. Confirm no admission webhook is dropping the socket hostPath mount from the pod spec.
5. Pin to an Argo version whose emissary implementation supports your container runtime.

// Ensure the runtime socket is mounted in the executor pod spec:
// kubectl get pod <executor-pod> -o jsonpath='{.spec.containers[*].volumeMounts}' | grep -E 'containerd|docker|crio'

try {
  await runWorkflow(containersetTemplate)
} catch (e) {
  if (String(e.cause).includes('failed to create emissary')) {
    // check runtime socket mount + container runtime config
  }
}

When it happens

Trigger: emissary.New() returns an error — typically the container runtime socket (/run/containerd/containerd.sock or docker.sock) is not mounted into the pod, the runtime gRPC connection is refused, or the detected runtime is unsupported.

Common situations: Running with the init-less/emissary container-set executor but the executor kubelet socket/runtime socket volume mounts were stripped by a PodSecurityPolicy/OPA policy; custom runtime class not supported; exec into a sidecar manually for debugging where no runtime socket exists.

Related errors

• failed to read container args file %s: %w
• failed to unmarshal container args: %w
• failed to read template: %w
• failed to start command: %w
• failed waiting for debug-pause-after marker: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/819dcce67fca3a82. Report an issue: GitHub.