argoproj/argo-workflows · critical

failed to start command: %w

Error message

failed to start command: %w

What it means

argoexec's emissary runner wraps startCommand inside a retry.OnError loop; if the child process cannot be started at all, the underlying exec error is wrapped as "failed to start command". Because the retry predicate returns true for every error, the start attempt is retried according to the template's retryStrategy before this error finally surfaces. It means the user's command never ran (or never re-ran) rather than that it ran and failed.

Source

Thrown at cmd/argoexec/commands/emissary.go:231

	if os.Getenv("ARGO_DEBUG_PAUSE_BEFORE") == "true" {
		// User can create the file: /ctr/NAME_OF_THE_CONTAINER/before
		// in order to break out of the wait and release the container from
		// the debugging state.
		if waitErr := file.WaitForCreate(ctx, varRunArgo+"/ctr/"+containerName+"/before"); waitErr != nil {
			return fmt.Errorf("failed waiting for debug-pause-before marker: %w", waitErr)
		}
	}

	backoff, err := template.GetRetryStrategy()
	if err != nil {
		return fmt.Errorf("failed to get retry strategy: %w", err)
	}

	cmdErr := retry.OnError(backoff, func(error) bool { return true }, func() error {
		command, closer, err := startCommand(ctx, name, args, template, containerName, includeScriptOutput)
		if err != nil {
			return fmt.Errorf("failed to start command: %w", err)
		}
		defer closer()

		forwardSignals(ctx, signals, command.Process.Pid, false)
		pid := command.Process.Pid
		innerCtx, cancel := context.WithCancel(ctx)
		defer cancel()
		startFileSignalHandler(innerCtx, pid, containerName)
		for _, sidecarName := range template.GetSidecarNames() {
			if sidecarName == containerName {
				em, err := emissary.New()
				if err != nil {
					return fmt.Errorf("failed to create emissary: %w", err)
				}

				go func() {
					mainContainerNames := template.GetMainContainerNames()
					err = em.Wait(innerCtx, mainContainerNames)

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the workflow pod's logs (argo logs / kubectl logs) for the underlying wrapped error to see the real cause.
2. Verify the template's command/args exist and are executable in the image (kubectl run <image> -- <cmd> --version).
3. Confirm the image contains a shell if the template relies on shell semantics.
4. Check container resource limits and node cgroup pressure that can make fork/exec fail.
5. If transient, add/adjust retryStrategy in the template — emissary already retries starts via retry.OnError.

Example fix

// before
command: ["python3", "train.py"]   # python3 not in image
// after
command: ["python", "train.py"]    # or use an image that ships python3

// Validate the command exists in the image before submitting:
// kubectl run --rm -it check --image=<image> -- sh -c 'command -v <binary>'
// Or in CI, lint the template and smoke-test the image:
docker run --rm --entrypoint sh <image> -c 'test -x "$(command -v mycmd)"'

try {
  await runWorkflow(template)
} catch (e) {
  if (String(e.cause).includes('failed to start command')) {
    // inspect pod logs for the wrapped exec error; fix image/command
  }
}

When it happens

Trigger: startCommand fails: os/exec cannot spawn the binary (binary missing despite LookPath passing, e.g. deleted between lookup and exec), fork/exec fails due to cgroup limits, the template references a script/command not present in the image, or an ioctl/TIOCSIG/pdeathsig setup error occurs while wiring up the process.

Common situations: Workflow template sets command to a binary that does not exist in the container image (typo, wrong image tag); image is distroless and lacks the shell; the pod is OOM-thrashing so fork fails; NFS/overlay corruption makes the binary unreadable; permission bits removed from the executable.

Related errors

• failed to read container args file %s: %w
• failed to unmarshal container args: %w
• failed to read template: %w
• neither %s nor %s is available
• failed to write large arg %d to file: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/c36dd841ff198a69. Report an issue: GitHub.