{"record":{"id":"c36dd841ff198a69","repo":"argoproj/argo-workflows","slug":"failed-to-start-command-w","errorCode":null,"errorMessage":"failed to start command: %w","messagePattern":"failed to start command: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":231,"sourceCode":"\n\tif os.Getenv(\"ARGO_DEBUG_PAUSE_BEFORE\") == \"true\" {\n\t\t// User can create the file: /ctr/NAME_OF_THE_CONTAINER/before\n\t\t// in order to break out of the wait and release the container from\n\t\t// the debugging state.\n\t\tif waitErr := file.WaitForCreate(ctx, varRunArgo+\"/ctr/\"+containerName+\"/before\"); waitErr != nil {\n\t\t\treturn fmt.Errorf(\"failed waiting for debug-pause-before marker: %w\", waitErr)\n\t\t}\n\t}\n\n\tbackoff, err := template.GetRetryStrategy()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get retry strategy: %w\", err)\n\t}\n\n\tcmdErr := retry.OnError(backoff, func(error) bool { return true }, func() error {\n\t\tcommand, closer, err := startCommand(ctx, name, args, template, containerName, includeScriptOutput)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to start command: %w\", err)\n\t\t}\n\t\tdefer closer()\n\n\t\tforwardSignals(ctx, signals, command.Process.Pid, false)\n\t\tpid := command.Process.Pid\n\t\tinnerCtx, cancel := context.WithCancel(ctx)\n\t\tdefer cancel()\n\t\tstartFileSignalHandler(innerCtx, pid, containerName)\n\t\tfor _, sidecarName := range template.GetSidecarNames() {\n\t\t\tif sidecarName == containerName {\n\t\t\t\tem, err := emissary.New()\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to create emissary: %w\", err)\n\t\t\t\t}\n\n\t\t\t\tgo func() {\n\t\t\t\t\tmainContainerNames := template.GetMainContainerNames()\n\t\t\t\t\terr = em.Wait(innerCtx, mainContainerNames)","sourceCodeStart":213,"sourceCodeEnd":249,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L213-L249","documentation":"argoexec's emissary runner wraps startCommand inside a retry.OnError loop; if the child process cannot be started at all, the underlying exec error is wrapped as \"failed to start command\". Because the retry predicate returns true for every error, the start attempt is retried according to the template's retryStrategy before this error finally surfaces. It means the user's command never ran (or never re-ran) rather than that it ran and failed.","triggerScenarios":"startCommand fails: os/exec cannot spawn the binary (binary missing despite LookPath passing, e.g. deleted between lookup and exec), fork/exec fails due to cgroup limits, the template references a script/command not present in the image, or an ioctl/TIOCSIG/pdeathsig setup error occurs while wiring up the process.","commonSituations":"Workflow template sets command to a binary that does not exist in the container image (typo, wrong image tag); image is distroless and lacks the shell; the pod is OOM-thrashing so fork fails; NFS/overlay corruption makes the binary unreadable; permission bits removed from the executable.","solutions":["Check the workflow pod's logs (argo logs / kubectl logs) for the underlying wrapped error to see the real cause.","Verify the template's command/args exist and are executable in the image (kubectl run <image> -- <cmd> --version).","Confirm the image contains a shell if the template relies on shell semantics.","Check container resource limits and node cgroup pressure that can make fork/exec fail.","If transient, add/adjust retryStrategy in the template — emissary already retries starts via retry.OnError."],"exampleFix":"// before\ncommand: [\"python3\", \"train.py\"]   # python3 not in image\n// after\ncommand: [\"python\", \"train.py\"]    # or use an image that ships python3","handlingStrategy":"validation","validationCode":"// Validate the command exists in the image before submitting:\n// kubectl run --rm -it check --image=<image> -- sh -c 'command -v <binary>'\n// Or in CI, lint the template and smoke-test the image:\ndocker run --rm --entrypoint sh <image> -c 'test -x \"$(command -v mycmd)\"'","typeGuard":null,"tryCatchPattern":"try {\n  await runWorkflow(template)\n} catch (e) {\n  if (String(e.cause).includes('failed to start command')) {\n    // inspect pod logs for the wrapped exec error; fix image/command\n  }\n}","preventionTips":["Smoke-test the image with the exact command before wiring it into a template.","Use full absolute paths for commands in templates.","Keep retryStrategy configured so transient fork failures are retried.","Watch for image tags that silently changed underneath the template."],"tags":["kubernetes","argo-workflows","process-exec","executor"],"backgroundTag":"command-not-found","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}