{"record":{"id":"a8668a1a839d66c4","repo":"argoproj/argo-workflows","slug":"failed-to-open-stdout-w","errorCode":null,"errorMessage":"failed to open stdout: %w","messagePattern":"failed to open stdout: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":656,"sourceCode":"\treturn current\n}\n\nfunc startCommand(ctx context.Context, name string, args []string, template *wfv1.Template, containerName string, includeScriptOutput bool) (*exec.Cmd, func(), error) {\n\tlogger := logging.RequireLoggerFromContext(ctx)\n\n\tcommand := exec.CommandContext(ctx, name, args...)\n\tcommand.Env = os.Environ()\n\n\tvar closer = func() {}\n\tvar stdout io.Writer = os.Stdout\n\tvar stderr io.Writer = os.Stderr\n\n\t// this may not be that important an optimisation, except for very long logs we don't want to capture\n\tif includeScriptOutput || template.SaveLogsAsArtifact() {\n\t\tlogger.Info(ctx, \"capturing logs\")\n\t\tstdoutf, err := os.OpenFile(varRunArgo+\"/ctr/\"+containerName+\"/stdout\", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)\n\t\tif err != nil {\n\t\t\treturn nil, nil, fmt.Errorf(\"failed to open stdout: %w\", err)\n\t\t}\n\t\tcombinedf, err := os.OpenFile(varRunArgo+\"/ctr/\"+containerName+\"/combined\", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)\n\t\tif err != nil {\n\t\t\t// Close stdoutf to avoid leaking the file descriptor opened above.\n\t\t\t_ = stdoutf.Close()\n\t\t\treturn nil, nil, fmt.Errorf(\"failed to open combined: %w\", err)\n\t\t}\n\t\tstdout = io.MultiWriter(stdout, stdoutf, combinedf)\n\t\tstderr = io.MultiWriter(stderr, combinedf)\n\n\t\tcloser = func() {\n\t\t\t_ = stdoutf.Close()\n\t\t\t_ = combinedf.Close()\n\t\t}\n\t}\n\n\tcommand.Stdout = stdout\n\tcommand.Stderr = stderr","sourceCodeStart":638,"sourceCodeEnd":674,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L638-L674","documentation":"argoexec's emissary PID-1 process failed to open the per-container stdout log file (/var/run/argo/ctr/<name>/stdout) for appending when log capture is enabled (includeScriptOutput or template.SaveLogsAsArtifact). The underlying OS error is wrapped with %w so the original cause (permissions, ENOSPC, missing directory) is preserved. This aborts startCommand, so the wrapped user command never starts correctly.","triggerScenarios":"os.OpenFile(varRunArgo+\"/ctr/\"+containerName+\"/stdout\", O_CREATE|O_WRONLY|O_APPEND, 0666) returns an error: the /var/run/argo/ctr/<containerName>/ directory does not exist or was deleted, the filesystem is read-only or full, or the file is owned by another UID with restrictive permissions.","commonSituations":"Custom executor images without /var/run/argo mounted or with wrong permissions; securityContext (readOnlyRootFilesystem, runAsUser mismatch) blocking writes to /var/run/argo; disk pressure (ENOSPC) on the node; volume mounted noexec/ro; containerName containing unexpected characters creating a bad path.","solutions":["Verify the /var/run/argo/ctr/<containerName> directory exists and is writable by the argoexec user (fix pod securityContext or emptyDir mount).","Check node disk space (df -h) and clear pressure if ENOSPC appears in the wrapped error.","Confirm the pod mounts an emptyDir (or writable volume) at /var/run/argo and the root filesystem is not read-only.","If script-output capture is not needed, remove outputs.script.result / SaveLogsAsArtifact usage from the template so the open path is skipped."],"exampleFix":"// before: pod spec with readOnlyRootFilesystem and no /var/run/argo volume\n// after:\nvolumes:\n- name: var-run-argo\n  emptyDir: {}\ncontainers:\n- name: main\n  volumeMounts:\n  - name: var-run-argo\n    mountPath: /var/run/argo","handlingStrategy":"try-catch","validationCode":"// shell, inside the workflow container before submitting steps relying on log capture\nls -ld /var/run/argo /var/run/argo/ctr || mkdir -p /var/run/argo/ctr\ndf -h /var/run/argo  # ensure free space and rw mount","typeGuard":null,"tryCatchPattern":"err := runStep(ctx)\nif err != nil && strings.Contains(err.Error(), \"failed to open stdout\") {\n    // inspect wrapped cause via errors.Unwrap; fix volume/securityContext, then retry\n    return retryStep(ctx, err)\n}","preventionTips":["Always mount a writable emptyDir at /var/run/argo when using custom executor images","Avoid readOnlyRootFilesystem without a rw /var/run/argo mount","Keep node disks below capacity; alert on kubelet DiskPressure","Pin runAsUser so argoexec can write its own runtime directory"],"tags":["argo-workflows","argoexec","filesystem","kubernetes"],"backgroundTag":"file-open-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}