{"record":{"id":"ff9f3453ddeed0d6","repo":"argoproj/argo-workflows","slug":"failed-to-open-combined-w","errorCode":null,"errorMessage":"failed to open combined: %w","messagePattern":"failed to open combined: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":662,"sourceCode":"\tcommand := exec.CommandContext(ctx, name, args...)\n\tcommand.Env = os.Environ()\n\n\tvar closer = func() {}\n\tvar stdout io.Writer = os.Stdout\n\tvar stderr io.Writer = os.Stderr\n\n\t// this may not be that important an optimisation, except for very long logs we don't want to capture\n\tif includeScriptOutput || template.SaveLogsAsArtifact() {\n\t\tlogger.Info(ctx, \"capturing logs\")\n\t\tstdoutf, err := os.OpenFile(varRunArgo+\"/ctr/\"+containerName+\"/stdout\", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)\n\t\tif err != nil {\n\t\t\treturn nil, nil, fmt.Errorf(\"failed to open stdout: %w\", err)\n\t\t}\n\t\tcombinedf, err := os.OpenFile(varRunArgo+\"/ctr/\"+containerName+\"/combined\", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)\n\t\tif err != nil {\n\t\t\t// Close stdoutf to avoid leaking the file descriptor opened above.\n\t\t\t_ = stdoutf.Close()\n\t\t\treturn nil, nil, fmt.Errorf(\"failed to open combined: %w\", err)\n\t\t}\n\t\tstdout = io.MultiWriter(stdout, stdoutf, combinedf)\n\t\tstderr = io.MultiWriter(stderr, combinedf)\n\n\t\tcloser = func() {\n\t\t\t_ = stdoutf.Close()\n\t\t\t_ = combinedf.Close()\n\t\t}\n\t}\n\n\tcommand.Stdout = stdout\n\tcommand.Stderr = stderr\n\n\tcmdCloser, err := osspecific.StartCommand(ctx, command)\n\tif err != nil {\n\t\treturn nil, nil, err\n\t}\n","sourceCodeStart":644,"sourceCodeEnd":680,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L644-L680","documentation":"emissary failed to open the combined stdout+stderr log file (/var/run/argo/ctr/<name>/combined) after the stdout file was already opened successfully. The stdout descriptor is explicitly closed before returning to avoid leaking it. This aborts startCommand and thus log capture for the container.","triggerScenarios":"os.OpenFile(varRunArgo+\"/ctr/\"+containerName+\"/combined\", O_CREATE|O_WRONLY|O_APPEND, 0666) fails after the stdout open succeeded: ENOSPC mid-way, inode exhaustion, directory permissions changed, or a race where the ctr directory was removed between the two opens.","commonSituations":"Disk fills up during pod startup so the first open succeeds and the second fails; aggressive cleanup sidecars/garbage collectors deleting /var/run/argo/ctr contents while the pod starts; read-only volume mounted at /var/run/argo.","solutions":["Check the wrapped OS error: ENOSPC means free node disk; EACCES/EROFS means fix volume/securityContext permissions.","Ensure /var/run/argo is a pod-private writable emptyDir so nothing else races with emissary on that directory.","Restart the pod/workflow step; the failure may be transient disk pressure.","If neither file is needed, avoid triggering capture (no outputs.script.result and no saveLogsAsArtifact)."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// shell\ntest -w /var/run/argo/ctr && df -h /var/run/argo || echo \"not writable or missing\"","typeGuard":null,"tryCatchPattern":"err := runStep(ctx)\nif err != nil && strings.Contains(err.Error(), \"failed to open combined\") {\n    // transient ENOSPC/pressure is likely; backoff and retry the step\n    return retryWithBackoff(ctx, err)\n}","preventionTips":["Monitor node disk usage; ENOSPC mid-startup is the usual cause","Do not run cleanup sidecars that delete /var/run/argo while the pod runs","Use a pod-private emptyDir for /var/run/argo to avoid races","Keep error-submission alerts: check errors.Unwrap for the OS-level cause"],"tags":["argo-workflows","argoexec","filesystem","kubernetes"],"backgroundTag":"file-open-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}