argoproj/argo-workflows · error

failed to open stdout: %w

Error message

failed to open stdout: %w

What it means

argoexec's emissary PID-1 process failed to open the per-container stdout log file (/var/run/argo/ctr/<name>/stdout) for appending when log capture is enabled (includeScriptOutput or template.SaveLogsAsArtifact). The underlying OS error is wrapped with %w so the original cause (permissions, ENOSPC, missing directory) is preserved. This aborts startCommand, so the wrapped user command never starts correctly.

Source

Thrown at cmd/argoexec/commands/emissary.go:656

	return current
}

func startCommand(ctx context.Context, name string, args []string, template *wfv1.Template, containerName string, includeScriptOutput bool) (*exec.Cmd, func(), error) {
	logger := logging.RequireLoggerFromContext(ctx)

	command := exec.CommandContext(ctx, name, args...)
	command.Env = os.Environ()

	var closer = func() {}
	var stdout io.Writer = os.Stdout
	var stderr io.Writer = os.Stderr

	// this may not be that important an optimisation, except for very long logs we don't want to capture
	if includeScriptOutput || template.SaveLogsAsArtifact() {
		logger.Info(ctx, "capturing logs")
		stdoutf, err := os.OpenFile(varRunArgo+"/ctr/"+containerName+"/stdout", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
		if err != nil {
			return nil, nil, fmt.Errorf("failed to open stdout: %w", err)
		}
		combinedf, err := os.OpenFile(varRunArgo+"/ctr/"+containerName+"/combined", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
		if err != nil {
			// Close stdoutf to avoid leaking the file descriptor opened above.
			_ = stdoutf.Close()
			return nil, nil, fmt.Errorf("failed to open combined: %w", err)
		}
		stdout = io.MultiWriter(stdout, stdoutf, combinedf)
		stderr = io.MultiWriter(stderr, combinedf)

		closer = func() {
			_ = stdoutf.Close()
			_ = combinedf.Close()
		}
	}

	command.Stdout = stdout
	command.Stderr = stderr

View on GitHub (pinned to 35bff19146)

Solutions

1. Verify the /var/run/argo/ctr/<containerName> directory exists and is writable by the argoexec user (fix pod securityContext or emptyDir mount).
2. Check node disk space (df -h) and clear pressure if ENOSPC appears in the wrapped error.
3. Confirm the pod mounts an emptyDir (or writable volume) at /var/run/argo and the root filesystem is not read-only.
4. If script-output capture is not needed, remove outputs.script.result / SaveLogsAsArtifact usage from the template so the open path is skipped.

Example fix

// before: pod spec with readOnlyRootFilesystem and no /var/run/argo volume
// after:
volumes:
- name: var-run-argo
  emptyDir: {}
containers:
- name: main
  volumeMounts:
  - name: var-run-argo
    mountPath: /var/run/argo

// shell, inside the workflow container before submitting steps relying on log capture
ls -ld /var/run/argo /var/run/argo/ctr || mkdir -p /var/run/argo/ctr
df -h /var/run/argo  # ensure free space and rw mount

err := runStep(ctx)
if err != nil && strings.Contains(err.Error(), "failed to open stdout") {
    // inspect wrapped cause via errors.Unwrap; fix volume/securityContext, then retry
    return retryStep(ctx, err)
}

When it happens

Trigger: os.OpenFile(varRunArgo+"/ctr/"+containerName+"/stdout", O_CREATE|O_WRONLY|O_APPEND, 0666) returns an error: the /var/run/argo/ctr/<containerName>/ directory does not exist or was deleted, the filesystem is read-only or full, or the file is owned by another UID with restrictive permissions.

Common situations: Custom executor images without /var/run/argo mounted or with wrong permissions; securityContext (readOnlyRootFilesystem, runAsUser mismatch) blocking writes to /var/run/argo; disk pressure (ENOSPC) on the node; volume mounted noexec/ro; containerName containing unexpected characters creating a bad path.

Understand the failure class

Background: "open() failed", "failed to open file", "cannot create file" — what a file open error means and how to fix it — this error's family across 42 libraries.

Related errors

• stat supervisor status: %w
• failed to open combined: %w
• failed to read container args file %s: %w
• failed to read template: %w
• failed to create dependency dir: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a8668a1a839d66c4. Report an issue: GitHub.