argoproj/argo-workflows · error

failed to open combined: %w

Error message

failed to open combined: %w

What it means

emissary failed to open the combined stdout+stderr log file (/var/run/argo/ctr/<name>/combined) after the stdout file was already opened successfully. The stdout descriptor is explicitly closed before returning to avoid leaking it. This aborts startCommand and thus log capture for the container.

Source

Thrown at cmd/argoexec/commands/emissary.go:662

	command := exec.CommandContext(ctx, name, args...)
	command.Env = os.Environ()

	var closer = func() {}
	var stdout io.Writer = os.Stdout
	var stderr io.Writer = os.Stderr

	// this may not be that important an optimisation, except for very long logs we don't want to capture
	if includeScriptOutput || template.SaveLogsAsArtifact() {
		logger.Info(ctx, "capturing logs")
		stdoutf, err := os.OpenFile(varRunArgo+"/ctr/"+containerName+"/stdout", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
		if err != nil {
			return nil, nil, fmt.Errorf("failed to open stdout: %w", err)
		}
		combinedf, err := os.OpenFile(varRunArgo+"/ctr/"+containerName+"/combined", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
		if err != nil {
			// Close stdoutf to avoid leaking the file descriptor opened above.
			_ = stdoutf.Close()
			return nil, nil, fmt.Errorf("failed to open combined: %w", err)
		}
		stdout = io.MultiWriter(stdout, stdoutf, combinedf)
		stderr = io.MultiWriter(stderr, combinedf)

		closer = func() {
			_ = stdoutf.Close()
			_ = combinedf.Close()
		}
	}

	command.Stdout = stdout
	command.Stderr = stderr

	cmdCloser, err := osspecific.StartCommand(ctx, command)
	if err != nil {
		return nil, nil, err
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped OS error: ENOSPC means free node disk; EACCES/EROFS means fix volume/securityContext permissions.
2. Ensure /var/run/argo is a pod-private writable emptyDir so nothing else races with emissary on that directory.
3. Restart the pod/workflow step; the failure may be transient disk pressure.
4. If neither file is needed, avoid triggering capture (no outputs.script.result and no saveLogsAsArtifact).

// shell
test -w /var/run/argo/ctr && df -h /var/run/argo || echo "not writable or missing"

err := runStep(ctx)
if err != nil && strings.Contains(err.Error(), "failed to open combined") {
    // transient ENOSPC/pressure is likely; backoff and retry the step
    return retryWithBackoff(ctx, err)
}

When it happens

Trigger: os.OpenFile(varRunArgo+"/ctr/"+containerName+"/combined", O_CREATE|O_WRONLY|O_APPEND, 0666) fails after the stdout open succeeded: ENOSPC mid-way, inode exhaustion, directory permissions changed, or a race where the ctr directory was removed between the two opens.

Common situations: Disk fills up during pod startup so the first open succeeds and the second fails; aggressive cleanup sidecars/garbage collectors deleting /var/run/argo/ctr contents while the pod starts; read-only volume mounted at /var/run/argo.

Understand the failure class

Background: "open() failed", "failed to open file", "cannot create file" — what a file open error means and how to fix it — this error's family across 42 libraries.

Related errors

• stat supervisor status: %w
• failed to open stdout: %w
• failed to read container args file %s: %w
• failed to read template: %w
• failed to create dependency dir: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ff9f3453ddeed0d6. Report an issue: GitHub.