argoproj/argo-workflows · critical

failed to create ctr directory: %w

Error message

failed to create ctr directory: %w

What it means

The emissary executor (argoexec emissary, PID 1 in workflow containers) creates `/var/run/argo/ctr/<containerName>` with mode 0777 so all containers (including non-root wait containers) can write exit codes and kill signals. If os.MkdirAll fails, runEmissary returns `failed to create ctr directory: %w`. This is a fatal executor startup failure — the container command never runs.

Source

Thrown at cmd/argoexec/commands/emissary.go:96

			logger.WithError(deferErr).Error(ctx, "Failed to shutdown tracing")
		}
	}()

	ctx = tracing.InjectTraceContext(ctx)
	workflowName := os.Getenv(common.EnvVarWorkflowName)
	namespace, _ := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/namespace")
	ctx, span := tracer.StartRunMainContainer(ctx, workflowName, string(namespace))
	defer span.End()
	injectTraceParent(ctx)

	osspecific.AllowGrantingAccessToEveryone()

	// Dir permission set to rwxrwxrwx, so that non-root wait container can also write kill signal to the folder.
	// Note it's important varRunArgo+"/ctr/" folder is writable by all, because multiple containers may want to
	// write to it with different users.
	// This also indicates we've started.
	if err = os.MkdirAll(varRunArgo+"/ctr/"+containerName, 0o777); err != nil {
		return fmt.Errorf("failed to create ctr directory: %w", err)
	}

	name, args := args[0], args[1:]

	// Check if args were offloaded to a file (for large args that exceed exec limit)
	if argsFile := os.Getenv(common.EnvVarContainerArgsFile); argsFile != "" {
		logger.WithField("argsFile", argsFile).Info(ctx, "Reading container args from file")
		argsData, readErr := os.ReadFile(argsFile)
		if readErr != nil {
			return fmt.Errorf("failed to read container args file %s: %w", argsFile, readErr)
		}
		var fileArgs []string
		if err = json.Unmarshal(argsData, &fileArgs); err != nil {
			return fmt.Errorf("failed to unmarshal container args: %w", err)
		}
		args = append(args, fileArgs...)
		logger.WithField("count", len(fileArgs)).Info(ctx, "Loaded container args from file")

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped cause: EACCES vs EROFS vs ENOSPC.
2. Ensure the pod template includes the /var/run/argo emptyDir volume mount (workflows must not override/remove it).
3. If readOnlyRootFilesystem or a restricted security policy is enforced, keep the /var/run/argo mount writable (emptyDir is fine) and adjust the policy.
4. Check node disk space if the error is ENOSPC.

Example fix

// before
securityContext:
  readOnlyRootFilesystem: true   # and no /var/run/argo volume
// after
volumes:
  - name: var-run-argo
    emptyDir: {}
volumeMounts:
  - name: var-run-argo
    mountPath: /var/run/argo

if err := os.MkdirAll("/var/run/argo/ctr/test", 0o777); err != nil { /* /var/run/argo not writable: check volume mounts & securityContext */ }

if err := os.MkdirAll(varRunArgo+"/ctr/"+containerName, 0o777); err != nil {
    return fmt.Errorf("failed to create ctr directory (is /var/run/argo mounted writable?): %w", err)
}

When it happens

Trigger: /var/run/argo does not exist or is read-only in the container (missing emptyDir volume mount from the pod spec, or read-only rootfs); disk full; permission denied creating the dir under a restrictive security context.

Common situations: Custom pod specs or security policies (readOnlyRootFilesystem, restricted PSP/PSA) stripping the /var/run/argo volume; tampered workflow pod specs not created by the controller; nodes under disk pressure.

Related errors

• failed to create dependency dir: %w
• stat supervisor status: %w
• failed to open stdout: %w
• failed to open combined: %w
• failed to create directory %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/82ccea7740c81836. Report an issue: GitHub.