argoproj/argo-workflows · critical

failed to read container args file %s: %w

Error message

failed to read container args file %s: %w

What it means

In `argoexec emissary`, when the controller offloads oversized container args to a file (env var ARGO_CONTAINER_ARGS_FILE is set, because the marshaled args exceeded MaxEnvVarLen), runEmissary reads that file with os.ReadFile. This error is thrown when that read fails (missing file, wrong path, permission denied). Without the args the emissary cannot exec the user command, so the container fails immediately. The underlying OS error is wrapped with %w so errors.Is/As still work.

Source

Thrown at cmd/argoexec/commands/emissary.go:106

	osspecific.AllowGrantingAccessToEveryone()

	// Dir permission set to rwxrwxrwx, so that non-root wait container can also write kill signal to the folder.
	// Note it's important varRunArgo+"/ctr/" folder is writable by all, because multiple containers may want to
	// write to it with different users.
	// This also indicates we've started.
	if err = os.MkdirAll(varRunArgo+"/ctr/"+containerName, 0o777); err != nil {
		return fmt.Errorf("failed to create ctr directory: %w", err)
	}

	name, args := args[0], args[1:]

	// Check if args were offloaded to a file (for large args that exceed exec limit)
	if argsFile := os.Getenv(common.EnvVarContainerArgsFile); argsFile != "" {
		logger.WithField("argsFile", argsFile).Info(ctx, "Reading container args from file")
		argsData, readErr := os.ReadFile(argsFile)
		if readErr != nil {
			return fmt.Errorf("failed to read container args file %s: %w", argsFile, readErr)
		}
		var fileArgs []string
		if err = json.Unmarshal(argsData, &fileArgs); err != nil {
			return fmt.Errorf("failed to unmarshal container args: %w", err)
		}
		args = append(args, fileArgs...)
		logger.WithField("count", len(fileArgs)).Info(ctx, "Loaded container args from file")

		// Check for a large args and offload to file if needed
		// This avoids the exec() "argument list too long" error
		// Downstream programs should support @filename for parsing large args
		for i := 0; i < len(args); i++ {
			if len(args[i]) > common.MaxEnvVarLen {
				filePath := fmt.Sprintf("/tmp/argo_arg_%d.txt", i)
				if err = os.WriteFile(filePath, []byte(args[i]), 0o644); err != nil {
					return fmt.Errorf("failed to write large arg %d to file: %w", i, err)
				}
				logger.WithFields(logging.Fields{

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the pod spec (`kubectl get pod <pod> -o yaml`): confirm the env var ARGO_CONTAINER_ARGS_FILE is set and the referenced ConfigMap exists and is mounted at that exact path.
2. Verify the ConfigMap content is valid JSON of the container args (`kubectl get cm <pod-name> -o jsonpath='{.data}'`).
3. Upgrade workflow-controller, argo-server and executor images to the same Argo Workflows version so the args-offload contract matches.
4. Reduce the container args size (e.g. shorten script/argument payloads) so the controller does not take the offload path at all.
5. If it reproduces, check controller logs for ConfigMap creation errors around pod creation time.

Example fix

// before: emissary started while ConfigMap volume was missing
// spec.containers[*].volumeMounts missing the args configmap
// after: ensure the offload configmap is mounted (controller does this automatically; custom patches must not remove it)
volumeMounts:
  - name: argo-task-args        # configmap named after the pod
    mountPath: /argo/staging    # must match ARGO_CONTAINER_ARGS_FILE dir

// Before submitting workflows with very large container args, verify the offload prerequisites:
import "os"

func validateArgsOffloadReady() error {
	path := os.Getenv("ARGO_CONTAINER_ARGS_FILE")
	if path == "" {
		return nil // offload not in use
	}
	f, err := os.Open(path)
	if err != nil {
		return fmt.Errorf("args offload file %s not accessible: %w", path, err)
	}
	defer f.Close()
	return nil
}

// On the error, surface the wrapped OS error for diagnosis:
if err := runEmissary(ctx, name, includeScript, args); err != nil {
	if errors.Is(err, os.ErrNotExist) {
		log.Errorf("offloaded args file missing — ConfigMap not mounted? %v", err)
	} else if errors.Is(err, os.ErrPermission) {
		log.Errorf("args offload file unreadable — check securityContext: %v", err)
	}
}

When it happens

Trigger: ARGO_CONTAINER_ARGS_FILE points to a path that does not exist or is unreadable at the time emissary starts: the ConfigMap holding the offloaded args (named after the pod, created in workflowpod.go build()) was not mounted, was deleted before the container started, or the mount path in the env var does not match the volumeMount.

Common situations: Running with a controller version that offloads args but an executor image too old/new to agree on the env var name or mount location; a ConfigMap-name collision or garbage-collection race in a busy namespace; a custom pod spec patch that drops or renames the args ConfigMap volume; read-only or restrictive securityContext blocking the mount.

Related errors

• failed to read template: %w
• failed to unmarshal container args: %w
• failed to write large arg %d to file: %w
• failed to start command: %w
• neither %s nor %s is available

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/0a59ea2973fce3ff. Report an issue: GitHub.