What causes "mkdir %s: %w" in argoproj/argo-workflows?

Check no file exists at the target directory path and use a distinct artifact path

mkdir %s: %w

Error message

mkdir %s: %w

What it means

When downloading a GCS object (or a directory-expanded set of objects), downloadObject computes the local path and creates parent directories with os.MkdirAll(objectDir, 0o700) before writing. This error wraps a failure of that mkdir. It indicates the local artifact destination directory tree could not be created.

Source

Thrown at workflow/artifacts/gcs/gcs.go:163

	}
	for _, objName := range objNames {
		err = downloadObject(ctx, client, bucket, key, objName, path)
		if err != nil {
			return err
		}
	}
	return nil
}

// download an object from the bucket
func downloadObject(ctx context.Context, client *storage.Client, bucket, key, objName, path string) error {
	objPrefix := normalizeGCSKey(filepath.Clean(key))
	relObjPath := strings.TrimPrefix(objName, objPrefix)
	localPath := filepath.Join(path, relObjPath)
	objectDir, _ := filepath.Split(localPath)
	if objectDir != "" {
		if err := os.MkdirAll(objectDir, 0o700); err != nil {
			return fmt.Errorf("mkdir %s: %w", objectDir, err)
		}
	}
	rc, err := client.Bucket(bucket).Object(objName).NewReader(ctx)
	if err != nil {
		if errors.Is(err, storage.ErrObjectNotExist) {
			return argoerrors.New(argoerrors.CodeNotFound, err.Error())
		}
		return fmt.Errorf("new bucket reader: %w", err)
	}
	defer rc.Close()
	out, err := os.Create(localPath)
	if err != nil {
		return fmt.Errorf("os create %s: %w", localPath, err)
	}
	defer func() {
		if closeErr := out.Close(); closeErr != nil {
			logger := logging.RequireLoggerFromContext(ctx)
			logger.WithField("path", localPath).WithError(closeErr).Error(ctx, "Error closing file")

View on GitHub (pinned to 35bff19146)

Solutions

Check no file exists at the target directory path and use a distinct artifact path
Make the output volume writable by the container user (securityContext fsGroup / chmod)
Verify the volume is not mounted read-only in the pod spec
Check disk and filesystem health on the node

Example fix

// before: path collides with an existing file
artifacts: [{name: out, path: /mnt/out}]
// /mnt/out exists as a FILE
// after
artifacts: [{name: out, path: /mnt/artifacts/out}]  // unique directory path

Defensive patterns

Strategy: validation

Validate before calling

import "os"
func ensureArtifactDirWritable(path string) error {
	if fi, err := os.Stat(path); err == nil && !fi.IsDir() {
		return fmt.Errorf("%s exists as a file", path)
	}
	probe := filepath.Join(path, ".argo-write-probe")
	if err := os.MkdirAll(filepath.Dir(probe), 0o700); err != nil { return err }
	return os.Remove(probe)
}

Try / catch

if err := downloadArtifacts(ctx); err != nil {
	if strings.Contains(err.Error(), "mkdir ") && os.IsPermission(errors.Unwrap(err)) {
		// fix volume permissions/fsGroup, then retry
	}
	return err
}

Prevention

Give each artifact a unique directory path to avoid file/dir collisions
Set securityContext.fsGroup to match volume ownership
Never mount output volumes readOnly for steps that save artifacts

When it happens

Trigger: os.MkdirAll fails while preparing the target path for an artifact download: parent path component exists as a regular file, permission denied on the output mount, path too long, or read-only filesystem.

Common situations: Artifact path collides with an existing file of the same name; output volume mounted read-only; non-root container lacking write permission to the path; NFS/emptyDir I/O errors.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/d860bb099f9e64c0. Report an issue: GitHub.