argoproj/argo-workflows · error

failed to stat artifact path %q at %s: %w

Error message

failed to stat artifact path %q at %s: %w

What it means

After ensuring the parent directory exists, the executor Lstats the artifact destination (art.Path) to decide whether a safe symlink create suffices or a guarded overwrite is needed. A stat error that is NOT 'does not exist' is fatal and wrapped here — the executor cannot determine what currently sits at the destination, so it refuses to proceed rather than risk destroying data with a RemoveAll.

Source

Thrown at cmd/argoexec/commands/emissary.go:409

			return fmt.Errorf("failed to stat input artifact %q at %s: %w", art.Name, src, statErr)
		}
		dst := art.Path
		if dst == "" {
			continue
		}
		if parent := filepath.Dir(dst); parent != "" && parent != "/" {
			if err := os.MkdirAll(parent, 0o755); err != nil {
				return fmt.Errorf("failed to create parent directory for artifact %q at %s: %w", art.Name, dst, err)
			}
		}
		// If nothing exists at art.Path, just create the symlink. Creating is
		// always safe — os.Symlink returns EEXIST rather than overwriting and the
		// MkdirAll above only ever creates — so even when art.Path resolves into a
		// user volume we deliberately let the artifact land there (the user asked
		// for it). Only an *overwrite* can destroy data, and that is gated below.
		if _, err := os.Lstat(dst); err != nil {
			if !os.IsNotExist(err) {
				return fmt.Errorf("failed to stat artifact path %q at %s: %w", art.Name, dst, err)
			}
		} else {
			// Something is already at art.Path. Replacing it (os.RemoveAll then
			// symlink) reproduces the legacy SubPath mount's shadowing — but only
			// when it is safe. RemoveAll resolves symlinks in the parent chain, so
			// resolve the parent to find where the delete would actually land
			// (resolve the parent, not the final element, so an image symlink *at*
			// art.Path is just unlinked rather than followed). If that resolved
			// path overlaps a user-declared volume, clearing it would recurse into
			// and destroy a live PVC/hostPath/emptyDir, so refuse. Benign system
			// mounts (tmpfs /run, the overlay rootfs) are not declared user volumes
			// and so remain safe to shadow.
			realParent, evalErr := filepath.EvalSymlinks(filepath.Dir(dst))
			if evalErr != nil {
				return fmt.Errorf("failed to resolve parent of artifact path %q at %s: %w", art.Name, dst, evalErr)
			}
			resolved := filepath.Join(realParent, filepath.Base(dst))
			if mnt := common.FindOverlappingVolume(tmpl, resolved); mnt != nil {

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped errno: EACCES → grant the executor user read/execute on the destination directory chain; ELOOP → fix the symlink cycle.
2. For NFS/network volumes, retry the workflow — transient EIO often resolves.
3. Ensure artifact.path's parent chain consists of real directories, not symlink loops.
4. Match volume permissions with securityContext fsGroup so the executor can traverse the path.
5. If the path should simply not exist, pre-clean it in the pod setup so Lstat returns ENOENT and the plain create path is used.

Example fix

// before (EACCES on traversal)
volumeMount:
  name: data
  mountPath: /data
# dir mode 0700 owned by 1000; executor runs as root... or vice versa
// after
chmod o+rx /data   # or set pod fsGroup so the executor can traverse

// Pre-flight: destination must be statable or nonexistent:
if [ -e '<artifact.path>' ] && [ ! -r "$(dirname '<artifact.path>')" ]; then
  echo "cannot inspect $(dirname '<artifact.path>')" >&2; exit 1
fi

try {
  await stageArtifacts(tmpl)
} catch (e) {
  if (String(e).includes('failed to stat artifact path')) {
    // fix traversal permissions or symlink loops; retry on transient EIO
  }
}

When it happens

Trigger: os.Lstat(dst) fails with something other than ENOENT: EACCES on the destination directory, ELOOP from a symlink cycle in the path, or EIO on the underlying volume while probing the destination path.

Common situations: Artifact paths on volumes with restrictive permissions (executor can list parent but not lstat children); symlink loops in user-mounted volumes; flaky network storage (NFS/EFS) returning transient I/O errors; destination inside a containerd subPath mount that is mid-unmount.

Related errors

• failed to resolve parent of artifact path %q at %s: %w
• failed to stat input artifact %q at %s: %w
• failed to create parent directory for artifact %q at %s: %w
• failed to clear existing path for artifact %q at %s: %w
• failed to symlink input artifact %q (%s -> %s): %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/62e6a9509117e462. Report an issue: GitHub.