What causes "Couldn't create symlink: %w" in hashicorp/nomad?

Inspect the wrapped error's Err value to learn the actual syscall failure (errno) for taskEntry

Couldn't create symlink: %w

Error message

Couldn't create symlink: %w

What it means

After successfully reading a host symlink, embedDirs recreates it inside the task directory via os.Symlink. This error is returned when Symlink fails for any reason other than the link already existing (which is tolerated as 'symlinking twice'). It indicates the task-directory side of the chroot could not be populated.

Source

Thrown at client/allocdir/task_dir.go:300

			if _, err := os.Lstat(taskEntry); err == nil {
				continue
			}

			if !entry.Mode().IsRegular() {
				// If it is a symlink we can create it, otherwise we skip it.
				if entry.Mode()&os.ModeSymlink == 0 {
					continue
				}

				link, err := os.Readlink(hostEntry)
				if err != nil {
					return fmt.Errorf("Couldn't resolve symlink for %v: %w", source, err)
				}

				if err := os.Symlink(link, taskEntry); err != nil {
					// Symlinking twice
					if err.(*os.LinkError).Err.Error() != "file exists" {
						return fmt.Errorf("Couldn't create symlink: %w", err)
					}
				}
				continue
			}

			uid, gid := getOwner(entry)
			if err := linkOrCopy(hostEntry, taskEntry, uid, gid, entry.Mode().Perm()); err != nil {
				return err
			}
		}
	}

	// Recurse on self to copy subdirectories.
	if len(subdirs) != 0 {
		return t.embedDirs(subdirs)
	}

	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped error's Err value to learn the actual syscall failure (errno) for taskEntry
Verify the Nomad client user owns/is writable the alloc/task directory (check st_uid and mode of the data dir)
Remove the stale allocation directory so the chroot is rebuilt from scratch, then reschedule the alloc
Ensure the client data dir lives on a local filesystem that supports symlinks (avoid exotic network mounts)

Example fix

// before
if err := os.Symlink(link, taskEntry); err != nil {
    if err.(*os.LinkError).Err.Error() != "file exists" {
        return fmt.Errorf("Couldn't create symlink: %w", err)
    }
}
// after
if err := os.Symlink(link, taskEntry); err != nil {
    if !errors.Is(err, os.ErrExist) {
        return fmt.Errorf("Couldn't create symlink: %w", err)
    }
}

Defensive patterns

Strategy: validation

Validate before calling

// Ensure task dir is writable and symlink-capable before building chroot
fi, err := os.Stat(taskDir)
if err != nil || !fi.IsDir() { return fmt.Errorf("bad task dir") }
if err := os.Chmod(taskDir, fi.Mode()|0700); err != nil { return err }
tmp := filepath.Join(taskDir, ".symlink_probe")
if err := os.Symlink(".", tmp); err != nil { return fmt.Errorf("symlinks unsupported: %w", err) }
os.Remove(tmp)

Try / catch

if err != nil {
  var le *os.LinkError
  if errors.As(err, &le) && errors.Is(le, os.ErrExist) {
    return nil // tolerated duplicate
  }
  return err
}

Prevention

Keep the client data dir owned solely by the Nomad client user
Place data_dir on a local filesystem that supports symlinks (ext4/xfs), not network mounts
Clean stale alloc directories after crashes before re-allocating on the same node
After OS/user changes, verify the agent can write its data dir

When it happens

Trigger: os.Symlink(link, taskEntry) fails with an error whose Err string is not 'file exists' — e.g. EACCES/EPERM (task dir not writable by the client user), ENAMETOOLONG, EPERM on filesystems forbidding symlinks, or a non-symlink file already occupying taskEntry.

Common situations: Task alloc dir permissions corrupted or owned by another user; allocating on a filesystem (some network mounts, Windows pre-DevMode) that disallows symlink creation; a leftover real file at the task path from a previous interrupted alloc.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/bc328e2b171044d1. Report an issue: GitHub.