hashicorp/nomad · error

Couldn't resolve symlink for %v: %w

Error message

Couldn't resolve symlink for %v: %w

What it means

embedDirs walks a host directory tree and mirrors symlinks into the task's chroot directory. This error wraps os.Readlink failing to resolve a symlink on the host, meaning the link target could not be read (e.g. the link vanished mid-walk, or a permission/IO problem). It aborts chroot building for the allocation.

Source

Thrown at client/allocdir/task_dir.go:294

				subdirs[hostEntry] = filepath.Join(dest, filepath.Base(hostEntry))
				continue
			}

			// Check if entry exists. This can happen if restarting a failed
			// task.
			if _, err := os.Lstat(taskEntry); err == nil {
				continue
			}

			if !entry.Mode().IsRegular() {
				// If it is a symlink we can create it, otherwise we skip it.
				if entry.Mode()&os.ModeSymlink == 0 {
					continue
				}

				link, err := os.Readlink(hostEntry)
				if err != nil {
					return fmt.Errorf("Couldn't resolve symlink for %v: %w", source, err)
				}

				if err := os.Symlink(link, taskEntry); err != nil {
					// Symlinking twice
					if err.(*os.LinkError).Err.Error() != "file exists" {
						return fmt.Errorf("Couldn't create symlink: %w", err)
					}
				}
				continue
			}

			uid, gid := getOwner(entry)
			if err := linkOrCopy(hostEntry, taskEntry, uid, gid, entry.Mode().Perm()); err != nil {
				return err
			}
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped underlying error (%w) to identify the failing path and errno; verify the host path exists and is a readable symlink with `ls -l` and `readlink` as the Nomad client user
2. Re-run the allocation — transient races (link removed during walk) usually succeed on retry
3. Ensure the Nomad agent runs with sufficient filesystem permissions to traverse all chroot-mapped host directories
4. Review the client's chroot map configuration and exclude volatile host directories

Example fix

// before
link, err := os.Readlink(hostEntry)
if err != nil {
    return fmt.Errorf("Couldn't resolve symlink for %v: %w", source, err)
}
// after
link, err := os.Readlink(hostEntry)
if err != nil {
    if os.IsNotExist(err) {
        continue // skip vanished symlink instead of failing the chroot
    }
    return fmt.Errorf("Couldn't resolve symlink for %v: %w", source, err)
}

// Go: pre-check the chroot source tree is traversable and links readable
func readableSymlinks(root string) error {
  return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
    if err != nil { return err }
    if d.Type()&fs.ModeSymlink != 0 {
      if _, err := os.Readlink(p); err != nil {
        return fmt.Errorf("unreadable symlink %s: %w", p, err)
      }
    }
    return nil
  })
}

func isSymlinkErr(err error) bool {
  var le *os.LinkError
  return errors.As(err, &le)
}

if err := embedDirs(...); err != nil {
  var le *os.LinkError
  if errors.As(err, &le) && os.IsNotExist(le) {
    // transient: skip/retry this entry
  } else {
    return err
  }
}

When it happens

Trigger: Building the chroot (buildChroot or recursive embedDirs) while reading host directories; os.Readlink(hostEntry) returns an error such as ENOENT (symlink removed concurrently), EACCES (no permission on parent dir), or ELOOP (symlink chain too deep).

Common situations: Host paths included in the chroot map (e.g. /etc, /usr) contain symlinks that are mutated during traversal by package managers or administrators; running the Nomad client with a user lacking read permission on the linked parent directory; deep symlink recursion on unusual filesystems.

Related errors

• Couldn't create symlink: %w
• error reading symlink: %v
• failed to resolve alloc directory: %w
• Couldn't change owner/group of %v to (uid: %v, gid: %v): %w
• Failed to mount shared directory for task: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/81aa5d387fc73d15. Report an issue: GitHub.