hashicorp/nomad · error

Failed to mount secrets dir: %v

Error message

Failed to mount secrets dir: %v

What it means

TaskDir.Build (Unveil mode) bind-mounts the task's secrets directory into the mounts path with mode 0710 owned by the task user. This error wraps mountDir failure for the secrets dir. Secrets dir provisioning (MakeSecretsDirs, incl. size limit) precedes this, so a failure here is mount/permission related.

Source

Thrown at client/allocdir/task_dir.go:210

			return fmt.Errorf("Failed to create task mount directory: %v", err)
		}
		if err = os.Chown(parent, uid, gid); err != nil {
			return fmt.Errorf("Failed to chown task mount directory: %v", err)
		}

		// create the taskdir mount point
		if err = mountDir(t.Dir, t.MountsTaskDir, uid, gid, fileMode710); err != nil {
			return fmt.Errorf("Failed to mount task dir: %v", err)
		}

		// create the allocdir mount point (owned by nobody)
		if err = mountDir(filepath.Join(t.AllocDir, "/alloc"), t.MountsAllocDir, nobodyUID, nobodyGID, fileMode777); err != nil {
			return fmt.Errorf("Failed to mount alloc dir: %v", err)
		}

		// create the secretsdir mount point
		if err = mountDir(t.SecretsDir, t.MountsSecretsDir, uid, gid, fileMode710); err != nil {
			return fmt.Errorf("Failed to mount secrets dir: %v", err)
		}
	}

	return nil
}

// buildChroot takes a mapping of absolute directory or file paths on the host
// to their intended, relative location within the task directory. This
// attempts hardlink and then defaults to copying. If the path exists on the
// host and can't be embedded an error is returned.
func (t *TaskDir) buildChroot(entries map[string]string) error {
	return t.embedDirs(entries)
}

func (t *TaskDir) embedDirs(entries map[string]string) error {
	subdirs := make(map[string]string)
	for source, dest := range entries {
		if t.skip.Contains(source) {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run the client as root or with CAP_SYS_ADMIN
2. Check the wrapped errno; EBUSY → unmount the stale secrets mount point and retry
3. Verify <data_dir> supports bind mounts (avoid exotic FUSE/network filesystems for data_dir)
4. Restart the client to clean the mounts tree: `rm -rf <data_dir>/client/mounts/<task>` while stopped

Example fix

# before
Failed to mount secrets dir: device or resource busy
# after
$ umount /var/nomad/client/mounts/<alloc-task>/secrets
$ systemctl restart nomad

// preflight bind-mount capability and a clean mount point
if _, err := os.Stat(mountsSecretsDir); err == nil {
    exec.Command("umount", mountsSecretsDir).Run() // clear stale mount
}
if err := syscall.Mount("tmpfs-probe", probesDir, "tmpfs", 0, ""); err != nil {
    log.Fatalf("mounts unavailable (need root/CAP_SYS_ADMIN): %v", err)
}
syscall.Unmount(probesDir, 0)

if err := taskDir.Build(fsi, chroot, username); err != nil {
    if strings.Contains(err.Error(), "Failed to mount secrets dir") {
        exec.Command("umount", mountsSecretsDir).Run()
        return retryBuild()
    }
    return err
}

When it happens

Trigger: mountDir(t.SecretsDir, t.MountsSecretsDir, uid, gid, 0710) fails during TaskDir.Build(Unveil) — mount(2) error or mount-point creation/chown failure.

Common situations: Client without CAP_SYS_ADMIN (containerized); stale mount at MountsSecretsDir from previous alloc; secrets tmpfs size limit misconfigured earlier leaving the dir in a bad state; data_dir on a filesystem that disallows bind mounts.

Related errors

• Failed to mount shared directory for task: %w
• Failed to mount task dir: %v
• Failed to mount alloc dir: %v
• mount point detection failed for volume (%s): %v
• failed to build mount for resolv.conf: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/96831c8f4cff3e24. Report an issue: GitHub.