hashicorp/nomad · error

Failed to create task mount directory: %v

Error message

Failed to create task mount directory: %v

What it means

During TaskDir.Build in Unveil mode, Nomad creates the per-task directory under the client's mounts path (`parent = filepath.Dir(t.MountsAllocDir)`) with mode 0710 via os.MkdirAll. This error wraps a failure of that MkdirAll. It means the parent mount-point tree for the task could not be created.

Source

Thrown at client/allocdir/task_dir.go:192

		}
	}

	// Only bind mount the task alloc/task dirs to the client.mounts_dir/<task>
	if fsi == fsisolation.Unveil {
		uid, gid, _, err := dynamic.LookupUser(username)
		if err != nil {
			return fmt.Errorf("Failed to lookup user: %v", err)
		}

		nobodyUID, nobodyGID, _, err := dynamic.LookupUser("nobody")
		if err != nil {
			return fmt.Errorf("Failed to lookup nobody user: %v", err)
		}

		// create the task unique directory under the client mounts path
		parent := filepath.Dir(t.MountsAllocDir)
		if err = os.MkdirAll(parent, fileMode710); err != nil {
			return fmt.Errorf("Failed to create task mount directory: %v", err)
		}
		if err = os.Chown(parent, uid, gid); err != nil {
			return fmt.Errorf("Failed to chown task mount directory: %v", err)
		}

		// create the taskdir mount point
		if err = mountDir(t.Dir, t.MountsTaskDir, uid, gid, fileMode710); err != nil {
			return fmt.Errorf("Failed to mount task dir: %v", err)
		}

		// create the allocdir mount point (owned by nobody)
		if err = mountDir(filepath.Join(t.AllocDir, "/alloc"), t.MountsAllocDir, nobodyUID, nobodyGID, fileMode777); err != nil {
			return fmt.Errorf("Failed to mount alloc dir: %v", err)
		}

		// create the secretsdir mount point
		if err = mountDir(t.SecretsDir, t.MountsSecretsDir, uid, gid, fileMode710); err != nil {
			return fmt.Errorf("Failed to mount secrets dir: %v", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check disk space and writability of the client data_dir (`df -h`, `touch <data_dir>/probe` as the nomad user)
2. Inspect the wrapped OS error: ENOSPC/EROFS → free space or remount read-write; ENOTDIR → remove the offending file; EACCES → fix ownership of data_dir
3. Ensure the nomad client process user owns data_dir: `chown -R nomad:nomad <data_dir>`
4. Clean stale mounts state (`rm -rf <data_dir>/client/mounts/<alloc>`) after stopping the client

Example fix

// before: data_dir on read-only volume
Failed to create task mount directory: mkdir /var/nomad/client/mounts/abc: read-only file system
// after (host)
$ sudo mount -o remount,rw /var/nomad
$ sudo chown -R nomad:nomad /var/nomad

// before client start, verify data_dir writable
if err := os.MkdirAll(filepath.Join(dataDir, "client", "mounts"), 0o710); err != nil {
    log.Fatalf("data_dir not writable for mounts tree: %v", err)
}

if err := taskDir.Build(fsi, chroot, username); err != nil {
    var perr *os.PathError
    if errors.As(err, &perr) && (perr.Err == syscall.ENOSPC || perr.Err == syscall.EROFS) {
        // alert: fix data_dir volume, then retry alloc
    }
    return err
}

When it happens

Trigger: os.MkdirAll(parent, 0710) fails while TaskDir.Build runs with fsisolation.Unveil — e.g. the client data/mounts directory is missing, on a read-only filesystem, or a non-directory file already occupies a path component.

Common situations: client data_dir on a full or read-only disk; data_dir moved without the mounts subtree existing; permission mismatch where the nomad user cannot write its own data dir; leftover file where a directory is expected after an upgrade/crash.

Related errors

• failed to recover vault token from %s: %v
• plugin not executable
• failed to snapshot %s: %w
• error creating task %q dir: %w
• failed to remove alloc dir %q: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a2e5e43698c564be. Report an issue: GitHub.