What causes "failed to make device out for %s: %v" in hashicorp/nomad?

Check the listed HostPath exists on the Nomad client node (ls -l /dev/...) and is a device node

failed to make device out for %s: %v

Error message

failed to make device out for %s: %v

What it means

cmdDevices converts driver-specified host device paths into libcontainer device rules via devices.DeviceFromPath. This error is returned when a host path listed as a device cannot be resolved to a device file, so no rule can be made for it.

Source

Thrown at drivers/shared/executor/executor_linux_cgo.go:932

	if l.compute.TotalCompute >= MaxCPUShares {
		return int64(float64(shares) / float64(l.compute.TotalCompute) * MaxCPUShares)
	}

	return shares
}

// cmdDevices converts a list of driver.DeviceConfigs into excutor.Devices.
func cmdDevices(driverDevices []*drivers.DeviceConfig) ([]*config.Device, error) {
	if len(driverDevices) == 0 {
		return nil, nil
	}

	r := make([]*config.Device, len(driverDevices))

	for i, d := range driverDevices {
		ed, err := devices.DeviceFromPath(d.HostPath, d.Permissions)
		if err != nil {
			return nil, fmt.Errorf("failed to make device out for %s: %v", d.HostPath, err)
		}
		ed.Path = d.TaskPath
		ed.Allow = true // rules will be used to allow devices via cgroups
		r[i] = ed
	}

	return r, nil
}

var userMountToUnixMount = map[string]int{
	// Empty string maps to `rprivate` for backwards compatibility in restored
	// older tasks, where mount propagation will not be present.
	"":                                       unix.MS_PRIVATE | unix.MS_REC, // rprivate
	structs.VolumeMountPropagationPrivate:    unix.MS_PRIVATE | unix.MS_REC, // rprivate
	structs.VolumeMountPropagationHostToTask: unix.MS_SLAVE | unix.MS_REC,   // rslave
	structs.VolumeMountPropagationBidirectional: unix.MS_SHARED | unix.MS_REC, // rshared
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Check the listed HostPath exists on the Nomad client node (ls -l /dev/...) and is a device node
Fix or remove the devices stanza entry in the job/task driver config
Use node-specific constraints or device plugins so tasks only run on nodes exposing the device
Verify permissions on the device file allow the Nomad client to stat/read it

Example fix

// before
devices = [{ host_path = "/dev/nvidia0" }]
// after
# ensure device exists on node, or guard:
constraint { attribute = "${meta.gpu}"; operator = "isset" }
devices = [{ host_path = "/dev/nvidia0" }]

Defensive patterns

Strategy: validation

Validate before calling

for _, d := range devices {
    if _, err := os.Stat(d.HostPath); err != nil {
        return fmt.Errorf("device %s missing on node", d.HostPath)
    }
}

Try / catch

if err != nil && strings.Contains(err.Error(), "failed to make device out for") {
    // inspect d.HostPath in the error text and fix job devices stanza
    return fmt.Errorf("bad device config: %w", err)
}

Prevention

List device host paths exactly as they appear in /dev on each node
Use device plugins / node constraints instead of hardcoding host paths
Validate devices stanza entries in CI against the target node inventory

When it happens

Trigger: A task/driver declares devices with host paths, and devices.DeviceFromPath fails for one of them — the path does not exist, is not a device node (char/block), or stat fails (permissions).

Common situations: Typo in a device host path in the task's devices stanza; device exists on dev machines but not on the Nomad client host; a plugin/driver hardcodes device paths (e.g. /dev/nvidia*) absent on that node; device plugged in after task config written.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/10591d975037460a. Report an issue: GitHub.