hashicorp/nomad · error

failed to reserve device %s: %v

Error message

failed to reserve device %s: %v

What it means

The device_hook Prestart step asks the Nomad device manager (plugin) to reserve the devices requested by the task (req.TaskResources.Devices). When dm.Reserve returns an error, the hook wraps it as 'failed to reserve device <id>: <reason>'. This indicates the device plugin could not satisfy or match the device request for the allocation.

Source

Thrown at client/allocrunner/taskrunner/device_hook.go:54

func (*deviceHook) Name() string {
	return HookNameDevices
}

func (h *deviceHook) Prestart(ctx context.Context, req *interfaces.TaskPrestartRequest, resp *interfaces.TaskPrestartResponse) error {
	//TODO Can the nil check be removed once the TODO in NewTaskRunner
	//     where this is set is addressed?
	if req.TaskResources == nil || len(req.TaskResources.Devices) == 0 {
		resp.Done = true
		return nil
	}

	// Capture the responses
	var reservations []*device.ContainerReservation
	for _, req := range req.TaskResources.Devices {
		// Ask the device manager for the reservation information
		res, err := h.dm.Reserve(req)
		if err != nil {
			return fmt.Errorf("failed to reserve device %s: %v", req.ID(), err)
		}

		reservations = append(reservations, res)
	}

	// Build the response
	for _, res := range reservations {
		for k, v := range res.Envs {
			if resp.Env == nil {
				resp.Env = make(map[string]string)
			}

			resp.Env[k] = v
		}

		for _, m := range res.Mounts {
			resp.Mounts = append(resp.Mounts, convertMount(m))
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v detail from the device plugin to see the exact refusal
2. Verify the matching device plugin is installed, running, and fingerprinted (nomad node status -verbose)
3. Check that enough healthy devices are available on the node for the requested count
4. Confirm job device block (name/vendor/type) matches the fingerprinted devices
5. Restart the device plugin / nomad client and resubmit the job

Example fix

// before: job requests GPU with no plugin present
devices: [{ name: "gpu", type: "nvidia" }]
// after: install/run the NVIDIA device plugin or remove the request
devices: [] // or deploy the device plugin first

// check the node can satisfy the device request before submitting
nodeAttrs, _ := client.Nodes().Info(nodeID, nil)
for _, dev := range job.TaskGroups[0].Tasks[0].Resources.Devices {
    fmt.Printf("verify plugin %s/%s/%s is fingerprinted and healthy\n",
        dev.Vendor, dev.Type, dev.Name)
}

if err := hook.Prestart(req); err != nil {
    if strings.Contains(err.Error(), "failed to reserve device") {
        log.Printf("device request unsatisfiable: %v; check device plugin health", err)
    }
}

When it happens

Trigger: h.dm.Reserve(req) returns an error for a given device request ID: no device plugin running for the requested vendor/type/name, requested device count exceeds available devices, or the plugin failed mid-request.

Common situations: Job requests devices (e.g. nvidia.com/gpu) but the device plugin is not deployed or crashed; requesting more GPU/FPGA devices than physically present; device plugin version mismatch after Nomad client upgrade; fingerprint succeeded but devices are unhealthy.

Related errors

• operation on unknown device(s) "%s/%s/%s" (%v): %v
• volume is currently unschedulable
• device name must be given as one of the following: type, ven
• Missing job datacenters
• Job datacenter must be non-empty string

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6f1250f11239cb5f. Report an issue: GitHub.