hashicorp/nomad · error · UnknownDeviceError

operation on unknown device(s) "%s/%s/%s" (%v): %v

Error message

operation on unknown device(s) "%s/%s/%s" (%v): %v

What it means

UnknownDeviceErrFromAllocated builds an UnknownDeviceError for a device listed in an allocation that the client's device manager can no longer resolve to an actual fingerprinted device (vendor/type/name/IDs unknown). It is produced by helpers like Reserve and DeviceStats when the allocated device resource references a device group absent from the node's current device inventory (e.g. a GPU that was removed or driver no longer loaded).

Source

Thrown at client/devicemanager/utils.go:45

	return &UnknownDeviceError{
		Err:    err,
		Name:   name,
		Vendor: vendor, Type: devType,
		IDs: ids,
	}
}

// Error returns an error formatting that reveals which unknown devices were
// requested
func (u *UnknownDeviceError) Error() string {
	return fmt.Sprintf("operation on unknown device(s) \"%s/%s/%s\" (%v): %v",
		u.Vendor, u.Type, u.Name, u.IDs, u.Err)
}

// UnknownDeviceErrFromAllocated is a helper that returns an UnknownDeviceError
// populating it via the AllocatedDeviceResource struct.
func UnknownDeviceErrFromAllocated(err string, d *structs.AllocatedDeviceResource) *UnknownDeviceError {
	return NewUnknownDeviceError(errors.New(err), d.Name, d.Vendor, d.Type, d.DeviceIDs)
}

// convertDeviceGroup converts a device group to a structs NodeDeviceResource
func convertDeviceGroup(d *device.DeviceGroup) *structs.NodeDeviceResource {
	if d == nil {
		return nil
	}

	return &structs.NodeDeviceResource{
		Vendor:     d.Vendor,
		Type:       d.Type,
		Name:       d.Name,
		Instances:  convertDevices(d.Devices),
		Attributes: psstructs.CopyMapStringAttribute(d.Attributes),
	}
}

func convertDevices(devs []*device.Device) []*structs.NodeDevice {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the device driver/plugin is running and the device is present (re-fingerprint; restart the device plugin)
2. Reschedule the allocation to a node that has the device (job resubmit / client drain)
3. Inspect the formatted error's underlying cause (%v) for the device manager's reason; restart the Nomad client to refresh device state

Example fix

// before
devices, err := dm.Reserve(alloc.ID, alloc.AllocatedResources.Devices) // fails: unknown device nvidia/gpu/0
// after
for _, d := range alloc.AllocatedResources.Devices {
    if !dm.hasDevice(d.Name, d.Vendor, d.Type) {
        return client.NotReserveable{Reason: fmt.Sprintf("device %s/%s/%s no longer on node; rescheduling", d.Vendor, d.Type, d.Name)}
    }
}
devices, err := dm.Reserve(alloc.ID, alloc.AllocatedResources.Devices)

for _, d := range alloc.AllocatedResources.Devices {
    if node.DeviceResourcesFor(d.Vendor, d.Type, d.Name) == nil {
        return fmt.Errorf("device %s/%s/%s not present on node", d.Vendor, d.Type, d.Name)
    }
}

func deviceKnown(d *structs.AllocatedDeviceResource, fingerprinted []*structs.NodeDeviceResource) bool {
    for _, g := range fingerprinted {
        if g.Vendor == d.Vendor && g.Type == d.Type && g.Name == d.Name { return true }
    }
    return false
}

devices, err := dm.Reserve(alloc.ID, alloc.AllocatedResources.Devices)
if err != nil {
    var ude *dmplugin.UnknownDeviceError
    if errors.As(err, &ude) {
        return rescheduleErr // device vanished; reschedule the alloc
    }
    return err
}

When it happens

Trigger: Calling Reserve or DeviceStats for an allocation whose AllocatedDeviceResource points to a device group the device manager no longer has — device unplugged, driver unloaded, or fingerprint results changed since scheduling.

Common situations: NVIDIA/other device plugin removed or crashed after the alloc was placed; node hardware hot-removed; client restarted and device fingerprinting now returns fewer devices; stale alloc state.

Related errors

• device name must be given as one of the following: type, ven
• missing accessor ID
• failed to reserve device %s: %v
• devices updates already batched
• host path must be set in configuration for devices

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/717c09e24295be34. Report an issue: GitHub.