{"record":{"id":"717c09e24295be34","repo":"hashicorp/nomad","slug":"operation-on-unknown-device-s-s-s-s-v-v","errorCode":null,"errorMessage":"operation on unknown device(s) \"%s/%s/%s\" (%v): %v","messagePattern":"operation on unknown device\\(s\\) \"(.+?)/(.+?)/(.+?)\" \\((.+?)\\): (.+?)","errorType":"exception","errorClass":"UnknownDeviceError","httpStatus":null,"severity":"error","filePath":"client/devicemanager/utils.go","lineNumber":45,"sourceCode":"\treturn &UnknownDeviceError{\n\t\tErr:    err,\n\t\tName:   name,\n\t\tVendor: vendor, Type: devType,\n\t\tIDs: ids,\n\t}\n}\n\n// Error returns an error formatting that reveals which unknown devices were\n// requested\nfunc (u *UnknownDeviceError) Error() string {\n\treturn fmt.Sprintf(\"operation on unknown device(s) \\\"%s/%s/%s\\\" (%v): %v\",\n\t\tu.Vendor, u.Type, u.Name, u.IDs, u.Err)\n}\n\n// UnknownDeviceErrFromAllocated is a helper that returns an UnknownDeviceError\n// populating it via the AllocatedDeviceResource struct.\nfunc UnknownDeviceErrFromAllocated(err string, d *structs.AllocatedDeviceResource) *UnknownDeviceError {\n\treturn NewUnknownDeviceError(errors.New(err), d.Name, d.Vendor, d.Type, d.DeviceIDs)\n}\n\n// convertDeviceGroup converts a device group to a structs NodeDeviceResource\nfunc convertDeviceGroup(d *device.DeviceGroup) *structs.NodeDeviceResource {\n\tif d == nil {\n\t\treturn nil\n\t}\n\n\treturn &structs.NodeDeviceResource{\n\t\tVendor:     d.Vendor,\n\t\tType:       d.Type,\n\t\tName:       d.Name,\n\t\tInstances:  convertDevices(d.Devices),\n\t\tAttributes: psstructs.CopyMapStringAttribute(d.Attributes),\n\t}\n}\n\nfunc convertDevices(devs []*device.Device) []*structs.NodeDevice {","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/devicemanager/utils.go#L27-L63","documentation":"UnknownDeviceErrFromAllocated builds an UnknownDeviceError for a device listed in an allocation that the client's device manager can no longer resolve to an actual fingerprinted device (vendor/type/name/IDs unknown). It is produced by helpers like Reserve and DeviceStats when the allocated device resource references a device group absent from the node's current device inventory (e.g. a GPU that was removed or driver no longer loaded).","triggerScenarios":"Calling Reserve or DeviceStats for an allocation whose AllocatedDeviceResource points to a device group the device manager no longer has — device unplugged, driver unloaded, or fingerprint results changed since scheduling.","commonSituations":"NVIDIA/other device plugin removed or crashed after the alloc was placed; node hardware hot-removed; client restarted and device fingerprinting now returns fewer devices; stale alloc state.","solutions":["Verify the device driver/plugin is running and the device is present (re-fingerprint; restart the device plugin)","Reschedule the allocation to a node that has the device (job resubmit / client drain)","Inspect the formatted error's underlying cause (%v) for the device manager's reason; restart the Nomad client to refresh device state"],"exampleFix":"// before\ndevices, err := dm.Reserve(alloc.ID, alloc.AllocatedResources.Devices) // fails: unknown device nvidia/gpu/0\n// after\nfor _, d := range alloc.AllocatedResources.Devices {\n    if !dm.hasDevice(d.Name, d.Vendor, d.Type) {\n        return client.NotReserveable{Reason: fmt.Sprintf(\"device %s/%s/%s no longer on node; rescheduling\", d.Vendor, d.Type, d.Name)}\n    }\n}\ndevices, err := dm.Reserve(alloc.ID, alloc.AllocatedResources.Devices)","handlingStrategy":"try-catch","validationCode":"for _, d := range alloc.AllocatedResources.Devices {\n    if node.DeviceResourcesFor(d.Vendor, d.Type, d.Name) == nil {\n        return fmt.Errorf(\"device %s/%s/%s not present on node\", d.Vendor, d.Type, d.Name)\n    }\n}","typeGuard":"func deviceKnown(d *structs.AllocatedDeviceResource, fingerprinted []*structs.NodeDeviceResource) bool {\n    for _, g := range fingerprinted {\n        if g.Vendor == d.Vendor && g.Type == d.Type && g.Name == d.Name { return true }\n    }\n    return false\n}","tryCatchPattern":"devices, err := dm.Reserve(alloc.ID, alloc.AllocatedResources.Devices)\nif err != nil {\n    var ude *dmplugin.UnknownDeviceError\n    if errors.As(err, &ude) {\n        return rescheduleErr // device vanished; reschedule the alloc\n    }\n    return err\n}","preventionTips":["Monitor device plugin health so devices don't disappear post-schedule","Avoid hot-removing devices with active allocations; drain first","Re-fingerprint and reconcile after client restarts","Verify drivers (e.g. NVIDIA) load before Nomad client starts"],"tags":["devices","gpu","nomad","fingerprinting"],"backgroundTag":"device-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}