{"record":{"id":"6f1250f11239cb5f","repo":"hashicorp/nomad","slug":"failed-to-reserve-device-s-v","errorCode":null,"errorMessage":"failed to reserve device %s: %v","messagePattern":"failed to reserve device (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/device_hook.go","lineNumber":54,"sourceCode":"func (*deviceHook) Name() string {\n\treturn HookNameDevices\n}\n\nfunc (h *deviceHook) Prestart(ctx context.Context, req *interfaces.TaskPrestartRequest, resp *interfaces.TaskPrestartResponse) error {\n\t//TODO Can the nil check be removed once the TODO in NewTaskRunner\n\t//     where this is set is addressed?\n\tif req.TaskResources == nil || len(req.TaskResources.Devices) == 0 {\n\t\tresp.Done = true\n\t\treturn nil\n\t}\n\n\t// Capture the responses\n\tvar reservations []*device.ContainerReservation\n\tfor _, req := range req.TaskResources.Devices {\n\t\t// Ask the device manager for the reservation information\n\t\tres, err := h.dm.Reserve(req)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to reserve device %s: %v\", req.ID(), err)\n\t\t}\n\n\t\treservations = append(reservations, res)\n\t}\n\n\t// Build the response\n\tfor _, res := range reservations {\n\t\tfor k, v := range res.Envs {\n\t\t\tif resp.Env == nil {\n\t\t\t\tresp.Env = make(map[string]string)\n\t\t\t}\n\n\t\t\tresp.Env[k] = v\n\t\t}\n\n\t\tfor _, m := range res.Mounts {\n\t\t\tresp.Mounts = append(resp.Mounts, convertMount(m))\n\t\t}","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/device_hook.go#L36-L72","documentation":"The device_hook Prestart step asks the Nomad device manager (plugin) to reserve the devices requested by the task (req.TaskResources.Devices). When dm.Reserve returns an error, the hook wraps it as 'failed to reserve device <id>: <reason>'. This indicates the device plugin could not satisfy or match the device request for the allocation.","triggerScenarios":"h.dm.Reserve(req) returns an error for a given device request ID: no device plugin running for the requested vendor/type/name, requested device count exceeds available devices, or the plugin failed mid-request.","commonSituations":"Job requests devices (e.g. nvidia.com/gpu) but the device plugin is not deployed or crashed; requesting more GPU/FPGA devices than physically present; device plugin version mismatch after Nomad client upgrade; fingerprint succeeded but devices are unhealthy.","solutions":["Read the wrapped %v detail from the device plugin to see the exact refusal","Verify the matching device plugin is installed, running, and fingerprinted (nomad node status -verbose)","Check that enough healthy devices are available on the node for the requested count","Confirm job device block (name/vendor/type) matches the fingerprinted devices","Restart the device plugin / nomad client and resubmit the job"],"exampleFix":"// before: job requests GPU with no plugin present\ndevices: [{ name: \"gpu\", type: \"nvidia\" }]\n// after: install/run the NVIDIA device plugin or remove the request\ndevices: [] // or deploy the device plugin first","handlingStrategy":"validation","validationCode":"// check the node can satisfy the device request before submitting\nnodeAttrs, _ := client.Nodes().Info(nodeID, nil)\nfor _, dev := range job.TaskGroups[0].Tasks[0].Resources.Devices {\n    fmt.Printf(\"verify plugin %s/%s/%s is fingerprinted and healthy\\n\",\n        dev.Vendor, dev.Type, dev.Name)\n}","typeGuard":null,"tryCatchPattern":"if err := hook.Prestart(req); err != nil {\n    if strings.Contains(err.Error(), \"failed to reserve device\") {\n        log.Printf(\"device request unsatisfiable: %v; check device plugin health\", err)\n    }\n}","preventionTips":["Deploy and verify the device plugin before scheduling device-hungry jobs","Fingerprint devices (nomad node status -verbose) to confirm availability","Keep requested device counts within node capacity","Upgrade Nomad client and device plugins in lockstep"],"tags":["devices","nomad","scheduling"],"backgroundTag":"device-reservation-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}