hashicorp/nomad · error
failed to reserve device
Error message
failed to reserve device %s: %v
What it means
The device_hook Prestart step asks the Nomad device manager (plugin) to reserve the devices requested by the task (req.TaskResources.Devices). When dm.Reserve returns an error, the hook wraps it as 'failed to reserve device <id>: <reason>'. This indicates the device plugin could not satisfy or match the device request for the allocation.
Solutions
- Read the wrapped %v detail from the device plugin to see the exact refusal
- Verify the matching device plugin is installed, running, and fingerprinted (nomad node status -verbose)
- Check that enough healthy devices are available on the node for the requested count
- Confirm job device block (name/vendor/type) matches the fingerprinted devices
- Restart the device plugin / nomad client and resubmit the job
Example fix
// before: job requests GPU with no plugin present
devices: [{ name: "gpu", type: "nvidia" }]
// after: install/run the NVIDIA device plugin or remove the request
devices: [] // or deploy the device plugin first Defensive patterns
Strategy: validation
Validate before calling
// check the node can satisfy the device request before submitting
nodeAttrs, _ := client.Nodes().Info(nodeID, nil)
for _, dev := range job.TaskGroups[0].Tasks[0].Resources.Devices {
fmt.Printf("verify plugin %s/%s/%s is fingerprinted and healthy\n",
dev.Vendor, dev.Type, dev.Name)
} Try / catch
if err := hook.Prestart(req); err != nil {
if strings.Contains(err.Error(), "failed to reserve device") {
log.Printf("device request unsatisfiable: %v; check device plugin health", err)
}
} Prevention
- Deploy and verify the device plugin before scheduling device-hungry jobs
- Fingerprint devices (nomad node status -verbose) to confirm availability
- Keep requested device counts within node capacity
- Upgrade Nomad client and device plugins in lockstep
When it happens
Trigger: h.dm.Reserve(req) returns an error for a given device request ID: no device plugin running for the requested vendor/type/name, requested device count exceeds available devices, or the plugin failed mid-request.
Common situations: Job requests devices (e.g. nvidia.com/gpu) but the device plugin is not deployed or crashed; requesting more GPU/FPGA devices than physically present; device plugin version mismatch after Nomad client upgrade; fingerprint succeeded but devices are unhealthy.
Related errors
- device name must be given as one of the following: type…
- dynamic port selection failed
- host path must be set in configuration for devices
- Job datacenter must be non-empty string
- Missing job datacenters
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/6f1250f11239cb5f.
Report an issue: GitHub.
Appendix: source
Thrown at client/allocrunner/taskrunner/device_hook.go:54
func (*deviceHook) Name() string {
return HookNameDevices
}
func (h *deviceHook) Prestart(ctx context.Context, req *interfaces.TaskPrestartRequest, resp *interfaces.TaskPrestartResponse) error {
//TODO Can the nil check be removed once the TODO in NewTaskRunner
// where this is set is addressed?
if req.TaskResources == nil || len(req.TaskResources.Devices) == 0 {
resp.Done = true
return nil
}
// Capture the responses
var reservations []*device.ContainerReservation
for _, req := range req.TaskResources.Devices {
// Ask the device manager for the reservation information
res, err := h.dm.Reserve(req)
if err != nil {
return fmt.Errorf("failed to reserve device %s: %v", req.ID(), err)
}
reservations = append(reservations, res)
}
// Build the response
for _, res := range reservations {
for k, v := range res.Envs {
if resp.Env == nil {
resp.Env = make(map[string]string)
}
resp.Env[k] = v
}
for _, m := range res.Mounts {
resp.Mounts = append(resp.Mounts, convertMount(m))
}View on GitHub (pinned to 482b49bf1a)