vllm-project/vllm · error · ValueError

Unsupported dynamic dimensions {dims} for argument {k} with

Error message

Unsupported dynamic dimensions {dims} for argument {k} with type {type(arg)}.

What it means

When marking dynamic dimensions per forward argument, vLLM only knows how to handle torch.Tensor arguments and IntermediateTensors containers (marking dim 0 / given dims, normalizing negative dims). If dynamic_arg_dims (or mark_unbacked_dims) names an argument that is neither, it raises ValueError because there is no defined semantics for marking dims on that type.

Source

Thrown at vllm/compilation/decorators.py:482

            if arg is not None:
                dims = list(dim_to_shape_id.keys())

                if isinstance(arg, torch.Tensor):
                    dim_shape_pairs = [
                        (arg.ndim + d if d < 0 else d, dim_to_shape_id.get(d))
                        for d in dims
                    ]
                    mark_dynamic(arg, dim_shape_pairs)
                elif isinstance(arg, IntermediateTensors):
                    for tensor in arg.tensors.values():
                        dim_shape_pairs = [
                            (tensor.ndim + d if d < 0 else d, dim_to_shape_id.get(d))
                            for d in dims
                        ]
                        mark_dynamic(tensor, dim_shape_pairs)
                else:
                    raise ValueError(
                        f"Unsupported dynamic dimensions {dims} "
                        f"for argument {k} with type {type(arg)}."
                    )

        if mark_unbacked_dims:
            for k, dims_val in mark_unbacked_dims.items():
                arg = bound_args.arguments.get(k)
                if arg is not None:
                    dims = [dims_val] if isinstance(dims_val, int) else list(dims_val)
                    if isinstance(arg, torch.Tensor):
                        dims = [arg.ndim + d if d < 0 else d for d in dims]
                        if is_torch_equal_or_newer("2.10.0"):
                            for dim in dims:
                                torch._dynamo.decorators.mark_unbacked(
                                    arg, dim, hint_override=arg.size()[dim]
                                )
                        else:
                            torch._dynamo.decorators.mark_unbacked(arg, dims)

View on GitHub (pinned to c794754062)

Solutions

1. Remove the entry for the non-tensor argument from dynamic_arg_dims / mark_unbacked_dims.
2. Change the forward parameter type to torch.Tensor or IntermediateTensors if it genuinely holds tensors.
3. For containers, wrap them in IntermediateTensors or mark dims on the individual tensor parameters instead.

Example fix

# before
@support_torch_compile(dynamic_arg_dims={"kv_cache": 0})
class L(nn.Module):
    def forward(self, hidden_states, kv_cache: list): ...
# after
@support_torch_compile(dynamic_arg_dims={"hidden_states": 0})
class L(nn.Module):
    def forward(self, hidden_states, kv_cache: list): ...

import inspect, torch
from vllm.distributed import IntermediateTensors

def valid_dynamic_arg_dims(cls, dims: dict) -> bool:
    hints = {k: v.annotation for k, v in inspect.signature(cls.forward).parameters.items()}
    return all(hints.get(k) in (torch.Tensor, torch.Tensor | None, IntermediateTensors, IntermediateTensors | None) for k in dims)

When it happens

Trigger: Passing dynamic_arg_dims={'attn_mask': 0} (or mark_unbacked_dims) where the forward parameter attn_mask is e.g. a list, tuple, dataclass, or plain object rather than torch.Tensor or IntermediateTensors; also when the argument is None at call time through the tensor branches is fine, but non-tensor types hit the else branch.

Common situations: Model forwards that take lists of tensors or custom input dataclasses; copying dynamic_arg_dims maps from multimodal models with different argument types.

Related errors

• No dynamic dimensions found in the forward method of {cls}.
• Argument {k} not found in the forward method of {cls}
• shape_id='{shape_id}' requires PyTorch >= 2.11.0
• vLLM failed to compile the model. The most likely reason for
• decorated class should have a forward method.

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/16afd88a90874f4c. Report an issue: GitHub.