vllm-project/vllm · error · NotImplementedError

only dim 0 all-gatherv is supported

Error message

only dim 0 all-gatherv is supported

What it means

XPUCommunicator.all_gatherv() implements the variable-size all-gather only along dim 0. Gathering along any other dim would require either transposes or a dim-aware gather kernel that the oneDPL/SYCL backend path here does not provide, so dim != 0 raises NotImplementedError rather than silently producing wrong layout.

Source

Thrown at vllm/distributed/device_communicators/xpu_communicator.py:118

            output_shape, dtype=input_tensor.dtype, device=input_tensor.device
        )
        if sizes is not None and sizes.count(sizes[0]) != len(sizes):
            # if inputs shape in different ranks is not the same using reduce_scatter
            input_splits = list(input_tensor.split(sizes, dim=0))
            dist.reduce_scatter(output, input_splits, group=self.device_group)
        else:
            dist.reduce_scatter_tensor(output, input_tensor, group=self.device_group)
        # Reshape before returning
        return output.movedim(0, dim).contiguous()

    def all_gatherv(
        self,
        input_: torch.Tensor | list[torch.Tensor],
        dim: int = 0,
        sizes: list[int] | None = None,
    ):
        if dim != 0:
            raise NotImplementedError("only dim 0 all-gatherv is supported")
        world_size = self.world_size

        # 'sizes' is not needed if all inputs in the same group have the same
        # shape
        if sizes is not None and all(s == sizes[0] for s in sizes):
            sizes = None

        def _all_gather_single(input_: torch.Tensor, sizes: list[int] | None = None):
            input_size = input_.size()
            if sizes is not None:
                assert len(sizes) == world_size
                assert input_.shape[dim] == sizes[self.rank_in_group], (
                    f"{input_.shape[dim]} != {sizes[self.rank_in_group]}"
                )
                output_size = (sum(sizes),) + input_size[1:]
            else:
                output_size = (input_size[0] * world_size,) + input_size[1:]
            # Allocate output tensor.

View on GitHub (pinned to c794754062)

Solutions

1. Move the gather dimension to 0 first: call all_gatherv(input_.movedim(dim, 0).contiguous(), dim=0, sizes=sizes) and movedim back afterwards
2. Restructure the caller to keep the gathered axis first (dim 0 layout) so the dim=0 implementation applies
3. If the sizes are uniform, use plain all_gather_in_place/all_gather which may support the shape you need

Example fix

# before
out = xpu_comm.all_gatherv(x, dim=1, sizes=sizes)
# NotImplementedError: only dim 0 all-gatherv is supported

# after
out = xpu_comm.all_gatherv(x.movedim(1, 0).contiguous(), dim=0, sizes=sizes).movedim(0, 1)

dim = 0 if dim == 0 else None
assert dim == 0, "XPUCommunicator.all_gatherv supports dim=0 only; move the axis first"

def xpu_all_gatherv_ok(dim: int) -> bool:
    return dim == 0

When it happens

Trigger: Calling all_gatherv(input, dim=1) or dim=-1 on an Intel GPU (XPU) device group; running model or parallelism code that was written against the PyTorch/NVSHMEM communicator which supports arbitrary dims.

Common situations: Porting a model from CUDA to Intel GPU (vLLM XPU backend) where a tensor-parallel op gathers along a non-zero dim; default code paths that pass dim explicitly instead of moving the dim afterwards.

Related errors

• asymmetric int8 activation quantization is unsupported on XP
• `{executor_backend}` does not support async scheduling yet.
• Sleep mode allocator is not available on platform {type(curr
• torch.xpu.memory is not available
• xpumem allocator extension is not available

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/67143a7de17a32ea. Report an issue: GitHub.