vllm-project/vllm · error · NotImplementedError

allgather is not supported

Error message

allgather is not supported

What it means

RayPPCommunicator only implements point-to-point send/recv (pipeline parallelism). The allgather method of the generic communicator interface deliberately raises NotImplementedError('allgather is not supported') because the underlying vLLM PP group over Ray has no collective broadcast/gather semantics.

Source

Thrown at vllm/distributed/device_communicators/ray_communicator.py:223

        size = torch.Size(shape)
        buf = self._comm.recv(size, dtype, src=peer_rank)

        # Buffer values are undefined if NCCL ops are aborted. Therefore, we
        # need to synchronize here and check that the channel is still
        # open to ensure that the receive buffer is valid.
        # TODO(swang): Avoid CUDA synchronization.
        current_stream().synchronize()

        if self._closed:
            raise RayChannelError("RayPPCommunicator has been destroyed.")
        return buf

    def allgather(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
    ):
        raise NotImplementedError("allgather is not supported")

    def allreduce(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
        op: ReduceOp = ReduceOp.SUM,
    ):
        raise NotImplementedError("allreduce is not supported")

    def reducescatter(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
        op: ReduceOp = ReduceOp.SUM,
    ):
        raise NotImplementedError("reducescatter is not supported")

    @property

View on GitHub (pinned to c794754062)

Solutions

1. Use send/recv point-to-point ops only with RayPPCommunicator
2. Route collective ops through torch.distributed.process_group or vLLM's TP group (PyNccl) instead
3. Restructure the algorithm (ring allgather via send/recv) if it must stay on this communicator

Example fix

# before
comm.allgather(send_buf, recv_buf)  # NotImplementedError on Ray backend

# after
# use the process-group backed collective
torch.distributed.all_gather_into_tensor(recv_buf, send_buf, group=tp_group)

if type(comm).__name__ == "RayPPCommunicator":
    raise NotImplementedError("allgather unsupported; route through torch.distributed TP group")

def supports_collectives(comm) -> bool:
    return all(callable(getattr(comm, m, None)) and not _raises(getattr(comm, m), 'allgather') for m in ())  # simplest: capability flag or isinstance check

try:
    comm.allgather(send_buf, recv_buf)
except NotImplementedError:
    torch.distributed.all_gather_into_tensor(recv_buf, send_buf, group=tp_group)

When it happens

Trigger: Calling comm.allgather(send_buf, recv_buf) on a RayPPCommunicator, e.g. generic code that switches on the interface and issues collectives for TP/EP-style coordination regardless of the concrete backend.

Common situations: Mixing pipeline-parallel Ray deployment with code paths assuming tensor-parallel collectives; porting a communicator abstraction that requires allgather for weight sync or logits gathering.

Related errors

• reducescatter is not supported
• use_communication_streams is not supported
• allreduce is not supported
• cuda_stream other than the current stream is not supported
• Actor {actor} not found in communicator group

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/3793a1049a17d805. Report an issue: GitHub.