vllm-project/vllm · error · NotImplementedError

reducescatter is not supported

Error message

reducescatter is not supported

What it means

RayPPCommunicator.reducescatter unconditionally raises NotImplementedError('reducescatter is not supported'). Like allgather/allreduce, the Ray pipeline communicator provides only send/recv over the vLLM PP group, so reduce-scatter semantics are not available on this backend.

Source

Thrown at vllm/distributed/device_communicators/ray_communicator.py:239

        recv_buf: "torch.Tensor",
    ):
        raise NotImplementedError("allgather is not supported")

    def allreduce(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
        op: ReduceOp = ReduceOp.SUM,
    ):
        raise NotImplementedError("allreduce is not supported")

    def reducescatter(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
        op: ReduceOp = ReduceOp.SUM,
    ):
        raise NotImplementedError("reducescatter is not supported")

    @property
    def recv_stream(self):
        return torch.cuda.StreamContext(current_stream())

    @property
    def send_stream(self):
        return torch.cuda.StreamContext(current_stream())

    def destroy(self) -> None:
        # Just sets a flag, vLLM manages the lifecycle of the underlying
        # _PP GroupCoordinator.
        self._closed = True

    def get_transport_name(self) -> str:
        return "nccl"

    @classmethod

View on GitHub (pinned to c794754062)

Solutions

1. Issue reduce_scatter through the TP group (torch.distributed / PyNccl) instead
2. Or implement it manually with send/recv plus local reduction across ranks
3. Feature-detect and disable the code path on RayPPCommunicator

Example fix

# before
comm.reducescatter(send_buf, recv_buf, op=ReduceOp.SUM)  # NotImplementedError

# after
torch.distributed.reduce_scatter_tensor(recv_buf, send_buf, group=tp_group)

def supports_reducescatter(comm) -> bool:
    return not type(comm).__name__ == "RayPPCommunicator"

try:
    comm.reducescatter(send_buf, recv_buf, op)
except NotImplementedError:
    torch.distributed.reduce_scatter_tensor(recv_buf, send_buf, op=op, group=tp_group)

When it happens

Trigger: Calling comm.reducescatter(send_buf, recv_buf, op=ReduceOp.SUM) on a RayPPCommunicator, typically from generic SPMD-style code (e.g. sequence parallelism or optimizer sharding helpers) that assumes every communicator supports collectives.

Common situations: Enabling a sequence-parallel / context-parallel feature on a Ray pipeline-parallel deployment; reusing a communicator abstraction across backends without capability checks.

Related errors

• allgather is not supported
• use_communication_streams is not supported
• allreduce is not supported
• cuda_stream other than the current stream is not supported
• Actor {actor} not found in communicator group

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/be461841932b30b9. Report an issue: GitHub.