What causes "allreduce is not supported" in vllm-project/vllm?

Perform allreduce through the tensor-parallel group's PyNccl/custom allreduce instead of the Ray PP communicator

vllm-project/vllm · error · NotImplementedError

allreduce is not supported

Error message

allreduce is not supported

What it means

RayPPCommunicator.allreduce unconditionally raises NotImplementedError('allreduce is not supported'). The Ray pipeline-parallel communicator wraps only send/recv on the vLLM PP group; there is no allreduce implementation behind it, even though the method exists to satisfy the communicator interface (including the op: ReduceOp = ReduceOp.SUM parameter).

Source

Thrown at vllm/distributed/device_communicators/ray_communicator.py:231

        if self._closed:
            raise RayChannelError("RayPPCommunicator has been destroyed.")
        return buf

    def allgather(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
    ):
        raise NotImplementedError("allgather is not supported")

    def allreduce(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
        op: ReduceOp = ReduceOp.SUM,
    ):
        raise NotImplementedError("allreduce is not supported")

    def reducescatter(
        self,
        send_buf: "torch.Tensor",
        recv_buf: "torch.Tensor",
        op: ReduceOp = ReduceOp.SUM,
    ):
        raise NotImplementedError("reducescatter is not supported")

    @property
    def recv_stream(self):
        return torch.cuda.StreamContext(current_stream())

    @property
    def send_stream(self):
        return torch.cuda.StreamContext(current_stream())

    def destroy(self) -> None:

View on GitHub (pinned to c794754062)

Solutions

Perform allreduce through the tensor-parallel group's PyNccl/custom allreduce instead of the Ray PP communicator
Emulate with point-to-point send/recv plus local reduction if ranks are few
Skip the feature when the communicator is RayPPCommunicator (feature-detect via hasattr/try)

Example fix

# before
comm.allreduce(send_buf, recv_buf)  # NotImplementedError

# after
torch.distributed.all_reduce(send_buf, op=ReduceOp.SUM, group=tp_group)
recv_buf.copy_(send_buf)

Defensive patterns

Strategy: fallback

Type guard

def supports_allreduce(comm) -> bool:
    return not type(comm).__name__ == "RayPPCommunicator"

Try / catch

try:
    comm.allreduce(send_buf, recv_buf, op)
except NotImplementedError:
    torch.distributed.all_reduce(send_buf, op=op, group=tp_group)
    recv_buf.copy_(send_buf)

Prevention

Send collectives to the TP group, not the PP/Ray communicator
Feature-detect before calling optional interface methods
Document which ops each communicator implements

When it happens

Trigger: Calling comm.allreduce(send_buf, recv_buf, op=...) on a RayPPCommunicator — e.g. generic worker code attempting gradient or logits averaging through the device communicator interface.

Common situations: Sharing communicator-handling code between TP and PP deployments; a framework feature (e.g. sync weights or logprobs averaging) that calls allreduce on whichever communicator is present.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/403b19222200795e. Report an issue: GitHub.