{"record":{"id":"403b19222200795e","repo":"vllm-project/vllm","slug":"allreduce-is-not-supported","errorCode":null,"errorMessage":"allreduce is not supported","messagePattern":"allreduce is not supported","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/device_communicators/ray_communicator.py","lineNumber":231,"sourceCode":"\n        if self._closed:\n            raise RayChannelError(\"RayPPCommunicator has been destroyed.\")\n        return buf\n\n    def allgather(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n    ):\n        raise NotImplementedError(\"allgather is not supported\")\n\n    def allreduce(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n        op: ReduceOp = ReduceOp.SUM,\n    ):\n        raise NotImplementedError(\"allreduce is not supported\")\n\n    def reducescatter(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n        op: ReduceOp = ReduceOp.SUM,\n    ):\n        raise NotImplementedError(\"reducescatter is not supported\")\n\n    @property\n    def recv_stream(self):\n        return torch.cuda.StreamContext(current_stream())\n\n    @property\n    def send_stream(self):\n        return torch.cuda.StreamContext(current_stream())\n\n    def destroy(self) -> None:","sourceCodeStart":213,"sourceCodeEnd":249,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/ray_communicator.py#L213-L249","documentation":"RayPPCommunicator.allreduce unconditionally raises NotImplementedError('allreduce is not supported'). The Ray pipeline-parallel communicator wraps only send/recv on the vLLM PP group; there is no allreduce implementation behind it, even though the method exists to satisfy the communicator interface (including the op: ReduceOp = ReduceOp.SUM parameter).","triggerScenarios":"Calling comm.allreduce(send_buf, recv_buf, op=...) on a RayPPCommunicator — e.g. generic worker code attempting gradient or logits averaging through the device communicator interface.","commonSituations":"Sharing communicator-handling code between TP and PP deployments; a framework feature (e.g. sync weights or logprobs averaging) that calls allreduce on whichever communicator is present.","solutions":["Perform allreduce through the tensor-parallel group's PyNccl/custom allreduce instead of the Ray PP communicator","Emulate with point-to-point send/recv plus local reduction if ranks are few","Skip the feature when the communicator is RayPPCommunicator (feature-detect via hasattr/try)"],"exampleFix":"# before\ncomm.allreduce(send_buf, recv_buf)  # NotImplementedError\n\n# after\ntorch.distributed.all_reduce(send_buf, op=ReduceOp.SUM, group=tp_group)\nrecv_buf.copy_(send_buf)","handlingStrategy":"fallback","validationCode":null,"typeGuard":"def supports_allreduce(comm) -> bool:\n    return not type(comm).__name__ == \"RayPPCommunicator\"","tryCatchPattern":"try:\n    comm.allreduce(send_buf, recv_buf, op)\nexcept NotImplementedError:\n    torch.distributed.all_reduce(send_buf, op=op, group=tp_group)\n    recv_buf.copy_(send_buf)","preventionTips":["Send collectives to the TP group, not the PP/Ray communicator","Feature-detect before calling optional interface methods","Document which ops each communicator implements"],"tags":["ray","pipeline-parallel","allreduce","not-implemented"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}