{"record":{"id":"3793a1049a17d805","repo":"vllm-project/vllm","slug":"allgather-is-not-supported","errorCode":null,"errorMessage":"allgather is not supported","messagePattern":"allgather is not supported","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/device_communicators/ray_communicator.py","lineNumber":223,"sourceCode":"        size = torch.Size(shape)\n        buf = self._comm.recv(size, dtype, src=peer_rank)\n\n        # Buffer values are undefined if NCCL ops are aborted. Therefore, we\n        # need to synchronize here and check that the channel is still\n        # open to ensure that the receive buffer is valid.\n        # TODO(swang): Avoid CUDA synchronization.\n        current_stream().synchronize()\n\n        if self._closed:\n            raise RayChannelError(\"RayPPCommunicator has been destroyed.\")\n        return buf\n\n    def allgather(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n    ):\n        raise NotImplementedError(\"allgather is not supported\")\n\n    def allreduce(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n        op: ReduceOp = ReduceOp.SUM,\n    ):\n        raise NotImplementedError(\"allreduce is not supported\")\n\n    def reducescatter(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n        op: ReduceOp = ReduceOp.SUM,\n    ):\n        raise NotImplementedError(\"reducescatter is not supported\")\n\n    @property","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/ray_communicator.py#L205-L241","documentation":"RayPPCommunicator only implements point-to-point send/recv (pipeline parallelism). The allgather method of the generic communicator interface deliberately raises NotImplementedError('allgather is not supported') because the underlying vLLM PP group over Ray has no collective broadcast/gather semantics.","triggerScenarios":"Calling comm.allgather(send_buf, recv_buf) on a RayPPCommunicator, e.g. generic code that switches on the interface and issues collectives for TP/EP-style coordination regardless of the concrete backend.","commonSituations":"Mixing pipeline-parallel Ray deployment with code paths assuming tensor-parallel collectives; porting a communicator abstraction that requires allgather for weight sync or logits gathering.","solutions":["Use send/recv point-to-point ops only with RayPPCommunicator","Route collective ops through torch.distributed.process_group or vLLM's TP group (PyNccl) instead","Restructure the algorithm (ring allgather via send/recv) if it must stay on this communicator"],"exampleFix":"# before\ncomm.allgather(send_buf, recv_buf)  # NotImplementedError on Ray backend\n\n# after\n# use the process-group backed collective\ntorch.distributed.all_gather_into_tensor(recv_buf, send_buf, group=tp_group)","handlingStrategy":"fallback","validationCode":"if type(comm).__name__ == \"RayPPCommunicator\":\n    raise NotImplementedError(\"allgather unsupported; route through torch.distributed TP group\")","typeGuard":"def supports_collectives(comm) -> bool:\n    return all(callable(getattr(comm, m, None)) and not _raises(getattr(comm, m), 'allgather') for m in ())  # simplest: capability flag or isinstance check","tryCatchPattern":"try:\n    comm.allgather(send_buf, recv_buf)\nexcept NotImplementedError:\n    torch.distributed.all_gather_into_tensor(recv_buf, send_buf, group=tp_group)","preventionTips":["Keep a backend capability matrix and branch on it","Reserve RayPPCommunicator for send/recv pipeline traffic","Add interface conformance tests per backend"],"tags":["ray","pipeline-parallel","collective","not-implemented"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}