{"record":{"id":"be461841932b30b9","repo":"vllm-project/vllm","slug":"reducescatter-is-not-supported","errorCode":null,"errorMessage":"reducescatter is not supported","messagePattern":"reducescatter is not supported","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/device_communicators/ray_communicator.py","lineNumber":239,"sourceCode":"        recv_buf: \"torch.Tensor\",\n    ):\n        raise NotImplementedError(\"allgather is not supported\")\n\n    def allreduce(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n        op: ReduceOp = ReduceOp.SUM,\n    ):\n        raise NotImplementedError(\"allreduce is not supported\")\n\n    def reducescatter(\n        self,\n        send_buf: \"torch.Tensor\",\n        recv_buf: \"torch.Tensor\",\n        op: ReduceOp = ReduceOp.SUM,\n    ):\n        raise NotImplementedError(\"reducescatter is not supported\")\n\n    @property\n    def recv_stream(self):\n        return torch.cuda.StreamContext(current_stream())\n\n    @property\n    def send_stream(self):\n        return torch.cuda.StreamContext(current_stream())\n\n    def destroy(self) -> None:\n        # Just sets a flag, vLLM manages the lifecycle of the underlying\n        # _PP GroupCoordinator.\n        self._closed = True\n\n    def get_transport_name(self) -> str:\n        return \"nccl\"\n\n    @classmethod","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/ray_communicator.py#L221-L257","documentation":"RayPPCommunicator.reducescatter unconditionally raises NotImplementedError('reducescatter is not supported'). Like allgather/allreduce, the Ray pipeline communicator provides only send/recv over the vLLM PP group, so reduce-scatter semantics are not available on this backend.","triggerScenarios":"Calling comm.reducescatter(send_buf, recv_buf, op=ReduceOp.SUM) on a RayPPCommunicator, typically from generic SPMD-style code (e.g. sequence parallelism or optimizer sharding helpers) that assumes every communicator supports collectives.","commonSituations":"Enabling a sequence-parallel / context-parallel feature on a Ray pipeline-parallel deployment; reusing a communicator abstraction across backends without capability checks.","solutions":["Issue reduce_scatter through the TP group (torch.distributed / PyNccl) instead","Or implement it manually with send/recv plus local reduction across ranks","Feature-detect and disable the code path on RayPPCommunicator"],"exampleFix":"# before\ncomm.reducescatter(send_buf, recv_buf, op=ReduceOp.SUM)  # NotImplementedError\n\n# after\ntorch.distributed.reduce_scatter_tensor(recv_buf, send_buf, group=tp_group)","handlingStrategy":"fallback","validationCode":null,"typeGuard":"def supports_reducescatter(comm) -> bool:\n    return not type(comm).__name__ == \"RayPPCommunicator\"","tryCatchPattern":"try:\n    comm.reducescatter(send_buf, recv_buf, op)\nexcept NotImplementedError:\n    torch.distributed.reduce_scatter_tensor(recv_buf, send_buf, op=op, group=tp_group)","preventionTips":["Use the process group for reduce-scatter workloads","Gate SPMD features on communicator capability checks","Write per-backend conformance tests for the communicator interface"],"tags":["ray","pipeline-parallel","collective","not-implemented"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}