{"record":{"id":"e1e498e92eaa371a","repo":"vllm-project/vllm","slug":"rayppcommunicator-has-been-destroyed","errorCode":null,"errorMessage":"RayPPCommunicator has been destroyed.","messagePattern":"RayPPCommunicator has been destroyed\\.","errorType":"exception","errorClass":"RayChannelError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/device_communicators/ray_communicator.py","lineNumber":175,"sourceCode":"\n    def send(self, buf: \"torch.Tensor\", peer_rank: int) -> None:\n        \"\"\"\n        Send a torch.Tensor to a peer.\n\n        This returns when the send kernel has been queued, but the kernel may\n        not have completed. Therefore, the caller should ensure that there are\n        no concurrent writes to the sent `buf` until the send has finished.\n        That is, either all writes should be submitted on the current stream\n        (self._cuda_stream) or, if on a different stream, that stream should\n        synchronize with the current stream.\n\n        Args:\n            buf: The torch.Tensor to send. It should already be on this\n                actor's default device.\n            peer_rank: The rank of the actor to send to.\n        \"\"\"\n        if self._closed:\n            raise RayChannelError(\"RayPPCommunicator has been destroyed.\")\n\n        assert self._comm is not None\n        self._comm.send(buf, peer_rank)\n\n    def recv(\n        self,\n        shape: tuple[int, ...],\n        dtype: \"torch.dtype\",\n        peer_rank: int,\n        allocator: TorchTensorAllocator,\n    ) -> \"torch.Tensor\":\n        \"\"\"\n        Receive a torch.Tensor from a peer and synchronize the current stream.\n\n        After this call returns, the receive buffer is safe to read from\n        any stream. An RayChannelError will be raised if an error occurred\n        (e.g., remote actor died), and the buffer is not safe to read.\n","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/ray_communicator.py#L157-L193","documentation":"RayPPCommunicator.send checks the _closed flag before every operation; destroy() sets it. Once closed, send raises RayChannelError('RayPPCommunicator has been destroyed.') signalling the channel is gone — typically because a peer actor died and someone tore the group down, or the engine shut down while a request was still in flight.","triggerScenarios":"Calling send(buf, peer_rank) after comm.destroy(), or concurrently with shutdown: another worker's failure causes the coordinator to destroy the group while this actor is still sending activations to the next pipeline stage.","commonSituations":"One Ray actor crashes (OOM, exception) during pipeline execution and survivors keep sending; a request-in-flight racing engine shutdown; user code holding a stale communicator after restart.","solutions":["Catch RayChannelError and treat it as a terminal channel failure: abort the in-flight request, do not retry on the same communicator","Find the originating dead actor in Ray logs (ray.get_actor / dashboard) and fix the underlying crash","Ensure no requests are in flight before calling destroy(), and re-create the communicator group after any actor restart"],"exampleFix":"# before\ncomm.send(buf, peer_rank)  # raises RayChannelError after peer crash\n\n# after\ntry:\n    comm.send(buf, peer_rank)\nexcept RayChannelError:\n    abort_request(); log(\"pipeline channel lost, peer died\")","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    comm.send(buf, peer_rank)\nexcept RayChannelError:\n    # channel is terminal: abort batch, surface peer failure, never retry same comm\n    abort_inflight(); raise","preventionTips":["Drain in-flight requests before destroy()","Watch peer actor health and stop sending once any peer dies","Treat RayChannelError as fatal, not transient"],"tags":["ray","pipeline-parallel","lifecycle","distributed"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}