vllm-project/vllm · critical · RayChannelError

RayPPCommunicator has been destroyed.

Error message

RayPPCommunicator has been destroyed.

What it means

RayPPCommunicator.send checks the _closed flag before every operation; destroy() sets it. Once closed, send raises RayChannelError('RayPPCommunicator has been destroyed.') signalling the channel is gone — typically because a peer actor died and someone tore the group down, or the engine shut down while a request was still in flight.

Source

Thrown at vllm/distributed/device_communicators/ray_communicator.py:175

    def send(self, buf: "torch.Tensor", peer_rank: int) -> None:
        """
        Send a torch.Tensor to a peer.

        This returns when the send kernel has been queued, but the kernel may
        not have completed. Therefore, the caller should ensure that there are
        no concurrent writes to the sent `buf` until the send has finished.
        That is, either all writes should be submitted on the current stream
        (self._cuda_stream) or, if on a different stream, that stream should
        synchronize with the current stream.

        Args:
            buf: The torch.Tensor to send. It should already be on this
                actor's default device.
            peer_rank: The rank of the actor to send to.
        """
        if self._closed:
            raise RayChannelError("RayPPCommunicator has been destroyed.")

        assert self._comm is not None
        self._comm.send(buf, peer_rank)

    def recv(
        self,
        shape: tuple[int, ...],
        dtype: "torch.dtype",
        peer_rank: int,
        allocator: TorchTensorAllocator,
    ) -> "torch.Tensor":
        """
        Receive a torch.Tensor from a peer and synchronize the current stream.

        After this call returns, the receive buffer is safe to read from
        any stream. An RayChannelError will be raised if an error occurred
        (e.g., remote actor died), and the buffer is not safe to read.

View on GitHub (pinned to c794754062)

Solutions

1. Catch RayChannelError and treat it as a terminal channel failure: abort the in-flight request, do not retry on the same communicator
2. Find the originating dead actor in Ray logs (ray.get_actor / dashboard) and fix the underlying crash
3. Ensure no requests are in flight before calling destroy(), and re-create the communicator group after any actor restart

Example fix

# before
comm.send(buf, peer_rank)  # raises RayChannelError after peer crash

# after
try:
    comm.send(buf, peer_rank)
except RayChannelError:
    abort_request(); log("pipeline channel lost, peer died")

try:
    comm.send(buf, peer_rank)
except RayChannelError:
    # channel is terminal: abort batch, surface peer failure, never retry same comm
    abort_inflight(); raise

When it happens

Trigger: Calling send(buf, peer_rank) after comm.destroy(), or concurrently with shutdown: another worker's failure causes the coordinator to destroy the group while this actor is still sending activations to the next pipeline stage.

Common situations: One Ray actor crashes (OOM, exception) during pipeline execution and survivors keep sending; a request-in-flight racing engine shutdown; user code holding a stale communicator after restart.

Related errors

• use_communication_streams is not supported
• cuda_stream other than the current stream is not supported
• Actor {actor} not found in communicator group
• allgather is not supported
• allreduce is not supported

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/e1e498e92eaa371a. Report an issue: GitHub.