sgl-project/sglang · error · ValueError

{self.transport_name} requires a CUDA tensor

Error message

{self.transport_name} requires a CUDA tensor

What it means

copy_tensor performs a device-to-device copy into the shared pool, so the source tensor must already be on CUDA. A CPU tensor would trigger an implicit slow H2D copy or crash, so the pool rejects it upfront.

Source

Thrown at python/sglang/srt/multimodal/transport/memory_pool.py:335

    def _recycle_loop(self) -> None:
        torch.cuda.set_device(self.device_id)
        while not self._recycler_stop_event.is_set():
            try:
                with self._lock, torch.cuda.device(self.device_id):
                    self._recycle_ready_leases_locked()
            except Exception:
                logger.warning(
                    "%s multimodal pool recycle failed",
                    self.transport_name,
                    exc_info=True,
                )
            self._recycler_stop_event.wait(self._recycle_interval)

    def copy_tensor(
        self, tensor: torch.Tensor
    ) -> tuple[Optional[PoolLease], Optional[torch.Tensor]]:
        if not tensor.is_cuda:
            raise ValueError(f"{self.transport_name} requires a CUDA tensor")
        source = tensor.contiguous()
        nbytes = source.numel() * source.element_size()
        if nbytes == 0:
            raise ValueError(f"{self.transport_name} cannot transport an empty tensor")
        with self._lock:
            lease = self._allocate_locked(nbytes)
        if lease is None:
            return None, None

        try:
            with torch.cuda.device(self.device_id):
                destination = self.byte_tensor[lease.start : lease.start + lease.nbytes]
                destination.copy_(
                    source.view(torch.uint8).reshape(-1), non_blocking=True
                )
                stream_write_value32(
                    self.device_id,
                    self.base_address + lease.ready_byte_offset,

View on GitHub (pinned to 0132848349)

Solutions

1. Call tensor = tensor.to(device, non_blocking=True) (with a sync if needed) before copy_tensor
2. Ensure the preprocessing stage is configured for GPU (e.g. move image encode onto the target device)
3. Add an assert tensor.is_cuda in debug builds near the producer

Example fix

# before
lease, view = pool.copy_tensor(cpu_tensor)
# after
lease, view = pool.copy_tensor(cpu_tensor.to('cuda', non_blocking=True))
torch.cuda.synchronize()

if not tensor.is_cuda:
    tensor = tensor.to(device, non_blocking=True)
    torch.cuda.synchronize()

def ensure_cuda(t: torch.Tensor, device: str) -> torch.Tensor:
    return t if t.is_cuda else t.to(device)

When it happens

Trigger: Passing a tensor loaded on CPU (e.g. from preprocessing done on host) to wrap_tensor/copy_tensor; using .to('cpu') for debugging and forgetting to move it back.

Common situations: Multimodal preprocessing pipelines that produce CPU tensors (PIL/numpy origins); mixed-device code across ranks.

Related errors

• byte_tensor must be a sufficiently large contiguous uint8 te
• --mm-feature-transport=cuda_ipc requires NVIDIA CUDA.
• A CUDA VMM-enabled model must provide a multimodal processor
• CUDA VMM feature transport requires each feature field to co
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ffbed2bb70a79900. Report an issue: GitHub.