sgl-project/sglang · error · TypeError

Input 'data' must be a torch.Tensor, but got {type}

Error message

Input 'data' must be a torch.Tensor, but got {type}

What it means

The CUDA IPC consumer-side handle constructor demands both data and info_data be torch.Tensor objects; anything else (numpy arrays, cupy, storage objects) fails this TypeError immediately. IPC handles wrap raw CUDA tensors shared across processes, so no implicit conversion is attempted.

Source

Thrown at python/sglang/srt/multimodal/transport/cuda_ipc.py:194

    or device-wide synchronization.
    """

    def __init__(
        self,
        data: torch.Tensor,
        info_data: torch.Tensor,
        pool_ipc_handle,
        pool_byte_offset: int,
        ready_byte_offset: int,
        ack_byte_offset: int,
        generation: int,
        total_consumer_count: int,
        use_pool_handle_cache: bool,
    ):
        if (not isinstance(data, torch.Tensor)) or (
            not isinstance(info_data, torch.Tensor)
        ):
            raise TypeError(
                f"Input 'data' must be a torch.Tensor, but got {type(data)}"
            )

        self._init_stream_ordered_consumer(
            ready_byte_offset=ready_byte_offset,
            ack_byte_offset=ack_byte_offset,
            generation=generation,
            total_consumer_count=total_consumer_count,
            transport_name="CUDA IPC",
        )

        self.proxy_state = {
            "ipc_extra": {
                "pool_handle": pool_ipc_handle,
                "pool_byte_offset": pool_byte_offset,
                "shape": data.shape,
                "dtype": data.dtype,
                "stride": data.stride(),

View on GitHub (pinned to 0132848349)

Solutions

1. Wrap buffers in torch.Tensor before constructing the handle: torch.frombuffer / tensor.view(torch.uint8)
2. Ensure CUDA tensors, not CPU numpy — the transport requires device tensors
3. Keep info_data as a uint8 CUDA tensor per the transport contract

Example fix

# before
handle = ConsumerHandle(data=np_array, info_data=info_tensor, ...)
# after
data = torch.frombuffer(np_array.get(), dtype=np.uint8).cuda()
handle = ConsumerHandle(data=data, info_data=info_tensor, ...)

assert isinstance(data, torch.Tensor) and data.is_cuda, 'data must be a CUDA tensor'

import torch
def is_cuda_tensor(x) -> bool:
    return isinstance(x, torch.Tensor) and x.is_cuda

When it happens

Trigger: Constructing the IPC consumer with data as np.ndarray, a UntypedStorage, or a cupy array — even if info_data is a tensor, either failing isinstance triggers the raise.

Common situations: Interoperability layers passing numpy-backed multimodal embeddings; refactors that switched internal buffers from tensors to other array types; a storage handle opened from bytes not yet wrapped via torch.Tensor._from_storage.

Related errors

• total_pool_size must be positive
• tokenizer_worker_num must be positive
• {transport_name} consumer rank {rank} is outside [0, {total_
• total_consumer_count must be positive
• {self.transport_name} acknowledgements support one consumer

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7930c86556268a7a. Report an issue: GitHub.