sgl-project/sglang · error · ValueError

byte_tensor must be a sufficiently large contiguous uint8 te

Error message

byte_tensor must be a sufficiently large contiguous uint8 tensor on cuda:{device_id}

What it means

The pool must be backed by a CUDA uint8 byte tensor that lives on the exact device (device_id), is contiguous, and has at least memory_size elements. Any mismatch (CPU tensor, wrong device index, wrong dtype, non-contiguous, or too small) fails this combined check because the pool does raw pointer arithmetic over that buffer.

Source

Thrown at python/sglang/srt/multimodal/transport/memory_pool.py:186

        transport_name: str,
        max_inflight_slices: int = DEFAULT_MAX_INFLIGHT_SLICES,
    ) -> None:
        if memory_size <= 0:
            raise ValueError("memory_size must be positive")
        if consumer_count <= 0:
            raise ValueError("consumer_count must be positive")
        if max_inflight_slices <= 0:
            raise ValueError("max_inflight_slices must be positive")
        if recycle_interval <= 0:
            raise ValueError("recycle_interval must be positive")
        if (
            not byte_tensor.is_cuda
            or byte_tensor.device.index != device_id
            or byte_tensor.dtype != torch.uint8
            or not byte_tensor.is_contiguous()
            or byte_tensor.numel() < memory_size
        ):
            raise ValueError(
                "byte_tensor must be a sufficiently large contiguous uint8 tensor "
                f"on cuda:{device_id}"
            )

        self.memory_size = memory_size
        self.byte_tensor = byte_tensor
        self.base_address = base_address
        self.device_id = device_id
        self.consumer_count = consumer_count
        self.control_words_per_slot = 1 + consumer_count
        self.max_inflight_slices = max_inflight_slices
        self.transport_name = transport_name
        control_bytes = (
            max_inflight_slices * self.control_words_per_slot * CONTROL_WORD_BYTES
        )
        self.data_start = align_up(control_bytes, DATA_ALIGNMENT)
        if memory_size <= self.data_start:
            raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate with torch.empty(memory_size, dtype=torch.uint8, device=f'cuda:{device_id}')
2. Verify byte_tensor.device.index == device_id and dtype is torch.uint8
3. Ensure the tensor is contiguous and numel >= memory_size

Example fix

# before
buf = torch.zeros(memory_size, dtype=torch.uint8)  # CPU
# after
buf = torch.zeros(memory_size, dtype=torch.uint8, device=f'cuda:{device_id}')

assert byte_tensor.is_cuda and byte_tensor.device.index == device_id
assert byte_tensor.dtype == torch.uint8 and byte_tensor.is_contiguous()
assert byte_tensor.numel() >= memory_size

def is_valid_pool_buffer(t: torch.Tensor, device_id: int, size: int) -> bool:
    return (t.is_cuda and t.device.index == device_id
            and t.dtype == torch.uint8 and t.is_contiguous()
            and t.numel() >= size)

When it happens

Trigger: Passing a torch.zeros(n, dtype=torch.uint8, device='cpu'); a tensor on cuda:1 when device_id=0; a float16 view; a strided/permuted tensor; or numel < memory_size.

Common situations: Tensor allocated before torch.cuda.set_device, device index taken from a different rank in TP, or reusing a pooled buffer from another process/device.

Related errors

• recycle_interval must be positive
• {transport_name} pool is too small after control metadata: p
• {self.transport_name} pool slot generation exhausted
• Cannot release inactive {self.transport_name} pool lease (sl
• {self.transport_name} requires a CUDA tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1f3a7de177a9f330. Report an issue: GitHub.