sgl-project/sglang · error · ValueError

UMBPHostTensorAllocator only supports CPU host memory, got d

Error message

UMBPHostTensorAllocator only supports CPU host memory, got device={}

What it means

UMBPHostTensorAllocator.allocate only manages CPU host memory; passing any other device string is rejected with this ValueError listing the offending device.

Source

Thrown at python/sglang/srt/mem_cache/storage/umbp/umbp_host_allocator.py:54

                "or fall back to the default torch host allocator."
            ) from exc

        self._mod = umbp_mod
        self._allocator = umbp_mod.UMBPHostMemAllocator()

        self._use_hugepage = _bool_env("SGLANG_HICACHE_HOST_HUGEPAGE", True)
        self._hugepage_size = _int_env(
            "SGLANG_HICACHE_HOST_HUGEPAGE_SIZE", 2 * 1024 * 1024
        )
        self._numa_node = _int_env("SGLANG_HICACHE_HOST_NUMA_NODE", -1)
        self._prefault = _bool_env("SGLANG_HICACHE_HOST_PREFAULT", True)
        self._handles: Dict[int, Any] = {}

    def allocate(
        self, dims: tuple, dtype: torch.dtype, device: str = "cpu"
    ) -> torch.Tensor:
        if device != "cpu":
            raise ValueError(
                "UMBPHostTensorAllocator only supports CPU host memory, "
                f"got device={device}"
            )

        self.dims = dims
        self.dtype = dtype

        element_size = torch.empty((), dtype=dtype).element_size()
        nbytes = math.prod(int(dim) for dim in dims) * element_size

        requested_backing = (
            self._mod.UMBPHostBufferBacking.AnonymousHugetlb
            if self._use_hugepage
            else self._mod.UMBPHostBufferBacking.Anonymous
        )

        handle = self._allocator.alloc(
            nbytes,

View on GitHub (pinned to 0132848349)

Solutions

1. Call allocate with device='cpu' (or omit it) for host memory
2. Allocate GPU tensors with the GPU allocator, not this class

Example fix

# before
t = host_allocator.allocate((n, d), dtype, device='cuda')
# after
t = host_allocator.allocate((n, d), dtype, device='cpu')

device = device or 'cpu'
assert device == 'cpu', 'host allocator is CPU-only; use the GPU allocator for device tensors'

def is_cpu_device(device) -> bool:
    return device in ('cpu', None) or str(device).startswith('cpu')

When it happens

Trigger: Calling allocate(dims, dtype, device='cuda') or any device != 'cpu'.

Common situations: Generic tensor-allocator code reused across GPU and host pools forwards a 'cuda' device into the UMBP host allocator.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• kv-canary: {name} must be on {reference_name}'s device {refe
• num_token_non_padded and x must be on the same device
• topk_ids must be a CUDA tensor
• v_cache must be provided
• q can only be None when only_qv=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/76fe67d3846aeb96. Report an issue: GitHub.