sgl-project/sglang · error · ValueError

topk_ids must be a CUDA tensor

Error message

topk_ids must be a CUDA tensor

What it means

ValueError from moe_permute_prepare: topk_ids must be a CUDA tensor. The function immediately calls torch.sort and launches device kernels expecting GPU residency; CPU tensors would crash or silently produce host-side garbage, so they are rejected up front.

Source

Thrown at python/sglang/kernels/ops/moe/moe_permute_prepare.py:57

        reorder_ids,
        expert_offsets,
        src2dst,
        num_experts,
        use_int64_offset,
        is_ep,
    )


def moe_permute_prepare(
    topk_ids: torch.Tensor,
    num_experts: int,
    use_int64_offset: bool = False,
    is_ep: bool = False,
) -> Tuple[torch.Tensor, torch.Tensor]:
    if topk_ids.dtype != torch.int32:
        raise TypeError(f"topk_ids must be int32, got {topk_ids.dtype}")
    if not topk_ids.is_cuda:
        raise ValueError("topk_ids must be a CUDA tensor")

    sorted_topk_ids, reorder_ids = torch.sort(topk_ids.flatten())
    offset_dtype = torch.int64 if use_int64_offset else torch.int32
    expert_offsets = torch.empty(
        (num_experts + 1,), dtype=offset_dtype, device=topk_ids.device
    )
    src2dst = torch.empty(
        (topk_ids.numel(),), dtype=torch.int32, device=topk_ids.device
    )

    _moe_permute_prepare_out(
        sorted_topk_ids,
        reorder_ids,
        expert_offsets,
        src2dst,
        num_experts,
        use_int64_offset,
        is_ep,

View on GitHub (pinned to 0132848349)

Solutions

1. Move the tensor: topk_ids = topk_ids.to("cuda") (ideally same device as the hidden states)
2. In tests, build tensors with device="cuda" or gate the test with @unittest.skipUnless(torch.cuda.is_available(), ...)

Example fix

// before
topk_ids = torch.randint(0, E, (T, K))  # CPU
expert_offsets, ... = moe_permute(topk_ids, E)
// after
topk_ids = torch.randint(0, E, (T, K), device="cuda")
expert_offsets, ... = moe_permute(topk_ids, E)

if not topk_ids.is_cuda:
    topk_ids = topk_ids.to("cuda")

def is_cuda_tensor(t) -> bool:
    return isinstance(t, torch.Tensor) and t.is_cuda

When it happens

Trigger: Calling moe_permute / moe_permute_prepare with a CPU tensor for topk_ids (e.g. unit-test fixtures built without device=, or CPU-computed routing during offline tooling).

Common situations: Unit tests constructing routing tensors on CPU; mock/metadata-level tests that never moved tensors to GPU; offline analysis scripts reusing the kernel without a CUDA device.

Related errors

• num_token_non_padded and x must be on the same device
• {name}_block_cnt and {name}_block_idx must be on the same de
• Invalid arch format: {arch_str}
• indices must be on q's device {device}, got {indices.device}
• topk_length must be a CUDA tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c845b997dc665ec3. Report an issue: GitHub.