{"record":{"id":"c845b997dc665ec3","repo":"sgl-project/sglang","slug":"topk-ids-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"topk_ids must be a CUDA tensor","messagePattern":"topk_ids must be a CUDA tensor","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/moe/moe_permute_prepare.py","lineNumber":57,"sourceCode":"        reorder_ids,\n        expert_offsets,\n        src2dst,\n        num_experts,\n        use_int64_offset,\n        is_ep,\n    )\n\n\ndef moe_permute_prepare(\n    topk_ids: torch.Tensor,\n    num_experts: int,\n    use_int64_offset: bool = False,\n    is_ep: bool = False,\n) -> Tuple[torch.Tensor, torch.Tensor]:\n    if topk_ids.dtype != torch.int32:\n        raise TypeError(f\"topk_ids must be int32, got {topk_ids.dtype}\")\n    if not topk_ids.is_cuda:\n        raise ValueError(\"topk_ids must be a CUDA tensor\")\n\n    sorted_topk_ids, reorder_ids = torch.sort(topk_ids.flatten())\n    offset_dtype = torch.int64 if use_int64_offset else torch.int32\n    expert_offsets = torch.empty(\n        (num_experts + 1,), dtype=offset_dtype, device=topk_ids.device\n    )\n    src2dst = torch.empty(\n        (topk_ids.numel(),), dtype=torch.int32, device=topk_ids.device\n    )\n\n    _moe_permute_prepare_out(\n        sorted_topk_ids,\n        reorder_ids,\n        expert_offsets,\n        src2dst,\n        num_experts,\n        use_int64_offset,\n        is_ep,","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/moe/moe_permute_prepare.py#L39-L75","documentation":"ValueError from moe_permute_prepare: topk_ids must be a CUDA tensor. The function immediately calls torch.sort and launches device kernels expecting GPU residency; CPU tensors would crash or silently produce host-side garbage, so they are rejected up front.","triggerScenarios":"Calling moe_permute / moe_permute_prepare with a CPU tensor for topk_ids (e.g. unit-test fixtures built without device=, or CPU-computed routing during offline tooling).","commonSituations":"Unit tests constructing routing tensors on CPU; mock/metadata-level tests that never moved tensors to GPU; offline analysis scripts reusing the kernel without a CUDA device.","solutions":["Move the tensor: topk_ids = topk_ids.to(\"cuda\") (ideally same device as the hidden states)","In tests, build tensors with device=\"cuda\" or gate the test with @unittest.skipUnless(torch.cuda.is_available(), ...)"],"exampleFix":"// before\ntopk_ids = torch.randint(0, E, (T, K))  # CPU\nexpert_offsets, ... = moe_permute(topk_ids, E)\n// after\ntopk_ids = torch.randint(0, E, (T, K), device=\"cuda\")\nexpert_offsets, ... = moe_permute(topk_ids, E)","handlingStrategy":"validation","validationCode":"if not topk_ids.is_cuda:\n    topk_ids = topk_ids.to(\"cuda\")","typeGuard":"def is_cuda_tensor(t) -> bool:\n    return isinstance(t, torch.Tensor) and t.is_cuda","tryCatchPattern":null,"preventionTips":["Create routing tensors with device='cuda' in tests","Skip GPU-kernel tests when torch.cuda.is_available() is False"],"tags":["moe","cuda","device-mismatch","validation"],"backgroundTag":"tensor-not-on-gpu","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}