{"record":{"id":"9bcfe721bab9e071","repo":"sgl-project/sglang","slug":"topk-ids-must-be-int32-got-topk-ids-dtype","errorCode":null,"errorMessage":"topk_ids must be int32, got {topk_ids.dtype}","messagePattern":"topk_ids must be int32, got (.+?)","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/moe/moe_permute_prepare.py","lineNumber":55,"sourceCode":"    module.moe_permute_prepare(\n        sorted_topk_ids,\n        reorder_ids,\n        expert_offsets,\n        src2dst,\n        num_experts,\n        use_int64_offset,\n        is_ep,\n    )\n\n\ndef moe_permute_prepare(\n    topk_ids: torch.Tensor,\n    num_experts: int,\n    use_int64_offset: bool = False,\n    is_ep: bool = False,\n) -> Tuple[torch.Tensor, torch.Tensor]:\n    if topk_ids.dtype != torch.int32:\n        raise TypeError(f\"topk_ids must be int32, got {topk_ids.dtype}\")\n    if not topk_ids.is_cuda:\n        raise ValueError(\"topk_ids must be a CUDA tensor\")\n\n    sorted_topk_ids, reorder_ids = torch.sort(topk_ids.flatten())\n    offset_dtype = torch.int64 if use_int64_offset else torch.int32\n    expert_offsets = torch.empty(\n        (num_experts + 1,), dtype=offset_dtype, device=topk_ids.device\n    )\n    src2dst = torch.empty(\n        (topk_ids.numel(),), dtype=torch.int32, device=topk_ids.device\n    )\n\n    _moe_permute_prepare_out(\n        sorted_topk_ids,\n        reorder_ids,\n        expert_offsets,\n        src2dst,\n        num_experts,","sourceCodeStart":37,"sourceCodeEnd":73,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/moe/moe_permute_prepare.py#L37-L73","documentation":"TypeError from moe_permute_prepare: the downstream permute/scatter Triton kernels index with 32-bit arithmetic and the buffers are allocated as int32, so topk_ids must arrive as torch.int32 exactly. Any other dtype (int64 from torch.topk, int16, etc.) is rejected before sorting.","triggerScenarios":"Calling moe_permute (or moe_permute_prepare / _moe_permute_prepare_out) with topk_ids of dtype torch.int64 — the classic case, since torch.topk returns int64 indices that were never cast.","commonSituations":"New MoE integration where router indices come straight from torch.topk or torch.argsort; mixed codebases where older kernels accepted int64; debugging removed a .to(torch.int32) as 'unnecessary'.","solutions":["Cast before calling: topk_ids = topk_ids.to(torch.int32) (use .to(dtype=torch.int32) to share codegen)","Fix the router to emit int32 directly, e.g. gate_topk already stores int32 — use it instead of raw torch.topk","For num_experts > 2^31 (never) or huge token counts, reconsider whether int32 offsets suffice; otherwise int32 is safe"],"exampleFix":"// before\nids = torch.topk(router_logits, k, dim=-1).indices\nsorted_ids, _ = moe_permute(ids, num_experts)\n// after\nids = torch.topk(router_logits, k, dim=-1).indices.to(torch.int32)\nsorted_ids, _ = moe_permute(ids, num_experts)","handlingStrategy":"type-guard","validationCode":"topk_ids = topk_ids.to(torch.int32) if topk_ids.dtype != torch.int32 else topk_ids","typeGuard":"def is_int32_ids(t: torch.Tensor) -> bool:\n    return t.dtype == torch.int32","tryCatchPattern":null,"preventionTips":["Cast router indices to int32 immediately after torch.topk","Prefer emitters (gate_topk) that produce int32 natively"],"tags":["moe","dtype","int32","triton"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}