sgl-project/sglang · error · TypeError

topk_ids must be int32, got {topk_ids.dtype}

Error message

topk_ids must be int32, got {topk_ids.dtype}

What it means

TypeError from moe_permute_prepare: the downstream permute/scatter Triton kernels index with 32-bit arithmetic and the buffers are allocated as int32, so topk_ids must arrive as torch.int32 exactly. Any other dtype (int64 from torch.topk, int16, etc.) is rejected before sorting.

Source

Thrown at python/sglang/kernels/ops/moe/moe_permute_prepare.py:55

    module.moe_permute_prepare(
        sorted_topk_ids,
        reorder_ids,
        expert_offsets,
        src2dst,
        num_experts,
        use_int64_offset,
        is_ep,
    )


def moe_permute_prepare(
    topk_ids: torch.Tensor,
    num_experts: int,
    use_int64_offset: bool = False,
    is_ep: bool = False,
) -> Tuple[torch.Tensor, torch.Tensor]:
    if topk_ids.dtype != torch.int32:
        raise TypeError(f"topk_ids must be int32, got {topk_ids.dtype}")
    if not topk_ids.is_cuda:
        raise ValueError("topk_ids must be a CUDA tensor")

    sorted_topk_ids, reorder_ids = torch.sort(topk_ids.flatten())
    offset_dtype = torch.int64 if use_int64_offset else torch.int32
    expert_offsets = torch.empty(
        (num_experts + 1,), dtype=offset_dtype, device=topk_ids.device
    )
    src2dst = torch.empty(
        (topk_ids.numel(),), dtype=torch.int32, device=topk_ids.device
    )

    _moe_permute_prepare_out(
        sorted_topk_ids,
        reorder_ids,
        expert_offsets,
        src2dst,
        num_experts,

View on GitHub (pinned to 0132848349)

Solutions

1. Cast before calling: topk_ids = topk_ids.to(torch.int32) (use .to(dtype=torch.int32) to share codegen)
2. Fix the router to emit int32 directly, e.g. gate_topk already stores int32 — use it instead of raw torch.topk
3. For num_experts > 2^31 (never) or huge token counts, reconsider whether int32 offsets suffice; otherwise int32 is safe

Example fix

// before
ids = torch.topk(router_logits, k, dim=-1).indices
sorted_ids, _ = moe_permute(ids, num_experts)
// after
ids = torch.topk(router_logits, k, dim=-1).indices.to(torch.int32)
sorted_ids, _ = moe_permute(ids, num_experts)

topk_ids = topk_ids.to(torch.int32) if topk_ids.dtype != torch.int32 else topk_ids

def is_int32_ids(t: torch.Tensor) -> bool:
    return t.dtype == torch.int32

When it happens

Trigger: Calling moe_permute (or moe_permute_prepare / _moe_permute_prepare_out) with topk_ids of dtype torch.int64 — the classic case, since torch.topk returns int64 indices that were never cast.

Common situations: New MoE integration where router indices come straight from torch.topk or torch.argsort; mixed codebases where older kernels accepted int64; debugging removed a .to(torch.int32) as 'unnecessary'.

Related errors

• num_token_non_padded must be an integer tensor, got {num_tok
• fuse_swiglu_interleaved set on an incompatible fused_moe cal
• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype
• sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dty

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9bcfe721bab9e071. Report an issue: GitHub.