{"record":{"id":"fd65e0ea5c057143","repo":"sgl-project/sglang","slug":"unsupported-cute-dtype-input-dtype","errorCode":null,"errorMessage":"Unsupported cute dtype {input.dtype}","messagePattern":"Unsupported cute dtype (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/moe/flashinfer_cutedsl_moe.py","lineNumber":19,"sourceCode":"from typing import Optional\n\nimport torch\nfrom flashinfer import (\n    scaled_fp4_grouped_quantize,\n    silu_and_mul_scaled_nvfp4_experts_quantize,\n)\nfrom flashinfer.cute_dsl.blockscaled_gemm import grouped_gemm_nt_masked\n\n\ndef get_cute_dtype(input: torch.Tensor) -> str:\n    if input.dtype == torch.bfloat16:\n        return \"bfloat16\"\n    elif input.dtype == torch.float16:\n        return \"float16\"\n    elif input.dtype == torch.float32:\n        return \"float32\"\n    else:\n        raise ValueError(f\"Unsupported cute dtype {input.dtype}\")\n\n\ndef flashinfer_cutedsl_moe_masked(\n    hidden_states: tuple[torch.Tensor, Optional[torch.Tensor]],\n    input_global_scale: torch.Tensor,\n    w1: torch.Tensor,\n    w1_blockscale: torch.Tensor,\n    w1_alpha,\n    w2: torch.Tensor,\n    a2_global_scale: torch.Tensor,\n    w2_blockscale: torch.Tensor,\n    w2_alpha,\n    masked_m: torch.Tensor,\n    down_sm_count: Optional[int] = None,\n    down_signals: Optional[torch.Tensor] = None,\n    down_start_event: Optional[torch.cuda.Event] = None,\n    activation: str = \"silu\",\n):","sourceCodeStart":1,"sourceCodeEnd":37,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/moe/flashinfer_cutedsl_moe.py#L1-L37","documentation":"get_cute_dtype maps the input tensor's torch dtype to the CuteDSL type-name string, supporting only bfloat16, float16, and float32. Any other dtype (float64, fp8, int types) raises this error before the kernel launch.","triggerScenarios":"Calling flashinfer_cutedsl_moe_masked with hidden_states (or routed output) in an unsupported dtype, e.g. float64 or an fp8 tensor that wasn't cast to a supported type first.","commonSituations":"Upstream quantization config leaving hidden states in fp8; debug code creating float64 tensors; passing int logits by mistake.","solutions":["Cast inputs to bfloat16/float16 before calling the kernel: hidden_states.to(torch.bfloat16)","Check your quant path — fp8 paths must dequantize/cast before this kernel or use the dedicated fp4/fp8 wrapper","Ensure model config's dtype is bf16/fp16, not float64"],"exampleFix":"// before\nout = flashinfer_cutedsl_moe_masked((hs_fp64, routed), ...)\n// after\nout = flashinfer_cutedsl_moe_masked((hs_fp64.to(torch.bfloat16), routed), ...)","handlingStrategy":"type-guard","validationCode":"assert hidden_states[0].dtype in (torch.bfloat16, torch.float16, torch.float32), f'unsupported dtype {hidden_states[0].dtype}'","typeGuard":"def is_cute_supported(t: torch.Tensor) -> bool:\n    return t.dtype in (torch.bfloat16, torch.float16, torch.float32)","tryCatchPattern":"try:\n    out = flashinfer_cutedsl_moe_masked(hs, ...)\nexcept ValueError as e:\n    if 'Unsupported cute dtype' in str(e):\n        hs = (hs[0].to(torch.bfloat16), hs[1].to(torch.bfloat16) if hs[1] is not None else None)\n        out = flashinfer_cutedsl_moe_masked(hs, ...)\n    else:\n        raise","preventionTips":["Cast model activations to bf16/fp16 at the boundary","Check dtype in tests for every tensor fed to CuteDSL kernels"],"tags":["flashinfer","cutedsl","moe","unsupported-dtype"],"backgroundTag":"unsupported-tensor-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}