sgl-project/sglang · error · ValueError

Unsupported cute dtype {input.dtype}

Error message

Unsupported cute dtype {input.dtype}

What it means

get_cute_dtype maps the input tensor's torch dtype to the CuteDSL type-name string, supporting only bfloat16, float16, and float32. Any other dtype (float64, fp8, int types) raises this error before the kernel launch.

Source

Thrown at python/sglang/srt/layers/moe/flashinfer_cutedsl_moe.py:19

from typing import Optional

import torch
from flashinfer import (
    scaled_fp4_grouped_quantize,
    silu_and_mul_scaled_nvfp4_experts_quantize,
)
from flashinfer.cute_dsl.blockscaled_gemm import grouped_gemm_nt_masked


def get_cute_dtype(input: torch.Tensor) -> str:
    if input.dtype == torch.bfloat16:
        return "bfloat16"
    elif input.dtype == torch.float16:
        return "float16"
    elif input.dtype == torch.float32:
        return "float32"
    else:
        raise ValueError(f"Unsupported cute dtype {input.dtype}")


def flashinfer_cutedsl_moe_masked(
    hidden_states: tuple[torch.Tensor, Optional[torch.Tensor]],
    input_global_scale: torch.Tensor,
    w1: torch.Tensor,
    w1_blockscale: torch.Tensor,
    w1_alpha,
    w2: torch.Tensor,
    a2_global_scale: torch.Tensor,
    w2_blockscale: torch.Tensor,
    w2_alpha,
    masked_m: torch.Tensor,
    down_sm_count: Optional[int] = None,
    down_signals: Optional[torch.Tensor] = None,
    down_start_event: Optional[torch.cuda.Event] = None,
    activation: str = "silu",
):

View on GitHub (pinned to 0132848349)

Solutions

1. Cast inputs to bfloat16/float16 before calling the kernel: hidden_states.to(torch.bfloat16)
2. Check your quant path — fp8 paths must dequantize/cast before this kernel or use the dedicated fp4/fp8 wrapper
3. Ensure model config's dtype is bf16/fp16, not float64

Example fix

// before
out = flashinfer_cutedsl_moe_masked((hs_fp64, routed), ...)
// after
out = flashinfer_cutedsl_moe_masked((hs_fp64.to(torch.bfloat16), routed), ...)

assert hidden_states[0].dtype in (torch.bfloat16, torch.float16, torch.float32), f'unsupported dtype {hidden_states[0].dtype}'

def is_cute_supported(t: torch.Tensor) -> bool:
    return t.dtype in (torch.bfloat16, torch.float16, torch.float32)

try:
    out = flashinfer_cutedsl_moe_masked(hs, ...)
except ValueError as e:
    if 'Unsupported cute dtype' in str(e):
        hs = (hs[0].to(torch.bfloat16), hs[1].to(torch.bfloat16) if hs[1] is not None else None)
        out = flashinfer_cutedsl_moe_masked(hs, ...)
    else:
        raise

When it happens

Trigger: Calling flashinfer_cutedsl_moe_masked with hidden_states (or routed output) in an unsupported dtype, e.g. float64 or an fp8 tensor that wasn't cast to a supported type first.

Common situations: Upstream quantization config leaving hidden states in fp8; debug code creating float64 tensors; passing int logits by mistake.

Related errors

• CuteDSL masked MoE supports activation 'silu' (gated) or 're
• flashinfer_cutedsl FP4 MoE only supports DeepEP low_latency
• Kimi expert-pack {role} quant type is unsupported
• Can't import trtllm_fp8_block_scale_moe from flashinfer. Ple
• Can't import trtllm_fp8_block_scale_routed_moe from flashinf

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fd65e0ea5c057143. Report an issue: GitHub.