sgl-project/sglang · error · TypeError

Type must match: {self.a_dtype} != {self.b_dtype}

Error message

Type must match: {self.a_dtype} != {self.b_dtype}

What it means

Raised inside the CUTLASS kernel's __call__ when the A (activation) tensor and B (weight) tensor of the fused NVFP4 GEMM+SwiGLU+quant operation have different element dtypes. The MMA instruction requires both operands to be the same type; NVFP4 paths expect float4_e2m1fn on both sides. The check runs at kernel setup time before any tile is computed.

Source

Thrown at python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py:492

        :type stream: cuda.CUstream
        :param epilogue_op: Optional elementwise lambda function to apply to the output tensor
        :type epilogue_op: cutlass.Constexpr
        :param use_pdl: Enable Programmatic Dependent Launch.
        :type use_pdl: cutlass.Constexpr
        :raises TypeError: If input data types are incompatible with the MMA instruction.
        """
        # Setup static attributes before smem/grid/tma computation
        self.a_dtype: Type[cutlass.Numeric] = a_tensor.element_type
        self.b_dtype: Type[cutlass.Numeric] = b_tensor.element_type
        self.sf_dtype: Type[cutlass.Numeric] = sfa_tensor.element_type
        self.c_dtype: Type[cutlass.Numeric] = c_tensor.element_type
        self.a_major_mode = utils.LayoutEnum.from_tensor(a_tensor).mma_major_mode()
        self.b_major_mode = utils.LayoutEnum.from_tensor(b_tensor).mma_major_mode()
        self.c_layout = utils.LayoutEnum.from_tensor(c_tensor)

        # Check if input data types are compatible with MMA instruction
        if cutlass.const_expr(self.a_dtype != self.b_dtype):
            raise TypeError(f"Type must match: {self.a_dtype} != {self.b_dtype}")

        # Setup attributes that dependent on gemm inputs
        self._setup_attributes()

        # Setup sfa/sfb tensor by filling A/B tensor to scale factor atom layout
        # ((Atom_M, Rest_M),(Atom_K, Rest_K),RestL)
        sfa_layout = blockscaled_utils.tile_atom_to_shape_SF(
            a_tensor.shape, self.sf_vec_size
        )
        sfa_tensor = cute.make_tensor(sfa_tensor.iterator, sfa_layout)

        # ((Atom_N, Rest_N),(Atom_K, Rest_K),RestL)
        sfb_layout = blockscaled_utils.tile_atom_to_shape_SF(
            b_tensor.shape, self.sf_vec_size
        )
        sfb_tensor = cute.make_tensor(sfb_tensor.iterator, sfb_layout)

        # Determine if we need to generate scale factor C for quantization

View on GitHub (pinned to 0132848349)

Solutions

1. Verify both a and b are torch.float4_e2m1fn before calling the kernel
2. Check the weight loader / process_weights_after_loading path that repacks B into FP4
3. Ensure the same quant config (NVFP4) is applied to both activations and weights, not just one side

Example fix

// before
out = fused_op(a_fp8, b_fp4)
// after
assert a.dtype == torch.float4_e2m1fn and b.dtype == torch.float4_e2m1fn
out = fused_op(a, b)

assert a.dtype == b.dtype == torch.float4_e2m1fn, (a.dtype, b.dtype)

def is_nvfp4_pair(a, b): return a.dtype == b.dtype == torch.float4_e2m1fn

When it happens

Trigger: Calling the fused nvfp4 gemm/swiglu op with an activation tensor quantized to one dtype (e.g. float4_e2m1fn) and a weight tensor in another (e.g. uint8-packed FP4 without decode, or fp8/bf16), or passing a dequantized B while A stays FP4.

Common situations: Mixing quantization schemes between layers (e.g. w4a16 weights fed to an w4a4 kernel), incorrect weight preprocessing in process_weights_after_loading that leaves B in a different dtype, or version changes where the packed-FP4 representation changed.

Related errors

• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale
• nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input
• nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/946bf2e8643f0e1d. Report an issue: GitHub.