huggingface/transformers · error · ValueError

DeepGEMM experts path requires bfloat16 hidden states, got {

Error message

DeepGEMM experts path requires bfloat16 hidden states, got {hidden_states.dtype}

What it means

Error "DeepGEMM experts path requires bfloat16 hidden states, got {hidden_states.dtype}" thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:629

        (qinput_2d, _coerce_sf_for_kernel(scale_2d, is_sm100(), expected_mn=qinput_2d.size(0))),
        (weight, _coerce_sf_for_kernel(weight_scale_inv, is_sm100(), expected_mn=weight.size(0))),
        output,
        recipe=sf_recipe,
    )
    output = output.view(input.shape[:-1] + (weight.shape[0],))
    if bias is not None:
        output.add_(bias)
    return output


def deepgemm_bf16_experts_forward(
    self: torch.nn.Module,
    hidden_states: torch.Tensor,
    top_k_index: torch.Tensor,
    top_k_weights: torch.Tensor,
) -> torch.Tensor:
    if hidden_states.dtype != torch.bfloat16:
        raise ValueError(f"DeepGEMM experts path requires bfloat16 hidden states, got {hidden_states.dtype}")

    deepgemm = load_deepgemm_kernel()
    # Non-transposed weights (E, N, K) → NT kernel; transposed (E, K, N) → NN kernel.
    grouped_bf16_matmul = deepgemm.grouped_bf16_matmul_nn if self.is_transposed else deepgemm.grouped_bf16_matmul_nt

    device = hidden_states.device
    num_top_k = top_k_index.size(-1)
    num_tokens = hidden_states.size(0)
    hidden_dim = hidden_states.size(-1)

    (
        sorted_hidden,
        sorted_weights,
        expert_ids_g,
        sentinel_mask,
        perm,
        sorted_to_padded,
        grouped_layout,

View on GitHub (pinned to a597f97485)

Solutions

1. Cast hidden states to bfloat16 before the DeepGEMM experts path.

When it happens

Trigger: Raised in DeepGEMM MoE expert path when hidden states are not bfloat16.

Common situations: DeepGEMM expert kernels invoked on fp16 or fp32 hidden states; experts require bf16.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/2585aabe2878d0de. Report an issue: GitHub.