huggingface/transformers · error · RuntimeError

DeepGEMM experts selected on a model spanning multiple CUDA

Error message

DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP).

What it means

Error "DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP)." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:696

        perm,
        sorted_to_padded,
        num_tokens,
        num_top_k,
        hidden_dim,
        hidden_states.dtype,
    )


def deepgemm_fp8_fp4_experts_forward(
    self: torch.nn.Module,
    hidden_states: torch.Tensor,
    top_k_index: torch.Tensor,
    top_k_weights: torch.Tensor,
) -> torch.Tensor:
    if self._deepgemm_disabled:
        # Set at load when the model spans >1 CUDA device in this process, where DeepGEMM's
        # context-bound kernels corrupt across devices (see `quantizer_finegrained_fp8.py`).
        raise RuntimeError(
            "DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; "
            "its kernels are bound to a single CUDA context and corrupt across devices. Use "
            "`experts_implementation='grouped_mm'`, or run one device per process (TP/EP)."
        )

    # Fail before the (hub-download + JIT) load if this device can't serve these dtypes.
    _assert_sm100_requirements(self.down_proj, self.down_proj_scale_inv)

    deepgemm = load_deepgemm_kernel()

    if self.activation_scheme == "static":
        raise NotImplementedError("DeepGEMM experts dispatch does not support static activation quantization.")
    if hidden_states.dtype != torch.bfloat16:
        raise ValueError(f"DeepGEMM experts path requires bfloat16 hidden states, got {hidden_states.dtype}")
    grouped_fp8_fp4_matmul = (
        deepgemm.grouped_fp8_fp4_matmul_nn if self.is_transposed else deepgemm.grouped_fp8_fp4_matmul_nt
    )

View on GitHub (pinned to a597f97485)

Solutions

1. Use `experts_implementation='grouped_mm'` for multi-device models.
2. Run one CUDA device per process (TP/EP) instead of spanning devices.

When it happens

Trigger: Raised in DeepGEMM expert dispatch when the model spans multiple CUDA devices within one process.

Common situations: DeepGEMM MoE on a multi-GPU single-process device_map setup, which the single-context kernels cannot handle.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/3d28e9da55b85e65. Report an issue: GitHub.