{"record":{"id":"3d28e9da55b85e65","repo":"huggingface/transformers","slug":"deepgemm-experts-selected-on-a-model-spanning-mult","errorCode":null,"errorMessage":"DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP).","messagePattern":"DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices\\. Use `experts_implementation='grouped_mm'`, or run one device per process \\(TP/EP\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepgemm.py","lineNumber":696,"sourceCode":"        perm,\n        sorted_to_padded,\n        num_tokens,\n        num_top_k,\n        hidden_dim,\n        hidden_states.dtype,\n    )\n\n\ndef deepgemm_fp8_fp4_experts_forward(\n    self: torch.nn.Module,\n    hidden_states: torch.Tensor,\n    top_k_index: torch.Tensor,\n    top_k_weights: torch.Tensor,\n) -> torch.Tensor:\n    if self._deepgemm_disabled:\n        # Set at load when the model spans >1 CUDA device in this process, where DeepGEMM's\n        # context-bound kernels corrupt across devices (see `quantizer_finegrained_fp8.py`).\n        raise RuntimeError(\n            \"DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; \"\n            \"its kernels are bound to a single CUDA context and corrupt across devices. Use \"\n            \"`experts_implementation='grouped_mm'`, or run one device per process (TP/EP).\"\n        )\n\n    # Fail before the (hub-download + JIT) load if this device can't serve these dtypes.\n    _assert_sm100_requirements(self.down_proj, self.down_proj_scale_inv)\n\n    deepgemm = load_deepgemm_kernel()\n\n    if self.activation_scheme == \"static\":\n        raise NotImplementedError(\"DeepGEMM experts dispatch does not support static activation quantization.\")\n    if hidden_states.dtype != torch.bfloat16:\n        raise ValueError(f\"DeepGEMM experts path requires bfloat16 hidden states, got {hidden_states.dtype}\")\n    grouped_fp8_fp4_matmul = (\n        deepgemm.grouped_fp8_fp4_matmul_nn if self.is_transposed else deepgemm.grouped_fp8_fp4_matmul_nt\n    )\n","sourceCodeStart":678,"sourceCodeEnd":714,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepgemm.py#L678-L714","documentation":"Error \"DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP).\" thrown in huggingface/transformers.","triggerScenarios":"Raised in DeepGEMM expert dispatch when the model spans multiple CUDA devices within one process.","commonSituations":"DeepGEMM MoE on a multi-GPU single-process device_map setup, which the single-context kernels cannot handle.","solutions":["Use `experts_implementation='grouped_mm'` for multi-device models.","Run one CUDA device per process (TP/EP) instead of spanning devices."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}