huggingface/transformers · error · NotImplementedError

DeepGEMM experts dispatch does not support static activation

Error message

DeepGEMM experts dispatch does not support static activation quantization.

What it means

Error "DeepGEMM experts dispatch does not support static activation quantization." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:708

    top_k_index: torch.Tensor,
    top_k_weights: torch.Tensor,
) -> torch.Tensor:
    if self._deepgemm_disabled:
        # Set at load when the model spans >1 CUDA device in this process, where DeepGEMM's
        # context-bound kernels corrupt across devices (see `quantizer_finegrained_fp8.py`).
        raise RuntimeError(
            "DeepGEMM experts selected on a model spanning multiple CUDA devices in one process; "
            "its kernels are bound to a single CUDA context and corrupt across devices. Use "
            "`experts_implementation='grouped_mm'`, or run one device per process (TP/EP)."
        )

    # Fail before the (hub-download + JIT) load if this device can't serve these dtypes.
    _assert_sm100_requirements(self.down_proj, self.down_proj_scale_inv)

    deepgemm = load_deepgemm_kernel()

    if self.activation_scheme == "static":
        raise NotImplementedError("DeepGEMM experts dispatch does not support static activation quantization.")
    if hidden_states.dtype != torch.bfloat16:
        raise ValueError(f"DeepGEMM experts path requires bfloat16 hidden states, got {hidden_states.dtype}")
    grouped_fp8_fp4_matmul = (
        deepgemm.grouped_fp8_fp4_matmul_nn if self.is_transposed else deepgemm.grouped_fp8_fp4_matmul_nt
    )

    device = hidden_states.device
    num_top_k = top_k_index.size(-1)
    num_tokens = hidden_states.size(0)
    hidden_dim = hidden_states.size(-1)

    weight_up = to_local(self.gate_up_proj if self.has_gate else self.up_proj)
    weight_scale_up = to_local(self.gate_up_proj_scale_inv if self.has_gate else self.up_proj_scale_inv)
    weight_down = to_local(self.down_proj)
    weight_scale_down = to_local(self.down_proj_scale_inv)

    cast_kwargs = _select_fp8_cast_kwargs(weight_up, weight_scale_up, self.block_size, is_sm100())
    (

View on GitHub (pinned to a597f97485)

Solutions

1. Use dynamic activation quantization for DeepGEMM experts dispatch.
2. Choose another experts dispatch that supports static quantization.

When it happens

Trigger: Raised in DeepGEMM expert dispatch when activation_scheme is 'static'.

Common situations: Static activation quantization configured for a MoE model using DeepGEMM expert dispatch.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/31bf3f31ba22d76a. Report an issue: GitHub.