huggingface/transformers · error · NotImplementedError

grouped_mm experts dispatch does not support activation_sche

Error message

grouped_mm experts dispatch does not support activation_scheme='static'. Use the default eager dispatch or switch to activation_scheme='dynamic'.

What it means

Same contract as the batched path, but for fp8_grouped_mm_experts_forward: the grouped-GEMM experts dispatch (matmul_grouped) quantizes activations dynamically per-block and cannot consume the pre-computed activation scales implied by activation_scheme='static'. The guard at the top of the forward raises NotImplementedError before any kernel call, telling you to either use the default eager dispatch or dynamic scheme.

Source

Thrown at src/transformers/integrations/finegrained_fp8.py:505

    # Post-mask sentinel rows: kernel left them uninitialized, so zero them out
    # before the reduction below (uninit may be NaN; NaN * 0 = NaN).
    weighted_out.masked_fill_(sentinel_mask, 0.0)

    # Accumulate results using deterministic reshape+sum instead of index_add_
    # (index_add_ with duplicate indices is non-deterministic on CUDA due to atomicAdd)
    final_hidden_states = weighted_out.view(num_tokens, num_top_k, hidden_dim).sum(dim=1)

    return final_hidden_states.to(hidden_states.dtype)


def fp8_grouped_mm_experts_forward(
    self: torch.nn.Module,
    hidden_states: torch.Tensor,
    top_k_index: torch.Tensor,
    top_k_weights: torch.Tensor,
) -> torch.Tensor:
    if self.activation_scheme == "static":
        raise NotImplementedError(
            "grouped_mm experts dispatch does not support activation_scheme='static'. "
            "Use the default eager dispatch or switch to activation_scheme='dynamic'."
        )

    finegrained_fp8 = load_finegrained_fp8_kernel()

    device = hidden_states.device
    num_top_k = top_k_index.size(-1)
    num_tokens = hidden_states.size(0)
    hidden_dim = hidden_states.size(-1)

    # S is the number of selected token-expert pairs (S = num_tokens * num_top_k)
    sample_weights = top_k_weights.reshape(-1)  # (S,)
    expert_ids = top_k_index.reshape(-1)  # (S,)

    # Sort by expert for grouped processing
    expert_ids_g, perm = torch.sort(expert_ids)
    selected_hidden_states_g = hidden_states[perm // num_top_k]

View on GitHub (pinned to a597f97485)

Solutions

Use activation_scheme='dynamic' in the quantization config when selecting grouped_mm dispatch
Keep 'static' and rely on the default eager experts dispatch
Re-quantize the checkpoint with dynamic activation scales if grouped kernels are required

Example fix

# before
quant_config = FineGrainedFP8Config(activation_scheme="static", dispatch="grouped_mm")

# after
quant_config = FineGrainedFP8Config(activation_scheme="dynamic", dispatch="grouped_mm")

Defensive patterns

Strategy: validation

Validate before calling

qc = FineGrainedFP8Config(activation_scheme="static", dispatch="grouped_mm")
assert not (qc.activation_scheme == "static" and qc.dispatch == "grouped_mm"), \
    "grouped_mm requires activation_scheme='dynamic'"

Prevention

Treat static activation scales as eager-dispatch-only in your config builders
Add a unit test asserting your shipped quantization configs never combine static schemes with fused MoE dispatch

When it happens

Trigger: Selecting dispatch='grouped_mm' (the finegrained-fp8 grouped experts implementation) on an MoE model whose fp8 config has activation_scheme='static', then running the forward.

Common situations: Trying grouped GEMM for throughput on a DeepSeek-V3-style MoE checkpoint that ships static activation scales; enabling the grouped kernel in a benchmark script over a statically quantized model.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/27162aad3a66ab02. Report an issue: GitHub.