huggingface/transformers · error · NotImplementedError

batched_mm experts dispatch does not support activation_sche

Error message

batched_mm experts dispatch does not support activation_scheme='static'. Use the default eager dispatch or switch to activation_scheme='dynamic'.

What it means

fp8_batched_mm_experts_forward is one of the finegrained-fp8 MoE dispatch implementations (selected via the dispatch/implementation knob on the MoE layer). It quantizes activations on the fly, which only works with activation_scheme='dynamic'; the 'static' scheme relies on pre-computed activation scales that this batched_matmul path does not consume, so it explicitly raises NotImplementedError instead of silently producing wrong numerics.

Source

Thrown at src/transformers/integrations/finegrained_fp8.py:431

            scale_inv,
            offsets=offsets,
            tokens_per_expert=tokens_per_expert,
            block_size=self.block_size,
        )
        y = y.reshape(self.n_groups, *input_shape, -1).movedim(0, -2)
        if self.has_bias:
            y.add_(self.bias.view(self.n_groups, -1))
        return y


def fp8_batched_mm_experts_forward(
    self: torch.nn.Module,
    hidden_states: torch.Tensor,
    top_k_index: torch.Tensor,
    top_k_weights: torch.Tensor,
) -> torch.Tensor:
    if self.activation_scheme == "static":
        raise NotImplementedError(
            "batched_mm experts dispatch does not support activation_scheme='static'. "
            "Use the default eager dispatch or switch to activation_scheme='dynamic'."
        )

    finegrained_fp8 = load_finegrained_fp8_kernel()

    num_top_k = top_k_index.size(-1)
    num_tokens = hidden_states.size(0)
    hidden_dim = hidden_states.size(-1)

    # S is the number of selected tokens-experts pairs (S = num_tokens * num_top_k)
    # Replicate each token num_top_k times to align with the flattened (S,) routing tensors.
    selected_hidden_states = hidden_states.repeat_interleave(num_top_k, dim=0)
    sample_weights = top_k_weights.reshape(-1)  # (S,)
    expert_ids = top_k_index.reshape(-1)  # (S,)

    # EP sentinel handling: leave `expert_ids` unclamped — the batched kernel early-returns on
    # `expert_id >= NUM_EXPERTS`, leaving sentinel output rows uninitialized. The post-mask below

View on GitHub (pinned to a597f97485)

Solutions

Switch to activation_scheme='dynamic' in the quantization config if you want batched_mm dispatch
Keep activation_scheme='static' and use the default eager experts dispatch (do not select batched_mm)
Re-quantize the model with dynamic activation scaling if you need the batched path

Example fix

# before
quant_config = FineGrainedFP8Config(activation_scheme="static", dispatch="batched_mm")

# after
quant_config = FineGrainedFP8Config(activation_scheme="dynamic", dispatch="batched_mm")
# or: keep "static" and omit dispatch to use the default eager path

Defensive patterns

Strategy: validation

Validate before calling

qc = FineGrainedFP8Config(activation_scheme="static", dispatch="batched_mm")
assert not (qc.activation_scheme == "static" and qc.dispatch == "batched_mm"), \
    "batched_mm requires activation_scheme='dynamic'"

Prevention

Keep (dispatch, activation_scheme) pairs in a validated preset dict instead of setting fields independently
Remember: both batched_mm and grouped_mm experts dispatch require dynamic activation scaling

When it happens

Trigger: Setting both quantization_config.dispatch = 'batched_mm' (or the equivalent implementation selector) and activation_scheme='static' on a finegrained-fp8-quantized MoE model (e.g. a Qwen/DeepSeek-style MoE), then running a forward pass.

Common situations: Loading a checkpoint quantized with static scales (e.g. a NV/Dell checkpoint with activation_scale tensors) and trying to swap the experts implementation to batched_mm for speed; copy-pasting a config that mixes options from two different recipes.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/aeb3a3018bb36d26. Report an issue: GitHub.