{"record":{"id":"aeb3a3018bb36d26","repo":"huggingface/transformers","slug":"batched-mm-experts-dispatch-does-not-support-activ","errorCode":null,"errorMessage":"batched_mm experts dispatch does not support activation_scheme='static'. Use the default eager dispatch or switch to activation_scheme='dynamic'.","messagePattern":"batched_mm experts dispatch does not support activation_scheme='static'\\. Use the default eager dispatch or switch to activation_scheme='dynamic'\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/finegrained_fp8.py","lineNumber":431,"sourceCode":"            scale_inv,\n            offsets=offsets,\n            tokens_per_expert=tokens_per_expert,\n            block_size=self.block_size,\n        )\n        y = y.reshape(self.n_groups, *input_shape, -1).movedim(0, -2)\n        if self.has_bias:\n            y.add_(self.bias.view(self.n_groups, -1))\n        return y\n\n\ndef fp8_batched_mm_experts_forward(\n    self: torch.nn.Module,\n    hidden_states: torch.Tensor,\n    top_k_index: torch.Tensor,\n    top_k_weights: torch.Tensor,\n) -> torch.Tensor:\n    if self.activation_scheme == \"static\":\n        raise NotImplementedError(\n            \"batched_mm experts dispatch does not support activation_scheme='static'. \"\n            \"Use the default eager dispatch or switch to activation_scheme='dynamic'.\"\n        )\n\n    finegrained_fp8 = load_finegrained_fp8_kernel()\n\n    num_top_k = top_k_index.size(-1)\n    num_tokens = hidden_states.size(0)\n    hidden_dim = hidden_states.size(-1)\n\n    # S is the number of selected tokens-experts pairs (S = num_tokens * num_top_k)\n    # Replicate each token num_top_k times to align with the flattened (S,) routing tensors.\n    selected_hidden_states = hidden_states.repeat_interleave(num_top_k, dim=0)\n    sample_weights = top_k_weights.reshape(-1)  # (S,)\n    expert_ids = top_k_index.reshape(-1)  # (S,)\n\n    # EP sentinel handling: leave `expert_ids` unclamped — the batched kernel early-returns on\n    # `expert_id >= NUM_EXPERTS`, leaving sentinel output rows uninitialized. The post-mask below","sourceCodeStart":413,"sourceCodeEnd":449,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/finegrained_fp8.py#L413-L449","documentation":"fp8_batched_mm_experts_forward is one of the finegrained-fp8 MoE dispatch implementations (selected via the dispatch/implementation knob on the MoE layer). It quantizes activations on the fly, which only works with activation_scheme='dynamic'; the 'static' scheme relies on pre-computed activation scales that this batched_matmul path does not consume, so it explicitly raises NotImplementedError instead of silently producing wrong numerics.","triggerScenarios":"Setting both quantization_config.dispatch = 'batched_mm' (or the equivalent implementation selector) and activation_scheme='static' on a finegrained-fp8-quantized MoE model (e.g. a Qwen/DeepSeek-style MoE), then running a forward pass.","commonSituations":"Loading a checkpoint quantized with static scales (e.g. a NV/Dell checkpoint with activation_scale tensors) and trying to swap the experts implementation to batched_mm for speed; copy-pasting a config that mixes options from two different recipes.","solutions":["Switch to activation_scheme='dynamic' in the quantization config if you want batched_mm dispatch","Keep activation_scheme='static' and use the default eager experts dispatch (do not select batched_mm)","Re-quantize the model with dynamic activation scaling if you need the batched path"],"exampleFix":"# before\nquant_config = FineGrainedFP8Config(activation_scheme=\"static\", dispatch=\"batched_mm\")\n\n# after\nquant_config = FineGrainedFP8Config(activation_scheme=\"dynamic\", dispatch=\"batched_mm\")\n# or: keep \"static\" and omit dispatch to use the default eager path","handlingStrategy":"validation","validationCode":"qc = FineGrainedFP8Config(activation_scheme=\"static\", dispatch=\"batched_mm\")\nassert not (qc.activation_scheme == \"static\" and qc.dispatch == \"batched_mm\"), \\\n    \"batched_mm requires activation_scheme='dynamic'\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep (dispatch, activation_scheme) pairs in a validated preset dict instead of setting fields independently","Remember: both batched_mm and grouped_mm experts dispatch require dynamic activation scaling"],"tags":["fp8","moe","quantization","config-mismatch","not-implemented"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}