{"record":{"id":"27162aad3a66ab02","repo":"huggingface/transformers","slug":"grouped-mm-experts-dispatch-does-not-support-activ","errorCode":null,"errorMessage":"grouped_mm experts dispatch does not support activation_scheme='static'. Use the default eager dispatch or switch to activation_scheme='dynamic'.","messagePattern":"grouped_mm experts dispatch does not support activation_scheme='static'\\. Use the default eager dispatch or switch to activation_scheme='dynamic'\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/finegrained_fp8.py","lineNumber":505,"sourceCode":"    # Post-mask sentinel rows: kernel left them uninitialized, so zero them out\n    # before the reduction below (uninit may be NaN; NaN * 0 = NaN).\n    weighted_out.masked_fill_(sentinel_mask, 0.0)\n\n    # Accumulate results using deterministic reshape+sum instead of index_add_\n    # (index_add_ with duplicate indices is non-deterministic on CUDA due to atomicAdd)\n    final_hidden_states = weighted_out.view(num_tokens, num_top_k, hidden_dim).sum(dim=1)\n\n    return final_hidden_states.to(hidden_states.dtype)\n\n\ndef fp8_grouped_mm_experts_forward(\n    self: torch.nn.Module,\n    hidden_states: torch.Tensor,\n    top_k_index: torch.Tensor,\n    top_k_weights: torch.Tensor,\n) -> torch.Tensor:\n    if self.activation_scheme == \"static\":\n        raise NotImplementedError(\n            \"grouped_mm experts dispatch does not support activation_scheme='static'. \"\n            \"Use the default eager dispatch or switch to activation_scheme='dynamic'.\"\n        )\n\n    finegrained_fp8 = load_finegrained_fp8_kernel()\n\n    device = hidden_states.device\n    num_top_k = top_k_index.size(-1)\n    num_tokens = hidden_states.size(0)\n    hidden_dim = hidden_states.size(-1)\n\n    # S is the number of selected token-expert pairs (S = num_tokens * num_top_k)\n    sample_weights = top_k_weights.reshape(-1)  # (S,)\n    expert_ids = top_k_index.reshape(-1)  # (S,)\n\n    # Sort by expert for grouped processing\n    expert_ids_g, perm = torch.sort(expert_ids)\n    selected_hidden_states_g = hidden_states[perm // num_top_k]","sourceCodeStart":487,"sourceCodeEnd":523,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/finegrained_fp8.py#L487-L523","documentation":"Same contract as the batched path, but for fp8_grouped_mm_experts_forward: the grouped-GEMM experts dispatch (matmul_grouped) quantizes activations dynamically per-block and cannot consume the pre-computed activation scales implied by activation_scheme='static'. The guard at the top of the forward raises NotImplementedError before any kernel call, telling you to either use the default eager dispatch or dynamic scheme.","triggerScenarios":"Selecting dispatch='grouped_mm' (the finegrained-fp8 grouped experts implementation) on an MoE model whose fp8 config has activation_scheme='static', then running the forward.","commonSituations":"Trying grouped GEMM for throughput on a DeepSeek-V3-style MoE checkpoint that ships static activation scales; enabling the grouped kernel in a benchmark script over a statically quantized model.","solutions":["Use activation_scheme='dynamic' in the quantization config when selecting grouped_mm dispatch","Keep 'static' and rely on the default eager experts dispatch","Re-quantize the checkpoint with dynamic activation scales if grouped kernels are required"],"exampleFix":"# before\nquant_config = FineGrainedFP8Config(activation_scheme=\"static\", dispatch=\"grouped_mm\")\n\n# after\nquant_config = FineGrainedFP8Config(activation_scheme=\"dynamic\", dispatch=\"grouped_mm\")","handlingStrategy":"validation","validationCode":"qc = FineGrainedFP8Config(activation_scheme=\"static\", dispatch=\"grouped_mm\")\nassert not (qc.activation_scheme == \"static\" and qc.dispatch == \"grouped_mm\"), \\\n    \"grouped_mm requires activation_scheme='dynamic'\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat static activation scales as eager-dispatch-only in your config builders","Add a unit test asserting your shipped quantization configs never combine static schemes with fused MoE dispatch"],"tags":["fp8","moe","quantization","config-mismatch","not-implemented"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}