huggingface/transformers · error · NotImplementedError

DeepGEMM Mega MoE requires FP4-packed expert weights (dtype=

Error message

DeepGEMM Mega MoE requires FP4-packed expert weights (dtype=`int8`), got `{self.gate_up_proj.dtype}`. Use the 'deepgemm' dispatch for FP8 experts.

What it means

Error "DeepGEMM Mega MoE requires FP4-packed expert weights (dtype=`int8`), got `{self.gate_up_proj.dtype}`. Use the 'deepgemm' dispatch for FP8 experts." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:869

    `process_group` is supplied automatically by `MoeTensorParalellExperts._prepare_input_fn`
    when the module is wrapped for TP — it's required for the symm-buffer rendezvous
    on first forward. `top_k_index` is GLOBAL expert ids (`-1` marks skipped slots).

    Caller-managed `self` attributes:
      - `gate_up_proj`, `gate_up_proj_scale_inv`: L1 weight + UE8M0 SF.
      - `down_proj`, `down_proj_scale_inv`: L2 weight + UE8M0 SF.
      Both pairs must be transformed together via
      `transform_weights_for_mega_moe((gate_up, gate_up_sf), (down, down_sf))`.
      - `config.swiglu_limit` (optional): SwiGLU clamp; absent → unclamped.
    """
    # Fail before the (hub-download + JIT) load if this device can't serve these dtypes. Mega MoE is
    # Blackwell-only, and its weights are always FP4 (int8) — so the FP4 arch check doubles as the
    # SM100 gate; the explicit `!= int8` check below covers a non-FP4 (misconfigured) checkpoint.
    _assert_sm100_requirements(self.gate_up_proj, self.down_proj_scale_inv)

    if self.gate_up_proj.dtype != torch.int8:
        raise NotImplementedError(
            f"DeepGEMM Mega MoE requires FP4-packed expert weights (dtype=`int8`), got "
            f"`{self.gate_up_proj.dtype}`. Use the 'deepgemm' dispatch for FP8 experts."
        )

    if process_group is None:
        raise ValueError(
            "DeepGEMM Mega MoE requires a `process_group` for the EP group. The TP wrapping "
            "(MoeTensorParalellMegaMoeExperts) supplies it automatically; pass it explicitly otherwise."
        )

    deepgemm = load_deepgemm_kernel()

    # First-forward one-shot: pack UE8M0 SFs and interleave the L1/L2 weights for UTCCP.
    # Kept lazy here (instead of in a quantizer load-time hook) so the megamoe-specific
    # setup lives alongside the megamoe forward — `set_experts_implementation` refuses
    # to flip in/out of `deepgemm_megamoe` at runtime, so the flag won't go stale.
    if not getattr(self, "_megamoe_transformed", False):
        setup_megamoe_weights(self)

View on GitHub (pinned to a597f97485)

Solutions

1. Use FP4-packed (int8 dtype) expert weights with Mega MoE.
2. Use the 'deepgemm' dispatch for FP8 experts instead.

When it happens

Trigger: Raised in DeepGEMM Mega MoE when expert weights are not FP4-packed int8.

Common situations: Using Mega MoE dispatch with FP8 expert weights; FP8 experts must use the regular deepgemm dispatch.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/84f79cda43c1de1a. Report an issue: GitHub.