huggingface/transformers · error · ValueError

DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hi

Error message

DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim={hidden_dim}, intermediate_hidden={intermediate_hidden}.

What it means

Error "DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim={hidden_dim}, intermediate_hidden={intermediate_hidden}." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:809

         the ``[E_local, 2*I, *]`` leading dims so downstream ``.size(...)`` reads
         stay valid.

    Unwraps any ``DTensor`` wrappers FSDP2/EP may have placed around the loader-
    side Parameters — the kernel takes raw pointers.
    """
    deepgemm = load_deepgemm_kernel()
    gate_up_sf_raw = to_local(module.gate_up_proj_scale_inv.data)
    down_sf_raw = to_local(module.down_proj_scale_inv.data)
    # Force int8 view: the kernel's interleave reshape/empty_like/copy_ is bit-level.
    gate_up_w = to_local(module.gate_up_proj.data).view(torch.int8).contiguous()
    down_w = to_local(module.down_proj.data).view(torch.int8).contiguous()

    intermediate_hidden = module.intermediate_dim
    num_local_experts = module.num_experts
    hidden_dim = module.hidden_dim

    if hidden_dim % 32 != 0 or intermediate_hidden % 32 != 0:
        raise ValueError(
            f"DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 "
            f"(FP8 SF granularity); got hidden_dim={hidden_dim}, intermediate_hidden={intermediate_hidden}."
        )

    gate_up_sf = deepgemm.transform_sf_into_required_layout(
        gate_up_sf_raw.float(),
        2 * intermediate_hidden,
        hidden_dim,
        recipe=(1, 32),
        num_groups=num_local_experts,
    )
    down_sf = deepgemm.transform_sf_into_required_layout(
        down_sf_raw.float(),
        hidden_dim,
        intermediate_hidden,
        recipe=(1, 32),
        num_groups=num_local_experts,
    )

View on GitHub (pinned to a597f97485)

Solutions

1. Use a model whose hidden_dim and intermediate_hidden are divisible by 32 for DeepGEMM Mega MoE.
2. Fall back to the standard DeepGEMM experts dispatch.

When it happens

Trigger: Raised in DeepGEMM Mega MoE when hidden_dim or intermediate_hidden is not divisible by 32.

Common situations: A MoE checkpoint with dimensions misaligned to the FP8 scale-factor granularity required by Mega MoE kernels.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/b950ab035ecb4f40. Report an issue: GitHub.