{"record":{"id":"b950ab035ecb4f40","repo":"huggingface/transformers","slug":"deepgemm-mega-moe-requires-hidden-dim-and-inter","errorCode":null,"errorMessage":"DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim={hidden_dim}, intermediate_hidden={intermediate_hidden}.","messagePattern":"DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 \\(FP8 SF granularity\\); got hidden_dim=(.+?), intermediate_hidden=(.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepgemm.py","lineNumber":809,"sourceCode":"         the ``[E_local, 2*I, *]`` leading dims so downstream ``.size(...)`` reads\n         stay valid.\n\n    Unwraps any ``DTensor`` wrappers FSDP2/EP may have placed around the loader-\n    side Parameters — the kernel takes raw pointers.\n    \"\"\"\n    deepgemm = load_deepgemm_kernel()\n    gate_up_sf_raw = to_local(module.gate_up_proj_scale_inv.data)\n    down_sf_raw = to_local(module.down_proj_scale_inv.data)\n    # Force int8 view: the kernel's interleave reshape/empty_like/copy_ is bit-level.\n    gate_up_w = to_local(module.gate_up_proj.data).view(torch.int8).contiguous()\n    down_w = to_local(module.down_proj.data).view(torch.int8).contiguous()\n\n    intermediate_hidden = module.intermediate_dim\n    num_local_experts = module.num_experts\n    hidden_dim = module.hidden_dim\n\n    if hidden_dim % 32 != 0 or intermediate_hidden % 32 != 0:\n        raise ValueError(\n            f\"DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 \"\n            f\"(FP8 SF granularity); got hidden_dim={hidden_dim}, intermediate_hidden={intermediate_hidden}.\"\n        )\n\n    gate_up_sf = deepgemm.transform_sf_into_required_layout(\n        gate_up_sf_raw.float(),\n        2 * intermediate_hidden,\n        hidden_dim,\n        recipe=(1, 32),\n        num_groups=num_local_experts,\n    )\n    down_sf = deepgemm.transform_sf_into_required_layout(\n        down_sf_raw.float(),\n        hidden_dim,\n        intermediate_hidden,\n        recipe=(1, 32),\n        num_groups=num_local_experts,\n    )","sourceCodeStart":791,"sourceCodeEnd":827,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepgemm.py#L791-L827","documentation":"Error \"DeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim={hidden_dim}, intermediate_hidden={intermediate_hidden}.\" thrown in huggingface/transformers.","triggerScenarios":"Raised in DeepGEMM Mega MoE when hidden_dim or intermediate_hidden is not divisible by 32.","commonSituations":"A MoE checkpoint with dimensions misaligned to the FP8 scale-factor granularity required by Mega MoE kernels.","solutions":["Use a model whose hidden_dim and intermediate_hidden are divisible by 32 for DeepGEMM Mega MoE.","Fall back to the standard DeepGEMM experts dispatch."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}