{"record":{"id":"963f0253782ae1db","repo":"huggingface/transformers","slug":"deepgemm-requires-block-size-128-128-1","errorCode":null,"errorMessage":"DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got {block_size}.","messagePattern":"DeepGEMM requires `block_size` ∈ (.+?), got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepgemm.py","lineNumber":442,"sourceCode":"\n    Cases mirror the kernel's recipes:\n      - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only.\n      - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4).\n      - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in\n        `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float\n        the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast).\n      - FP8 weights + float SF: gran_k=128 float SF (DSv3).\n    \"\"\"\n    if weight.dtype == torch.int8:  # FP4\n        return {\"use_ue8m0\": True, \"gran_k\": 32, \"use_packed_ue8m0\": True}\n    # FP8 weights: validate block_size (informational; kernel infers recipe from SF dtype/shape).\n    if block_size is None:\n        raise ValueError(\n            \"DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.\"\n        )\n    block_size = tuple(block_size)\n    if block_size not in ((128, 128), (1, 128)):\n        raise ValueError(f\"DeepGEMM requires `block_size` ∈ {{(128, 128), (1, 128)}}, got {block_size}.\")\n    if weight_scale_inv.dtype == torch.float8_e8m0fnu and is_sm100:\n        return {\"use_ue8m0\": True, \"gran_k\": 128, \"use_packed_ue8m0\": True}\n    return {\"use_ue8m0\": False, \"gran_k\": 128}\n\n\n# ── Layout helpers (M-grouped contiguous, TMA-aligned) ─────────────────────────\n\n\ndef _build_deepgemm_contiguous_layout(\n    expert_ids_sorted: torch.Tensor, num_experts: int, alignment: int, use_psum_layout: bool\n) -> tuple[torch.Tensor, torch.Tensor, int]:\n    \"\"\"Build the TMA-aligned grouped layout DeepGEMM expects.\n\n    Returns `(sorted_to_padded, grouped_layout, total_padded_rows)`:\n      - `grouped_layout` is per-row expert id (Hopper, with `-1` for padding /\n        sentinels) or a cumsum of aligned per-expert counts (Blackwell).\n      - EP sentinels (values == `num_experts`) are routed past the last expert\n        block so DeepGEMM skips them.","sourceCodeStart":424,"sourceCodeEnd":460,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepgemm.py#L424-L460","documentation":"Error \"DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got {block_size}.\" thrown in huggingface/transformers.","triggerScenarios":"Raised in DeepGEMM when the FP8 block_size is not one of the supported (128,128) or (1,128) shapes.","commonSituations":"Checkpoints quantized with a non-standard FP8 block granularity loaded with the DeepGEMM backend.","solutions":["Use weights quantized with block_size (128, 128) or (1, 128).","Pick a different experts/linear implementation for other block sizes."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}