{"record":{"id":"01cdaf0552f0d136","repo":"huggingface/transformers","slug":"deepgemm-requires-block-wise-quantized-fp8-weights","errorCode":null,"errorMessage":"DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.","messagePattern":"DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepgemm.py","lineNumber":437,"sourceCode":"\ndef _select_fp8_cast_kwargs(\n    weight: torch.Tensor, weight_scale_inv: torch.Tensor, block_size: tuple | None, is_sm100: bool\n) -> dict:\n    \"\"\"Pick the `per_token_cast_to_fp8` kwargs from weight dtype + SF dtype + arch.\n\n    Cases mirror the kernel's recipes:\n      - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only.\n      - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4).\n      - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in\n        `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float\n        the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast).\n      - FP8 weights + float SF: gran_k=128 float SF (DSv3).\n    \"\"\"\n    if weight.dtype == torch.int8:  # FP4\n        return {\"use_ue8m0\": True, \"gran_k\": 32, \"use_packed_ue8m0\": True}\n    # FP8 weights: validate block_size (informational; kernel infers recipe from SF dtype/shape).\n    if block_size is None:\n        raise ValueError(\n            \"DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.\"\n        )\n    block_size = tuple(block_size)\n    if block_size not in ((128, 128), (1, 128)):\n        raise ValueError(f\"DeepGEMM requires `block_size` ∈ {{(128, 128), (1, 128)}}, got {block_size}.\")\n    if weight_scale_inv.dtype == torch.float8_e8m0fnu and is_sm100:\n        return {\"use_ue8m0\": True, \"gran_k\": 128, \"use_packed_ue8m0\": True}\n    return {\"use_ue8m0\": False, \"gran_k\": 128}\n\n\n# ── Layout helpers (M-grouped contiguous, TMA-aligned) ─────────────────────────\n\n\ndef _build_deepgemm_contiguous_layout(\n    expert_ids_sorted: torch.Tensor, num_experts: int, alignment: int, use_psum_layout: bool\n) -> tuple[torch.Tensor, torch.Tensor, int]:\n    \"\"\"Build the TMA-aligned grouped layout DeepGEMM expects.\n","sourceCodeStart":419,"sourceCodeEnd":455,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepgemm.py#L419-L455","documentation":"Error \"DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.\" thrown in huggingface/transformers.","triggerScenarios":"Raised in DeepGEMM expert quantization when FP8 expert weights lack block_size in their quantization config.","commonSituations":"Loading FP8 MoE experts quantized without block-wise scaling metadata into the DeepGEMM path.","solutions":["Use FP8 weights quantized with a `block_size` in the quantization config.","Switch the experts implementation to one that supports non-block-quantized weights."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}