huggingface/transformers · error · ValueError

DeepGEMM requires block-wise quantized FP8 weights, but the

Error message

DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.

What it means

Error "DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:437

def _select_fp8_cast_kwargs(
    weight: torch.Tensor, weight_scale_inv: torch.Tensor, block_size: tuple | None, is_sm100: bool
) -> dict:
    """Pick the `per_token_cast_to_fp8` kwargs from weight dtype + SF dtype + arch.

    Cases mirror the kernel's recipes:
      - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only.
      - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4).
      - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in
        `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float
        the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast).
      - FP8 weights + float SF: gran_k=128 float SF (DSv3).
    """
    if weight.dtype == torch.int8:  # FP4
        return {"use_ue8m0": True, "gran_k": 32, "use_packed_ue8m0": True}
    # FP8 weights: validate block_size (informational; kernel infers recipe from SF dtype/shape).
    if block_size is None:
        raise ValueError(
            "DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set."
        )
    block_size = tuple(block_size)
    if block_size not in ((128, 128), (1, 128)):
        raise ValueError(f"DeepGEMM requires `block_size` ∈ {{(128, 128), (1, 128)}}, got {block_size}.")
    if weight_scale_inv.dtype == torch.float8_e8m0fnu and is_sm100:
        return {"use_ue8m0": True, "gran_k": 128, "use_packed_ue8m0": True}
    return {"use_ue8m0": False, "gran_k": 128}


# ── Layout helpers (M-grouped contiguous, TMA-aligned) ─────────────────────────


def _build_deepgemm_contiguous_layout(
    expert_ids_sorted: torch.Tensor, num_experts: int, alignment: int, use_psum_layout: bool
) -> tuple[torch.Tensor, torch.Tensor, int]:
    """Build the TMA-aligned grouped layout DeepGEMM expects.

View on GitHub (pinned to a597f97485)

Solutions

1. Use FP8 weights quantized with a `block_size` in the quantization config.
2. Switch the experts implementation to one that supports non-block-quantized weights.

When it happens

Trigger: Raised in DeepGEMM expert quantization when FP8 expert weights lack block_size in their quantization config.

Common situations: Loading FP8 MoE experts quantized without block-wise scaling metadata into the DeepGEMM path.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/01cdaf0552f0d136. Report an issue: GitHub.