huggingface/transformers · error · ValueError

DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got

Error message

DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got {block_size}.

What it means

Error "DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got {block_size}." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:442

    Cases mirror the kernel's recipes:
      - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only.
      - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4).
      - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in
        `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float
        the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast).
      - FP8 weights + float SF: gran_k=128 float SF (DSv3).
    """
    if weight.dtype == torch.int8:  # FP4
        return {"use_ue8m0": True, "gran_k": 32, "use_packed_ue8m0": True}
    # FP8 weights: validate block_size (informational; kernel infers recipe from SF dtype/shape).
    if block_size is None:
        raise ValueError(
            "DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set."
        )
    block_size = tuple(block_size)
    if block_size not in ((128, 128), (1, 128)):
        raise ValueError(f"DeepGEMM requires `block_size` ∈ {{(128, 128), (1, 128)}}, got {block_size}.")
    if weight_scale_inv.dtype == torch.float8_e8m0fnu and is_sm100:
        return {"use_ue8m0": True, "gran_k": 128, "use_packed_ue8m0": True}
    return {"use_ue8m0": False, "gran_k": 128}


# ── Layout helpers (M-grouped contiguous, TMA-aligned) ─────────────────────────


def _build_deepgemm_contiguous_layout(
    expert_ids_sorted: torch.Tensor, num_experts: int, alignment: int, use_psum_layout: bool
) -> tuple[torch.Tensor, torch.Tensor, int]:
    """Build the TMA-aligned grouped layout DeepGEMM expects.

    Returns `(sorted_to_padded, grouped_layout, total_padded_rows)`:
      - `grouped_layout` is per-row expert id (Hopper, with `-1` for padding /
        sentinels) or a cumsum of aligned per-expert counts (Blackwell).
      - EP sentinels (values == `num_experts`) are routed past the last expert
        block so DeepGEMM skips them.

View on GitHub (pinned to a597f97485)

Solutions

1. Use weights quantized with block_size (128, 128) or (1, 128).
2. Pick a different experts/linear implementation for other block sizes.

When it happens

Trigger: Raised in DeepGEMM when the FP8 block_size is not one of the supported (128,128) or (1,128) shapes.

Common situations: Checkpoints quantized with a non-standard FP8 block granularity loaded with the DeepGEMM backend.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/963f0253782ae1db. Report an issue: GitHub.