huggingface/transformers · error · ValueError

DeepGEMM linear requires FP16 or BF16 activations, got {inpu

Error message

DeepGEMM linear requires FP16 or BF16 activations, got {input.dtype}

What it means

Error "DeepGEMM linear requires FP16 or BF16 activations, got {input.dtype}" thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:595

@deprecate_kwarg("output_dtype", version="v5.16")
def deepgemm_fp8_fp4_linear(
    input: torch.Tensor,
    weight: torch.Tensor,
    weight_scale_inv: torch.Tensor,
    bias: torch.Tensor | None = None,
    block_size: tuple[int, int] | None = None,
    output_dtype: torch.dtype | None = None,
    activation_scale: torch.Tensor | None = None,
) -> torch.Tensor:
    """End-to-end DeepGEMM linear: per-token activation quant + FP8/FP4 matmul.

    Static (per-tensor) activation quantization is rejected — DeepGEMM needs
    per-row SFs. Callers should route static activations through the Triton fallback.
    """
    if activation_scale is not None:
        raise NotImplementedError("DeepGEMM linear does not support static activation quantization.")
    if input.dtype not in (torch.bfloat16, torch.float16):
        raise ValueError(f"DeepGEMM linear requires FP16 or BF16 activations, got {input.dtype}")

    # Fail before the (hub-download + JIT) load if this device can't serve these dtypes.
    _assert_sm100_requirements(weight, weight_scale_inv)

    deepgemm = load_deepgemm_kernel()
    cast_kwargs = _select_fp8_cast_kwargs(weight, weight_scale_inv, block_size, is_sm100())

    input_2d = input.view(-1, input.shape[-1])
    qinput_2d, scale_2d = deepgemm.per_token_cast_to_fp8(input_2d, **cast_kwargs)
    output = torch.empty(qinput_2d.shape[0], weight.shape[0], device=input.device, dtype=input.dtype)

    # Pass `(1, 1, gran_k)` for int-SF paths so the kernel uses the right K granularity
    # (the default `(1, 1, 128)` mismatches FP4's gran_k=32). Float-SF leaves it None.
    sf_recipe = (1, 1, cast_kwargs["gran_k"]) if cast_kwargs.get("use_packed_ue8m0") else None
    deepgemm.fp8_fp4_matmul(
        (qinput_2d, _coerce_sf_for_kernel(scale_2d, is_sm100(), expected_mn=qinput_2d.size(0))),
        (weight, _coerce_sf_for_kernel(weight_scale_inv, is_sm100(), expected_mn=weight.size(0))),
        output,

View on GitHub (pinned to a597f97485)

Solutions

1. Cast activations to float16 or bfloat16 before the DeepGEMM linear.
2. Load the model with torch_dtype=torch.bfloat16.

When it happens

Trigger: Raised in DeepGEMM linear forward when input activations are not float16 or bfloat16.

Common situations: Running a DeepGEMM FP8 model with float32 activations instead of half precision.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/4be5d96fe272816a. Report an issue: GitHub.