huggingface/transformers · error · NotImplementedError

DeepGEMM linear does not support static activation quantizat

Error message

DeepGEMM linear does not support static activation quantization.

What it means

Error "DeepGEMM linear does not support static activation quantization." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:593

@deprecate_kwarg("output_dtype", version="v5.16")
def deepgemm_fp8_fp4_linear(
    input: torch.Tensor,
    weight: torch.Tensor,
    weight_scale_inv: torch.Tensor,
    bias: torch.Tensor | None = None,
    block_size: tuple[int, int] | None = None,
    output_dtype: torch.dtype | None = None,
    activation_scale: torch.Tensor | None = None,
) -> torch.Tensor:
    """End-to-end DeepGEMM linear: per-token activation quant + FP8/FP4 matmul.

    Static (per-tensor) activation quantization is rejected — DeepGEMM needs
    per-row SFs. Callers should route static activations through the Triton fallback.
    """
    if activation_scale is not None:
        raise NotImplementedError("DeepGEMM linear does not support static activation quantization.")
    if input.dtype not in (torch.bfloat16, torch.float16):
        raise ValueError(f"DeepGEMM linear requires FP16 or BF16 activations, got {input.dtype}")

    # Fail before the (hub-download + JIT) load if this device can't serve these dtypes.
    _assert_sm100_requirements(weight, weight_scale_inv)

    deepgemm = load_deepgemm_kernel()
    cast_kwargs = _select_fp8_cast_kwargs(weight, weight_scale_inv, block_size, is_sm100())

    input_2d = input.view(-1, input.shape[-1])
    qinput_2d, scale_2d = deepgemm.per_token_cast_to_fp8(input_2d, **cast_kwargs)
    output = torch.empty(qinput_2d.shape[0], weight.shape[0], device=input.device, dtype=input.dtype)

    # Pass `(1, 1, gran_k)` for int-SF paths so the kernel uses the right K granularity
    # (the default `(1, 1, 128)` mismatches FP4's gran_k=32). Float-SF leaves it None.
    sf_recipe = (1, 1, cast_kwargs["gran_k"]) if cast_kwargs.get("use_packed_ue8m0") else None
    deepgemm.fp8_fp4_matmul(
        (qinput_2d, _coerce_sf_for_kernel(scale_2d, is_sm100(), expected_mn=qinput_2d.size(0))),

View on GitHub (pinned to a597f97485)

Solutions

1. Use dynamic activation quantization instead of static for DeepGEMM linear.
2. Switch to an implementation supporting static activation quantization.

When it happens

Trigger: Raised in DeepGEMM linear forward when activation_scheme is 'static' quantization.

Common situations: Static-quantized activations passed to DeepGEMM linear layers, which only support dynamic activation quantization.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f830f0c5f9eb515f. Report an issue: GitHub.