huggingface/transformers · error · NotImplementedError

DeepGEMM has no float32 scale-factor path on Blackwell (SM10

Error message

DeepGEMM has no float32 scale-factor path on Blackwell (SM100): these scales are plain float32 (quantization_config.scale_fmt='float'), and rounding them to UE8M0 would silently corrupt the output. Use a checkpoint quantized with scale_fmt='ue8m0', or a path that consumes float32 block scales directly — the FP8 linear falls back to Triton automatically; for experts use `model.set_experts_implementation('grouped_mm')`.

What it means

Error "DeepGEMM has no float32 scale-factor path on Blackwell (SM100): these scales are plain float32 (quantization_config.scale_fmt='float'), and rounding them to UE8M0 would silently corrupt the output. Use a checkpoint quantized with scale_fmt='ue8m0', or a path that consumes float32 block scales directly — the FP8 linear falls back to Triton automatically; for experts use `model.set_experts_implementation('grouped_mm')`." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:338

    Uses `is_sm100()` (compile-safe via `assume_constant_result`), so the whole guard folds to a constant under
    ``torch.compile``: the valid case compiles away to nothing, while an unsupported combo fails loud
    rather than letting the hot path silently corrupt (unlike an ``is_compiling`` skip, which would miss a
    model compiled from cold with no eager warmup). Both raise ``NotImplementedError``, which
    ``fp8_linear`` treats as "DeepGEMM declined" and falls back to Triton (SM90 consuming float32 SFs
    directly is fine, so those cases are no-ops).
    """
    if not is_sm100():
        # SM90: DeepGEMM has no FP4 (int8-packed) kernel, but consumes float32 SFs directly.
        if weight.dtype == torch.int8:
            raise NotImplementedError(
                "DeepGEMM's FP4 (int8-packed) path requires a Blackwell (SM100+) GPU; FP4 weights have no "
                "Hopper (SM90) kernel. Use an FP8 checkpoint, or run on a Blackwell GPU."
            )
        return

    # SM100: DeepGEMM has no float32 scale-factor path.
    if scale.dtype == torch.float32:
        raise NotImplementedError(
            "DeepGEMM has no float32 scale-factor path on Blackwell (SM100): these scales are plain float32 "
            "(quantization_config.scale_fmt='float'), and rounding them to UE8M0 would silently corrupt the "
            "output. Use a checkpoint quantized with scale_fmt='ue8m0', or a path that consumes float32 block "
            "scales directly — the FP8 linear falls back to Triton automatically; for experts use "
            "`model.set_experts_implementation('grouped_mm')`."
        )


def _ceil_to_ue8m0(sf: torch.Tensor) -> torch.Tensor:
    """Round each fp32 SF up to the nearest power of 2 (zero mantissa).

    Mirrors `deep_gemm.utils.math.ceil_to_ue8m0`. On SM100 the kernel's
    `pack_fp32_into_ue8m0` cleanly extracts the biased exponent only when the
    mantissa is already zero — its inner shifts (`>> 15`, `>> 7`, `<< 1`)
    otherwise leak mantissa bits into adjacent UE8M0 byte slots and silently
    corrupt the SF. SM90 consumes raw fp32 SFs without going through this path.
    """
    int_view = sf.view(torch.int32)

View on GitHub (pinned to a597f97485)

Solutions

  1. Use a checkpoint quantized with scale_fmt='ue8m0'.
  2. Call `model.set_experts_implementation('grouped_mm')` for experts so float32 scales are consumed directly.

When it happens

Trigger: Raised in DeepGEMM FP8 path on Blackwell when scale factors are plain float32 instead of UE8M0.

Common situations: Running a scale_fmt='float' FP8 checkpoint on SM100 hardware where only UE8M0 scales are supported.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/2d6354c47940690f. Report an issue: GitHub.