huggingface/transformers · error · NotImplementedError

DeepGEMM's FP4 (int8-packed) path requires a Blackwell (SM10

Error message

DeepGEMM's FP4 (int8-packed) path requires a Blackwell (SM100+) GPU; FP4 weights have no Hopper (SM90) kernel. Use an FP8 checkpoint, or run on a Blackwell GPU.

What it means

Error "DeepGEMM's FP4 (int8-packed) path requires a Blackwell (SM100+) GPU; FP4 weights have no Hopper (SM90) kernel. Use an FP8 checkpoint, or run on a Blackwell GPU." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:330

    """Before-load guard for DeepGEMM's FP8/FP4 arch constraints on the given weight/scale dtypes:

      - FP4 (``int8``-packed) weights have no Hopper (SM90) kernel — they need Blackwell (SM100+).
      - Blackwell has no float32 scale-factor path: ``_coerce_sf_for_kernel`` would silently round a
        ``float32`` scale to UE8M0 and corrupt the output. UE8M0 scales load as ``float8_e8m0fnu`` (the
        loader normalizes even float32-container checkpoints like dsv4-flash-base), so a ``float32`` scale
        on SM100 means a genuine non-UE8M0 checkpoint.

    Uses `is_sm100()` (compile-safe via `assume_constant_result`), so the whole guard folds to a constant under
    ``torch.compile``: the valid case compiles away to nothing, while an unsupported combo fails loud
    rather than letting the hot path silently corrupt (unlike an ``is_compiling`` skip, which would miss a
    model compiled from cold with no eager warmup). Both raise ``NotImplementedError``, which
    ``fp8_linear`` treats as "DeepGEMM declined" and falls back to Triton (SM90 consuming float32 SFs
    directly is fine, so those cases are no-ops).
    """
    if not is_sm100():
        # SM90: DeepGEMM has no FP4 (int8-packed) kernel, but consumes float32 SFs directly.
        if weight.dtype == torch.int8:
            raise NotImplementedError(
                "DeepGEMM's FP4 (int8-packed) path requires a Blackwell (SM100+) GPU; FP4 weights have no "
                "Hopper (SM90) kernel. Use an FP8 checkpoint, or run on a Blackwell GPU."
            )
        return

    # SM100: DeepGEMM has no float32 scale-factor path.
    if scale.dtype == torch.float32:
        raise NotImplementedError(
            "DeepGEMM has no float32 scale-factor path on Blackwell (SM100): these scales are plain float32 "
            "(quantization_config.scale_fmt='float'), and rounding them to UE8M0 would silently corrupt the "
            "output. Use a checkpoint quantized with scale_fmt='ue8m0', or a path that consumes float32 block "
            "scales directly — the FP8 linear falls back to Triton automatically; for experts use "
            "`model.set_experts_implementation('grouped_mm')`."
        )


def _ceil_to_ue8m0(sf: torch.Tensor) -> torch.Tensor:
    """Round each fp32 SF up to the nearest power of 2 (zero mantissa).

View on GitHub (pinned to a597f97485)

Solutions

1. Use an FP8 checkpoint instead of FP4 on Hopper GPUs.
2. Run on a Blackwell (SM100+) GPU for the FP4 path.

When it happens

Trigger: Raised in DeepGEMM FP4 path when the GPU is not Blackwell (SM100+) since FP4 kernels only exist there.

Common situations: Loading an FP4 (int8-packed) DeepGEMM checkpoint on Hopper or older GPUs.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/bf66defacebf2d48. Report an issue: GitHub.