huggingface/transformers · error · ImportError

Failed to load `kernels-community/deep-gemm` — check that a

Error message

Failed to load `kernels-community/deep-gemm` — check that a build matches the current torch/CUDA.

What it means

Error "Failed to load `kernels-community/deep-gemm` — check that a build matches the current torch/CUDA." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepgemm.py:250

def _load_deepgemm_kernel() -> None:
    """Load DeepGEMM once into the `_DEEPGEMM` module global, raising `ImportError` if the env or any
    required symbol is missing. Under NO circumstances may this function return a value: it rides
    through `@allow_in_graph` as an opaque fx node, whose return must be proxyable — returning the
    bundle (e.g. from the warm-cache short-circuit) breaks torch.compile with
    `Unsupported: torch.* op returned non-Tensor`. Callers read `_DEEPGEMM` back from the global.

    `@allow_in_graph` makes `torch.compile` treat the untraceable cold path (hub download + dynamic
    import via `lazy_load_kernel`) as a single opaque node instead of tracing into it; it returns `None`
    (proxyable) and populates the global, which `load_deepgemm_kernel` then returns.
    """
    global _DEEPGEMM
    if _DEEPGEMM is not None:
        return

    is_deepgemm_loadable(raise_error=True)
    kernel = lazy_load_kernel("deep-gemm")
    if kernel is None:
        raise ImportError(
            "Failed to load `kernels-community/deep-gemm` — check that a build matches the current torch/CUDA."
        )

    fp8_fp4_matmul = getattr(kernel, "fp8_fp4_gemm_nt", None)
    grouped_fp8_fp4_matmul_nt = getattr(kernel, "m_grouped_fp8_fp4_gemm_nt_contiguous", None)
    grouped_fp8_fp4_matmul_nn = getattr(kernel, "m_grouped_fp8_fp4_gemm_nn_contiguous", None)
    grouped_bf16_matmul_nt = getattr(kernel, "m_grouped_bf16_gemm_nt_contiguous", None)
    grouped_bf16_matmul_nn = getattr(kernel, "m_grouped_bf16_gemm_nn_contiguous", None)
    per_token_cast_to_fp8 = resolve_internal_import(kernel, chained_path="utils.per_token_cast_to_fp8")
    transform_sf_into_required_layout = getattr(kernel, "transform_sf_into_required_layout", None)
    transform_weights_for_mega_moe = getattr(kernel, "transform_weights_for_mega_moe", None)
    get_symm_buffer_for_mega_moe = getattr(kernel, "get_symm_buffer_for_mega_moe", None)
    get_mk_alignment = getattr(kernel, "get_mk_alignment_for_contiguous_layout", None)
    fp8_fp4_mega_moe = getattr(kernel, "fp8_fp4_mega_moe", None)

    missing = [
        name
        for name, attr in [

View on GitHub (pinned to a597f97485)

Solutions

1. Install/update the kernels package so `kernels-community/deep-gemm` has a build matching your torch/CUDA: `pip install -U kernels`.

When it happens

Trigger: Raised when loading the kernels-community/deep-gemm kernel fails due to a torch/CUDA build mismatch.

Common situations: Using DeepGEMM FP8 quantization on a torch/CUDA combination with no prebuilt deep-gemm kernel.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/013865a3822cabe2. Report an issue: GitHub.