huggingface/transformers · error · ImportError

Failed to load the finegrained-fp8 kernel — check that `kern

Error message

Failed to load the finegrained-fp8 kernel — check that `kernels-community/finegrained-fp8` has a build matching the current torch/CUDA.

What it means

Raised when the 'kernels' package IS installed but lazy_load_kernel("finegrained-fp8") returns None, meaning the kernel could not be fetched/built for the current environment. The kernels runtime downloads prebuilt kernels from kernels-community/finegrained-fp8 matching the installed torch version and CUDA runtime; if no matching wheel/build exists (or download/build failed), loading silently yields None and transformers raises this ImportError with an actionable hint.

Source

Thrown at src/transformers/integrations/finegrained_fp8.py:113

    global, which `load_finegrained_fp8_kernel` then returns.

    Under NO circumstances may this function return a value: an `@allow_in_graph` fx node's
    return must be proxyable, and returning the bundle (e.g. from the warm-cache
    short-circuit) breaks torch.compile with `Unsupported: torch.* op returned non-Tensor`.

    Raises `ImportError` if the `kernels` package is missing, or the kernel or required
    symbols cannot be found.
    """
    global _FINEGRAINED_FP8
    if _FINEGRAINED_FP8 is not None:
        return

    if not is_kernels_available():
        raise ImportError(f"finegrained-fp8 kernel unavailable: {_MISSING_KERNELS_MESSAGE}")

    kernel = lazy_load_kernel("finegrained-fp8")
    if kernel is None:
        raise ImportError(
            "Failed to load the finegrained-fp8 kernel — check that `kernels-community/finegrained-fp8` "
            "has a build matching the current torch/CUDA."
        )

    matmul = getattr(kernel, "matmul_2d", None)
    batched_matmul = getattr(kernel, "matmul_batched", None)
    grouped_matmul = getattr(kernel, "matmul_grouped", None)

    missing = [
        name
        for name, attr in [
            ("matmul_2d", matmul),
            ("matmul_batched", batched_matmul),
            ("matmul_grouped", grouped_matmul),
        ]
        if attr is None
    ]
    if missing:

View on GitHub (pinned to a597f97485)

Solutions

1. Check torch/CUDA compatibility: python -c "import torch; print(torch.__version__, torch.version.cuda)" and pin to a stable torch release that kernels-community/finegrained-fp8 supports
2. Ensure network access to huggingface.co so the kernels package can fetch the kernel, and warm the cache once on a connected machine
3. Upgrade the kernels package (pip install -U kernels) so it can resolve newer kernel builds
4. Fall back to another fp8 path (torchao/compressed-tensors) if this torch build is unsupported

Example fix

# before: torch 2.9.0.dev20250101+cu128 with no matching kernel build
load_finegrained_fp8_kernel()  # ImportError

# after: pin a stable torch with a published kernel build
pip install "torch==2.7.*" --index-url https://download.pytorch.org/whl/cu126
load_finegrained_fp8_kernel()

import torch
from transformers.utils import is_kernels_available

assert is_kernels_available(), "kernels package missing"
assert torch.cuda.is_available(), "finegrained-fp8 requires CUDA"
# warm the kernel once at startup so failures surface before training
from kernels import lazy_load_kernel
assert lazy_load_kernel("finegrained-fp8") is not None, "no matching build for torch/CUDA — pin a stable torch"

try:
    load_finegrained_fp8_kernel()
except ImportError as e:
    if "matching the current torch/CUDA" in str(e):
        raise RuntimeError("Pin a stable torch release; this build has no finegrained-fp8 kernel") from e
    raise

When it happens

Trigger: load_finegrained_fp8_kernel() on a machine where `kernels` is importable but the finegrained-fp8 kernel has no build for the installed torch/CUDA combination — e.g. a nightly torch build, an unsupported CUDA version, an offline machine, or an incompatible GPU architecture.

Common situations: Upgrading torch to a very new/nightly version before kernel-community publishes a matching build; running inside an air-gapped cluster where the kernel cannot be downloaded; CUDA driver/runtime mismatch after a system update.

Related errors

• finegrained-fp8 kernel unavailable: {_MISSING_KERNELS_MESSAG
• finegrained-fp8 kernel is missing required symbols: {', '.jo
• You need to install optimum-quanto in order to use KV cache
• You need to install `HQQ` in order to use KV cache quantizat
• batched_mm experts dispatch does not support activation_sche

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ca918fdec2cd17a0. Report an issue: GitHub.