huggingface/transformers · error · ImportError

finegrained-fp8 kernel unavailable: {_MISSING_KERNELS_MESSAG

Error message

finegrained-fp8 kernel unavailable: {_MISSING_KERNELS_MESSAGE}

What it means

Raised by _load_finegrained_fp8_kernel() in src/transformers/integrations/finegrained_fp8.py when the lazy finegrained-fp8 kernel loader runs and is_kernels_available() returns False. Transformers only vendors the integration glue; the actual Triton/CUDA kernels live in the external 'kernels' package (kernels-community/finegrained-fp8 on Hugging Face). Without that package installed the integration cannot construct the FineGrainedFP8 bundle and refuses to continue.

Source

Thrown at src/transformers/integrations/finegrained_fp8.py:109

    Load the finegrained-fp8 Triton kernel once into the `_FINEGRAINED_FP8` module global.

    `@allow_in_graph` makes `torch.compile` treat the untraceable hub download + dynamic import as a
    single opaque node instead of tracing into it; it returns `None` (proxyable) and populates the
    global, which `load_finegrained_fp8_kernel` then returns.

    Under NO circumstances may this function return a value: an `@allow_in_graph` fx node's
    return must be proxyable, and returning the bundle (e.g. from the warm-cache
    short-circuit) breaks torch.compile with `Unsupported: torch.* op returned non-Tensor`.

    Raises `ImportError` if the `kernels` package is missing, or the kernel or required
    symbols cannot be found.
    """
    global _FINEGRAINED_FP8
    if _FINEGRAINED_FP8 is not None:
        return

    if not is_kernels_available():
        raise ImportError(f"finegrained-fp8 kernel unavailable: {_MISSING_KERNELS_MESSAGE}")

    kernel = lazy_load_kernel("finegrained-fp8")
    if kernel is None:
        raise ImportError(
            "Failed to load the finegrained-fp8 kernel — check that `kernels-community/finegrained-fp8` "
            "has a build matching the current torch/CUDA."
        )

    matmul = getattr(kernel, "matmul_2d", None)
    batched_matmul = getattr(kernel, "matmul_batched", None)
    grouped_matmul = getattr(kernel, "matmul_grouped", None)

    missing = [
        name
        for name, attr in [
            ("matmul_2d", matmul),
            ("matmul_batched", batched_matmul),
            ("matmul_grouped", grouped_matmul),

View on GitHub (pinned to a597f97485)

Solutions

1. pip install kernels (the HF kernels runtime, https://github.com/huggingface/kernels)
2. If kernels is installed but broken, verify with `python -c "import kernels"` and reinstall it
3. If you did not intend to use finegrained-fp8, remove or change the quantization config that triggers this code path

Example fix

# before
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=FineGrainedFP8Config())
# ImportError: finegrained-fp8 kernel unavailable

# after
pip install kernels
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=FineGrainedFP8Config())

from transformers.utils import is_kernels_available

if not is_kernels_available():
    raise SystemExit("pip install kernels before enabling finegrained-fp8")
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=finegrained_fp8_cfg)

try:
    from transformers.integrations.finegrained_fp8 import load_finegrained_fp8_kernel
    load_finegrained_fp8_kernel()
except ImportError as e:
    if "kernels" in str(e):
        print("Missing dependency: pip install kernels")
    raise

When it happens

Trigger: Calling load_finegrained_fp8_kernel(), or any code path that reaches it (e.g. _apply_finegrained_fp8 / enabling an fp8 recipe on a model) after importing transformers, on an environment where `import kernels` fails or the kernels package was never installed.

Common situations: User enables finegrained FP8 quantization (e.g. via a quantization_config selecting finegrained-fp8) in a fresh venv or a Docker image that only pip-installed transformers[torch]. CI images that strip optional dependencies also hit this.

Related errors

• Failed to load the finegrained-fp8 kernel — check that `kern
• finegrained-fp8 kernel is missing required symbols: {', '.jo
• You need to install optimum-quanto in order to use KV cache
• You need to install `HQQ` in order to use KV cache quantizat
• batched_mm experts dispatch does not support activation_sche

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/99a919e7ad7c93d8. Report an issue: GitHub.