{"record":{"id":"ca918fdec2cd17a0","repo":"huggingface/transformers","slug":"failed-to-load-the-finegrained-fp8-kernel-check","errorCode":null,"errorMessage":"Failed to load the finegrained-fp8 kernel — check that `kernels-community/finegrained-fp8` has a build matching the current torch/CUDA.","messagePattern":"Failed to load the finegrained-fp8 kernel — check that `kernels-community/finegrained-fp8` has a build matching the current torch/CUDA\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/finegrained_fp8.py","lineNumber":113,"sourceCode":"    global, which `load_finegrained_fp8_kernel` then returns.\n\n    Under NO circumstances may this function return a value: an `@allow_in_graph` fx node's\n    return must be proxyable, and returning the bundle (e.g. from the warm-cache\n    short-circuit) breaks torch.compile with `Unsupported: torch.* op returned non-Tensor`.\n\n    Raises `ImportError` if the `kernels` package is missing, or the kernel or required\n    symbols cannot be found.\n    \"\"\"\n    global _FINEGRAINED_FP8\n    if _FINEGRAINED_FP8 is not None:\n        return\n\n    if not is_kernels_available():\n        raise ImportError(f\"finegrained-fp8 kernel unavailable: {_MISSING_KERNELS_MESSAGE}\")\n\n    kernel = lazy_load_kernel(\"finegrained-fp8\")\n    if kernel is None:\n        raise ImportError(\n            \"Failed to load the finegrained-fp8 kernel — check that `kernels-community/finegrained-fp8` \"\n            \"has a build matching the current torch/CUDA.\"\n        )\n\n    matmul = getattr(kernel, \"matmul_2d\", None)\n    batched_matmul = getattr(kernel, \"matmul_batched\", None)\n    grouped_matmul = getattr(kernel, \"matmul_grouped\", None)\n\n    missing = [\n        name\n        for name, attr in [\n            (\"matmul_2d\", matmul),\n            (\"matmul_batched\", batched_matmul),\n            (\"matmul_grouped\", grouped_matmul),\n        ]\n        if attr is None\n    ]\n    if missing:","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/finegrained_fp8.py#L95-L131","documentation":"Raised when the 'kernels' package IS installed but lazy_load_kernel(\"finegrained-fp8\") returns None, meaning the kernel could not be fetched/built for the current environment. The kernels runtime downloads prebuilt kernels from kernels-community/finegrained-fp8 matching the installed torch version and CUDA runtime; if no matching wheel/build exists (or download/build failed), loading silently yields None and transformers raises this ImportError with an actionable hint.","triggerScenarios":"load_finegrained_fp8_kernel() on a machine where `kernels` is importable but the finegrained-fp8 kernel has no build for the installed torch/CUDA combination — e.g. a nightly torch build, an unsupported CUDA version, an offline machine, or an incompatible GPU architecture.","commonSituations":"Upgrading torch to a very new/nightly version before kernel-community publishes a matching build; running inside an air-gapped cluster where the kernel cannot be downloaded; CUDA driver/runtime mismatch after a system update.","solutions":["Check torch/CUDA compatibility: python -c \"import torch; print(torch.__version__, torch.version.cuda)\" and pin to a stable torch release that kernels-community/finegrained-fp8 supports","Ensure network access to huggingface.co so the kernels package can fetch the kernel, and warm the cache once on a connected machine","Upgrade the kernels package (pip install -U kernels) so it can resolve newer kernel builds","Fall back to another fp8 path (torchao/compressed-tensors) if this torch build is unsupported"],"exampleFix":"# before: torch 2.9.0.dev20250101+cu128 with no matching kernel build\nload_finegrained_fp8_kernel()  # ImportError\n\n# after: pin a stable torch with a published kernel build\npip install \"torch==2.7.*\" --index-url https://download.pytorch.org/whl/cu126\nload_finegrained_fp8_kernel()","handlingStrategy":"validation","validationCode":"import torch\nfrom transformers.utils import is_kernels_available\n\nassert is_kernels_available(), \"kernels package missing\"\nassert torch.cuda.is_available(), \"finegrained-fp8 requires CUDA\"\n# warm the kernel once at startup so failures surface before training\nfrom kernels import lazy_load_kernel\nassert lazy_load_kernel(\"finegrained-fp8\") is not None, \"no matching build for torch/CUDA — pin a stable torch\"","typeGuard":null,"tryCatchPattern":"try:\n    load_finegrained_fp8_kernel()\nexcept ImportError as e:\n    if \"matching the current torch/CUDA\" in str(e):\n        raise RuntimeError(\"Pin a stable torch release; this build has no finegrained-fp8 kernel\") from e\n    raise","preventionTips":["Use stable torch releases, not nightlies, when running fp8 kernels","Warm/fetch kernels on a connected machine and cache them into the image for offline clusters","Log torch.__version__ and torch.version.cuda at startup to correlate with kernel build availability"],"tags":["fp8","quantization","kernels","torch-cuda-mismatch","import-error"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}