huggingface/transformers · error · RuntimeError

CUDA is not available in this environment; cannot export to

Error message

CUDA is not available in this environment; cannot export to the ExecuTorch CUDA backend.

What it means

prepare_for_cuda — the preparation hook for the ExecuTorch CUDA backend — requires a visible GPU because the backend delegates ops to Triton kernels compiled by AOTInductor, which needs a GPU to compile and autotune. If torch.cuda.is_available() is False it raises this RuntimeError before any model work. Notably the model's own device is irrelevant (CPU tensors are fine); only the compiler needs the GPU.

Source

Thrown at src/transformers/exporters/exporter_executorch.py:251

    model.requires_grad_(False)
    model = model.to(device="cpu")
    # XNNPACK has no `_grouped_mm.out` kernel — force MoE experts to `batched_mm`.
    if isinstance(model, PreTrainedModel) and model._can_set_experts_implementation():
        model.set_experts_implementation("batched_mm")
    partitioner = [XnnpackPartitioner()]
    return model, _make_contiguous(sample_inputs), partitioner


def prepare_for_cuda(model: PreTrainedModel, sample_inputs: dict[str, Any]):
    """GPU inference via the ExecuTorch CUDA backend, decoupled from the model's device.

    The backend requires bfloat16 (upcast here) and a visible GPU — it delegates ops to Triton
    kernels compiled by AOTInductor, which needs a GPU to compile/autotune. The model itself can
    stay on any device (e.g. CPU): AOTInductor targets the machine's GPU regardless of where the
    traced tensors live, so no `.to("cuda")` is needed."""
    if not torch.cuda.is_available():
        raise RuntimeError("CUDA is not available in this environment; cannot export to the ExecuTorch CUDA backend.")

    model.requires_grad_(False)
    dtype = module_dtype(model)
    if dtype is not None and dtype != torch.bfloat16:
        logger.warning(f"ExecuTorch CUDA backend requires bfloat16; upcasting model from {dtype}.")
        model = model.to(dtype=torch.bfloat16)
    partitioner = [CudaPartitioner([CudaBackend.generate_method_name_compile_spec(model.__class__.__name__)])]
    return model, _make_contiguous(sample_inputs), partitioner


_BACKEND_PREPARE = {
    "xnnpack": prepare_for_xnnpack,
    "cuda": prepare_for_cuda,
}


# ── Stage 2: Torch patches ────────────────────────────────────────────────────
# Reversible swaps of `torch` ops the ExecuTorch backends can't lower (`split_copy`,

View on GitHub (pinned to a597f97485)

Solutions

1. Run on a machine with a working GPU: verify python -c "import torch; print(torch.cuda.is_available())" is True.
2. If torch is the CPU-only wheel, reinstall a CUDA build (e.g. pip install torch --index-url https://download.pytorch.org/whl/cu*).
3. Check CUDA_VISIBLE_DEVICES is not empty and the driver/CUDA toolkit match torch's requirements.
4. On CPU-only targets, use backend="xnnpack" instead.

Example fix

# before (CPU-only host)
ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend="cuda"))  # RuntimeError

# after
import torch
assert torch.cuda.is_available(), "need a GPU for the ExecuTorch CUDA backend"
ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend="cuda"))

import torch

if not torch.cuda.is_available():
    raise SystemExit("ExecuTorch CUDA backend needs a GPU; use backend='xnnpack' on CPU-only hosts")
ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend="cuda"))

def can_use_executorch_cuda() -> bool:
    import torch
    return torch.cuda.is_available()

try:
    ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend="cuda"))
except RuntimeError as e:
    if "CUDA is not available" in str(e):
        ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend="xnnpack"))  # CPU fallback
    else:
        raise

When it happens

Trigger: ExecutorchConfig(backend="cuda") on a CPU-only machine, a CI runner without GPU, or a node where CUDA is masked (CUDA_VISIBLE_DEVICES="" or a driver/CUDA mismatch making is_available() False).

Common situations: Developing locally on a laptop then shipping to GPU CI; SLURM/K8s job landed on a CPU node; GPU present but torch built without CUDA (pip cpu wheel) so is_available() is False.

Related errors

• TP and DP cannot be used together
• To use {type(self).__name__}, please install the following d
• {type(self).__name__} requires newer versions of: {', '.join
• Expected config to be an ExecutorchConfig or dict, got {type
• Unsupported backend {config.backend} for ExecuTorch export

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/4c65b0e7e6423b29. Report an issue: GitHub.