{"record":{"id":"4c65b0e7e6423b29","repo":"huggingface/transformers","slug":"cuda-is-not-available-in-this-environment-cannot","errorCode":null,"errorMessage":"CUDA is not available in this environment; cannot export to the ExecuTorch CUDA backend.","messagePattern":"CUDA is not available in this environment; cannot export to the ExecuTorch CUDA backend\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/exporters/exporter_executorch.py","lineNumber":251,"sourceCode":"\n    model.requires_grad_(False)\n    model = model.to(device=\"cpu\")\n    # XNNPACK has no `_grouped_mm.out` kernel — force MoE experts to `batched_mm`.\n    if isinstance(model, PreTrainedModel) and model._can_set_experts_implementation():\n        model.set_experts_implementation(\"batched_mm\")\n    partitioner = [XnnpackPartitioner()]\n    return model, _make_contiguous(sample_inputs), partitioner\n\n\ndef prepare_for_cuda(model: PreTrainedModel, sample_inputs: dict[str, Any]):\n    \"\"\"GPU inference via the ExecuTorch CUDA backend, decoupled from the model's device.\n\n    The backend requires bfloat16 (upcast here) and a visible GPU — it delegates ops to Triton\n    kernels compiled by AOTInductor, which needs a GPU to compile/autotune. The model itself can\n    stay on any device (e.g. CPU): AOTInductor targets the machine's GPU regardless of where the\n    traced tensors live, so no `.to(\"cuda\")` is needed.\"\"\"\n    if not torch.cuda.is_available():\n        raise RuntimeError(\"CUDA is not available in this environment; cannot export to the ExecuTorch CUDA backend.\")\n\n    model.requires_grad_(False)\n    dtype = module_dtype(model)\n    if dtype is not None and dtype != torch.bfloat16:\n        logger.warning(f\"ExecuTorch CUDA backend requires bfloat16; upcasting model from {dtype}.\")\n        model = model.to(dtype=torch.bfloat16)\n    partitioner = [CudaPartitioner([CudaBackend.generate_method_name_compile_spec(model.__class__.__name__)])]\n    return model, _make_contiguous(sample_inputs), partitioner\n\n\n_BACKEND_PREPARE = {\n    \"xnnpack\": prepare_for_xnnpack,\n    \"cuda\": prepare_for_cuda,\n}\n\n\n# ── Stage 2: Torch patches ────────────────────────────────────────────────────\n# Reversible swaps of `torch` ops the ExecuTorch backends can't lower (`split_copy`,","sourceCodeStart":233,"sourceCodeEnd":269,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/exporters/exporter_executorch.py#L233-L269","documentation":"prepare_for_cuda — the preparation hook for the ExecuTorch CUDA backend — requires a visible GPU because the backend delegates ops to Triton kernels compiled by AOTInductor, which needs a GPU to compile and autotune. If torch.cuda.is_available() is False it raises this RuntimeError before any model work. Notably the model's own device is irrelevant (CPU tensors are fine); only the compiler needs the GPU.","triggerScenarios":"ExecutorchConfig(backend=\"cuda\") on a CPU-only machine, a CI runner without GPU, or a node where CUDA is masked (CUDA_VISIBLE_DEVICES=\"\" or a driver/CUDA mismatch making is_available() False).","commonSituations":"Developing locally on a laptop then shipping to GPU CI; SLURM/K8s job landed on a CPU node; GPU present but torch built without CUDA (pip cpu wheel) so is_available() is False.","solutions":["Run on a machine with a working GPU: verify python -c \"import torch; print(torch.cuda.is_available())\" is True.","If torch is the CPU-only wheel, reinstall a CUDA build (e.g. pip install torch --index-url https://download.pytorch.org/whl/cu*).","Check CUDA_VISIBLE_DEVICES is not empty and the driver/CUDA toolkit match torch's requirements.","On CPU-only targets, use backend=\"xnnpack\" instead."],"exampleFix":"# before (CPU-only host)\nExecutorchExporter().export(model, inputs, ExecutorchConfig(backend=\"cuda\"))  # RuntimeError\n\n# after\nimport torch\nassert torch.cuda.is_available(), \"need a GPU for the ExecuTorch CUDA backend\"\nExecutorchExporter().export(model, inputs, ExecutorchConfig(backend=\"cuda\"))","handlingStrategy":"validation","validationCode":"import torch\n\nif not torch.cuda.is_available():\n    raise SystemExit(\"ExecuTorch CUDA backend needs a GPU; use backend='xnnpack' on CPU-only hosts\")\nExecutorchExporter().export(model, inputs, ExecutorchConfig(backend=\"cuda\"))","typeGuard":"def can_use_executorch_cuda() -> bool:\n    import torch\n    return torch.cuda.is_available()","tryCatchPattern":"try:\n    ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend=\"cuda\"))\nexcept RuntimeError as e:\n    if \"CUDA is not available\" in str(e):\n        ExecutorchExporter().export(model, inputs, ExecutorchConfig(backend=\"xnnpack\"))  # CPU fallback\n    else:\n        raise","preventionTips":["Gate backend=\"cuda\" behind torch.cuda.is_available() at the CLI/config layer","Remember the model may stay on CPU — only the compiler needs the GPU","In schedulers, request GPU nodes explicitly so exports never land on CPU-only machines"],"tags":["export","executorch","cuda","environment","hardware"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}