{"record":{"id":"ab56358b46d0cb04","repo":"huggingface/transformers","slug":"async-batching-requires-cuda-but-torch-cuda-is-a","errorCode":null,"errorMessage":"Async batching requires CUDA, but {torch.cuda.is_available() = }","messagePattern":"Async batching requires CUDA, but (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/continuous_batching/input_outputs.py","lineNumber":668,"sourceCode":"          - <-N: device to host transfer of batch N\n          - UP N: update of batch N\n\n    You can see that the GPU is almost always busy, except where the █ is.\n    Proper ordering of steps is ensured through the use of CUDA events and streams.\n    \"\"\"\n\n    def __init__(\n        self,\n        cache: PagedAttentionCache,\n        config: PretrainedConfig,\n        continuous_batching_config: ContinuousBatchingConfig,\n        device: torch.device,\n        model_dtype: torch.dtype,\n        logit_processor: ContinuousBatchingLogitsProcessorList,\n    ) -> None:\n        # Async batching needs streams to function, so check is CUDA is available\n        if not torch.cuda.is_available():\n            raise RuntimeError(f\"Async batching requires CUDA, but {torch.cuda.is_available() = }\")\n        # IO pairs used to avoid race conditions\n        self.current_pair = 0\n        self.io_pairs = [\n            HostDeviceIOPair(\n                cache=cache,\n                config=config,\n                continuous_batching_config=continuous_batching_config,\n                device=device,\n                model_dtype=model_dtype,\n                logit_processor=logit_processor,\n            )\n            for _ in range(2)\n        ]\n        # CUDA streams\n        self.h2d_stream = torch.cuda.Stream(device=device)\n        self.d2h_stream = torch.cuda.Stream(device=device)\n        self.compute_stream = torch.cuda.Stream(device=device)\n        # Set all unused compute streams to None","sourceCodeStart":650,"sourceCodeEnd":686,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/input_outputs.py#L650-L686","documentation":"Raised by the async IO pair manager in input_outputs.py at construction: async batching overlaps host/device transfers using CUDA streams, and CUDA streams only exist on CUDA devices. torch.cuda.is_available() is False (CPU-only build, no driver, or CUDA_VISIBLE_DEVICES=''), so construction aborts.","triggerScenarios":"Enabling async batching (async mode in ContinuousBatchingConfig) on a CPU-only machine, an Apple MPS machine, or a CUDA container without a visible GPU. The check is unconditional at __init__, before any stream is created.","commonSituations":"Development on laptop then deploying to GPU; CI runners without GPUs; CUDA_VISIBLE_DEVICES set to empty by a job scheduler; CPU-only torch wheel installed by mistake.","solutions":["Run on a CUDA machine with a working driver and CUDA-enabled torch build","Disable async batching in ContinuousBatchingConfig to use the synchronous path on CPU","Fix visibility: unset CUDA_VISIBLE_DEVICES='' or correct the scheduler's GPU allocation"],"exampleFix":"# before\ncfg = ContinuousBatchingConfig(async_batching=True)  # on CPU-only box\n\n# after\ncfg = ContinuousBatchingConfig(async_batching=torch.cuda.is_available())","handlingStrategy":"validation","validationCode":"import torch\nif not torch.cuda.is_available():\n    cfg.async_batching = False  # or whatever field enables async IO pairs","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Gate async features on torch.cuda.is_available()","Verify GPU visibility (nvidia-smi, CUDA_VISIBLE_DEVICES) in CI","Install CUDA-enabled torch builds on inference hosts"],"tags":["cuda","environment","async","continuous-batching"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}