huggingface/transformers · error · RuntimeError

Async batching requires CUDA, but {torch.cuda.is_available()

Error message

Async batching requires CUDA, but {torch.cuda.is_available() = }

What it means

Raised by the async IO pair manager in input_outputs.py at construction: async batching overlaps host/device transfers using CUDA streams, and CUDA streams only exist on CUDA devices. torch.cuda.is_available() is False (CPU-only build, no driver, or CUDA_VISIBLE_DEVICES=''), so construction aborts.

Source

Thrown at src/transformers/generation/continuous_batching/input_outputs.py:668

          - <-N: device to host transfer of batch N
          - UP N: update of batch N

    You can see that the GPU is almost always busy, except where the █ is.
    Proper ordering of steps is ensured through the use of CUDA events and streams.
    """

    def __init__(
        self,
        cache: PagedAttentionCache,
        config: PretrainedConfig,
        continuous_batching_config: ContinuousBatchingConfig,
        device: torch.device,
        model_dtype: torch.dtype,
        logit_processor: ContinuousBatchingLogitsProcessorList,
    ) -> None:
        # Async batching needs streams to function, so check is CUDA is available
        if not torch.cuda.is_available():
            raise RuntimeError(f"Async batching requires CUDA, but {torch.cuda.is_available() = }")
        # IO pairs used to avoid race conditions
        self.current_pair = 0
        self.io_pairs = [
            HostDeviceIOPair(
                cache=cache,
                config=config,
                continuous_batching_config=continuous_batching_config,
                device=device,
                model_dtype=model_dtype,
                logit_processor=logit_processor,
            )
            for _ in range(2)
        ]
        # CUDA streams
        self.h2d_stream = torch.cuda.Stream(device=device)
        self.d2h_stream = torch.cuda.Stream(device=device)
        self.compute_stream = torch.cuda.Stream(device=device)
        # Set all unused compute streams to None

View on GitHub (pinned to a597f97485)

Solutions

1. Run on a CUDA machine with a working driver and CUDA-enabled torch build
2. Disable async batching in ContinuousBatchingConfig to use the synchronous path on CPU
3. Fix visibility: unset CUDA_VISIBLE_DEVICES='' or correct the scheduler's GPU allocation

Example fix

# before
cfg = ContinuousBatchingConfig(async_batching=True)  # on CPU-only box

# after
cfg = ContinuousBatchingConfig(async_batching=torch.cuda.is_available())

import torch
if not torch.cuda.is_available():
    cfg.async_batching = False  # or whatever field enables async IO pairs

When it happens

Trigger: Enabling async batching (async mode in ContinuousBatchingConfig) on a CPU-only machine, an Apple MPS machine, or a CUDA container without a visible GPU. The check is unconditional at __init__, before any stream is created.

Common situations: Development on laptop then deploying to GPU; CI runners without GPUs; CUDA_VISIBLE_DEVICES set to empty by a job scheduler; CPU-only torch wheel installed by mistake.

Related errors

• CUDA is not available in this environment; cannot export to
• flash_attn_with_kvcache_fn does not have a block_table or pa
• TP and DP cannot be used together
• No baseline with name '{name}' in {RESULTS_DIR}
• Audio transcription requires sequential generation (not CB)

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ab56358b46d0cb04. Report an issue: GitHub.