vllm-project/vllm · error · RuntimeError

CUDART error: {error_str}

Error message

CUDART error: {error_str}

What it means

CudaRTLibrary loads libcudart via ctypes and wraps each CUDA Runtime call with CUDART_CHECK. Any non-zero cudaError_t returned by cudaSetDevice, cudaDeviceSynchronize, cudaDeviceReset, etc. is translated into a RuntimeError prefixed 'CUDART error:' with the text from cudaGetErrorString. The error text after the prefix is the upstream CUDA runtime error string, so the root cause is always a failed CUDA Runtime API call.

Source

Thrown at vllm/distributed/device_communicators/cuda_wrapper.py:140

        if so_file not in CudaRTLibrary.path_to_dict_mapping:
            _funcs = {}
            for func in CudaRTLibrary.exported_functions:
                f = getattr(
                    self.lib,
                    CudaRTLibrary.cuda_to_hip_mapping[func.name]
                    if current_platform.is_rocm()
                    else func.name,
                )
                f.restype = func.restype
                f.argtypes = func.argtypes
                _funcs[func.name] = f
            CudaRTLibrary.path_to_dict_mapping[so_file] = _funcs
        self.funcs = CudaRTLibrary.path_to_dict_mapping[so_file]

    def CUDART_CHECK(self, result: cudaError_t) -> None:
        if result != 0:
            error_str = self.cudaGetErrorString(result)
            raise RuntimeError(f"CUDART error: {error_str}")

    def cudaGetErrorString(self, error: cudaError_t) -> str:
        return self.funcs["cudaGetErrorString"](error).decode("utf-8")

    def cudaSetDevice(self, device: int) -> None:
        self.CUDART_CHECK(self.funcs["cudaSetDevice"](device))

    def cudaDeviceSynchronize(self) -> None:
        self.CUDART_CHECK(self.funcs["cudaDeviceSynchronize"]())

    def cudaDeviceReset(self) -> None:
        self.CUDART_CHECK(self.funcs["cudaDeviceReset"]())

    def cudaMalloc(self, size: int) -> ctypes.c_void_p:
        devPtr = ctypes.c_void_p()
        self.CUDART_CHECK(self.funcs["cudaMalloc"](ctypes.byref(devPtr), size))
        return devPtr

View on GitHub (pinned to c794754062)

Solutions

1. Run nvidia-smi and confirm the GPU is healthy and the ordinal you pass is within torch.cuda.device_count() / CUDA_VISIBLE_DEVICES
2. Check dmesg for NVIDIA Xid errors (Xid 79, 63, 48...) indicating a hardware or illegal-memory fault, and reinitialize the process on a clean device
3. Verify driver version supports the CUDA runtime vLLM was built against (nvidia-smi vs nvcc --version) and reinstall the matching build if they diverged
4. Wrap initialization in try/except RuntimeError to capture the error string and re-run with CUDA_LAUNCH_BLOCKING=1 to find the true faulting kernel

Example fix

// before
lib = CudaRTLibrary()
lib.cudaSetDevice(rank)  # rank may exceed visible devices

// after
assert rank < torch.cuda.device_count(), f"rank {rank} >= {torch.cuda.device_count()} visible devices"
lib = CudaRTLibrary()
lib.cudaSetDevice(rank)

import torch
assert torch.cuda.is_available(), "CUDA unavailable"
assert device < torch.cuda.device_count(), (
    f"device {device} >= {torch.cuda.device_count()} visible devices; check CUDA_VISIBLE_DEVICES")

try:
    lib.cudaSetDevice(device)
except RuntimeError as e:
    if "CUDART error" in str(e):
        # inspect str(e) for the cudaError string, check dmesg/Xid before retrying
        raise

When it happens

Trigger: Calling CudaRTLibrary.cudaSetDevice(device) with an ordinal outside the visible-device range (cudaErrorInvalidDevice), cudaDeviceSynchronize() after a device fault (cudaErrorDeviceUnsuitable / cudaErrorIllegalAddress surfacing at sync), or cudaDeviceReset() while a context/stream is still in use.

Common situations: CUDA_VISIBLE_DEVICES or device ordinal mismatch in multi-GPU workers (rank >= visible device count); ECC or thermal hardware fault that reset the GPU mid-run; driver/toolkit mismatch after an upgrade; a prior async CUDA kernel error that only surfaces at the next runtime call.

Related errors

• managed Python headless engine exited unexpectedly with stat
• failed to build request runtime
• CUDA is required for this proof.
• The `_qutlass_C` extension is not loaded. Make sure your cus
• Nested BreakableCUDAGraphCapture is not supported.

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/622c58650b2fa68f. Report an issue: GitHub.