sgl-project/sglang · critical · RuntimeError

CUDART error: {error_str}

Error message

CUDART error: {error_str}

What it means

A CUDA Runtime API call (cudaSetDevice, cudaMalloc, cudaFree, cudaMemset, cudaDeviceSynchronize, cudaDeviceReset) returned a non-zero cudaError_t code, and this wrapper raised RuntimeError with the decoded error string from cudaGetErrorString. This is sglang's ctypes binding of libcudart, so the message text comes straight from the CUDA runtime. Common codes include out-of-memory (cudaErrorMemoryAllocation), invalid device ordinal (cudaErrorInvalidDevice), and driver/runtime mismatch.

Source

Thrown at python/sglang/srt/distributed/device_communicators/cuda_wrapper.py:149

        if so_file not in CudaRTLibrary.path_to_library_cache:
            lib = ctypes.CDLL(so_file)
            CudaRTLibrary.path_to_library_cache[so_file] = lib
        self.lib = CudaRTLibrary.path_to_library_cache[so_file]

        if so_file not in CudaRTLibrary.path_to_dict_mapping:
            _funcs = {}
            for func in CudaRTLibrary.exported_functions:
                f = getattr(self.lib, func.name)
                f.restype = func.restype
                f.argtypes = func.argtypes
                _funcs[func.name] = f
            CudaRTLibrary.path_to_dict_mapping[so_file] = _funcs
        self.funcs = CudaRTLibrary.path_to_dict_mapping[so_file]

    def CUDART_CHECK(self, result: cudaError_t) -> None:
        if result != 0:
            error_str = self.cudaGetErrorString(result)
            raise RuntimeError(f"CUDART error: {error_str}")

    def cudaGetErrorString(self, error: cudaError_t) -> str:
        return self.funcs["cudaGetErrorString"](error).decode("utf-8")

    def cudaSetDevice(self, device: int) -> None:
        self.CUDART_CHECK(self.funcs["cudaSetDevice"](device))

    def cudaDeviceSynchronize(self) -> None:
        self.CUDART_CHECK(self.funcs["cudaDeviceSynchronize"]())

    def cudaDeviceReset(self) -> None:
        self.CUDART_CHECK(self.funcs["cudaDeviceReset"]())

    def cudaMalloc(self, size: int) -> ctypes.c_void_p:
        devPtr = ctypes.c_void_p()
        self.CUDART_CHECK(self.funcs["cudaMalloc"](ctypes.byref(devPtr), size))
        return devPtr

View on GitHub (pinned to 0132848349)

Solutions

1. Decode the error string (e.g. 'out of memory' vs 'invalid device ordinal') and address that specific CUDA condition
2. Verify CUDA_VISIBLE_DEVICES and that device ids passed to cudaSetDevice are < torch.cuda.device_count()
3. Check nvidia-smi for free memory and driver version; free VRAM or lower --mem-fraction-static / batch size before retrying the allocating call
4. If a driver/runtime mismatch is reported, upgrade the NVIDIA driver to match the CUDA version of your PyTorch/sglang build

Example fix

// before
lib.CudaRTLibrary().cudaSetDevice(7)  # only 4 GPUs visible

// after
lib = CudaRTLibrary()
assert device_id < torch.cuda.device_count(), f"device {device_id} not visible"
lib.cudaSetDevice(device_id)

import torch
def assert_device_ok(device_id: int) -> None:
    n = torch.cuda.device_count()
    if device_id < 0 or device_id >= n:
        raise ValueError(f"device {device_id} out of range (count={n})")

def free_mem_gb() -> float:
    free, _ = torch.cuda.mem_get_info()
    return free / 1024**3

try:
    lib.cudaSetDevice(device_id)
except RuntimeError as e:
    if "invalid device ordinal" in str(e):
        raise ValueError(f"bad device id {device_id}; check CUDA_VISIBLE_DEVICES") from e
    raise  # OOM and driver errors are fatal; surface them

When it happens

Trigger: Calling any of CudaRTLibrary's cudaSetDevice, cudaDeviceSynchronize, cudaDeviceReset, cudaMalloc, cudaFree, or cudaMemset wrappers when the underlying CUDA runtime call fails, e.g. cudaSetDevice with an ordinal >= device count, or cudaMalloc when GPU memory is exhausted.

Common situations: Passing a device id that exceeds the number of visible GPUs (CUDA_VISIBLE_DEVICES misconfiguration), allocating more memory than free VRAM during KV cache / weight allocation, CUDA driver version older than the runtime shipped with the PyTorch build, or a prior asynchronous error surfacing on the next runtime call.

Related errors

• CUDA error {int(result[0])}({_cudaGetErrorString(result[0])}
• CUDA error: {err}
• Invalid device_uuid=
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f6a2865b285957d3. Report an issue: GitHub.