{"record":{"id":"f6a2865b285957d3","repo":"sgl-project/sglang","slug":"cudart-error-error-str","errorCode":null,"errorMessage":"CUDART error: {error_str}","messagePattern":"CUDART error: (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/distributed/device_communicators/cuda_wrapper.py","lineNumber":149,"sourceCode":"        if so_file not in CudaRTLibrary.path_to_library_cache:\n            lib = ctypes.CDLL(so_file)\n            CudaRTLibrary.path_to_library_cache[so_file] = lib\n        self.lib = CudaRTLibrary.path_to_library_cache[so_file]\n\n        if so_file not in CudaRTLibrary.path_to_dict_mapping:\n            _funcs = {}\n            for func in CudaRTLibrary.exported_functions:\n                f = getattr(self.lib, func.name)\n                f.restype = func.restype\n                f.argtypes = func.argtypes\n                _funcs[func.name] = f\n            CudaRTLibrary.path_to_dict_mapping[so_file] = _funcs\n        self.funcs = CudaRTLibrary.path_to_dict_mapping[so_file]\n\n    def CUDART_CHECK(self, result: cudaError_t) -> None:\n        if result != 0:\n            error_str = self.cudaGetErrorString(result)\n            raise RuntimeError(f\"CUDART error: {error_str}\")\n\n    def cudaGetErrorString(self, error: cudaError_t) -> str:\n        return self.funcs[\"cudaGetErrorString\"](error).decode(\"utf-8\")\n\n    def cudaSetDevice(self, device: int) -> None:\n        self.CUDART_CHECK(self.funcs[\"cudaSetDevice\"](device))\n\n    def cudaDeviceSynchronize(self) -> None:\n        self.CUDART_CHECK(self.funcs[\"cudaDeviceSynchronize\"]())\n\n    def cudaDeviceReset(self) -> None:\n        self.CUDART_CHECK(self.funcs[\"cudaDeviceReset\"]())\n\n    def cudaMalloc(self, size: int) -> ctypes.c_void_p:\n        devPtr = ctypes.c_void_p()\n        self.CUDART_CHECK(self.funcs[\"cudaMalloc\"](ctypes.byref(devPtr), size))\n        return devPtr\n","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/device_communicators/cuda_wrapper.py#L131-L167","documentation":"A CUDA Runtime API call (cudaSetDevice, cudaMalloc, cudaFree, cudaMemset, cudaDeviceSynchronize, cudaDeviceReset) returned a non-zero cudaError_t code, and this wrapper raised RuntimeError with the decoded error string from cudaGetErrorString. This is sglang's ctypes binding of libcudart, so the message text comes straight from the CUDA runtime. Common codes include out-of-memory (cudaErrorMemoryAllocation), invalid device ordinal (cudaErrorInvalidDevice), and driver/runtime mismatch.","triggerScenarios":"Calling any of CudaRTLibrary's cudaSetDevice, cudaDeviceSynchronize, cudaDeviceReset, cudaMalloc, cudaFree, or cudaMemset wrappers when the underlying CUDA runtime call fails, e.g. cudaSetDevice with an ordinal >= device count, or cudaMalloc when GPU memory is exhausted.","commonSituations":"Passing a device id that exceeds the number of visible GPUs (CUDA_VISIBLE_DEVICES misconfiguration), allocating more memory than free VRAM during KV cache / weight allocation, CUDA driver version older than the runtime shipped with the PyTorch build, or a prior asynchronous error surfacing on the next runtime call.","solutions":["Decode the error string (e.g. 'out of memory' vs 'invalid device ordinal') and address that specific CUDA condition","Verify CUDA_VISIBLE_DEVICES and that device ids passed to cudaSetDevice are < torch.cuda.device_count()","Check nvidia-smi for free memory and driver version; free VRAM or lower --mem-fraction-static / batch size before retrying the allocating call","If a driver/runtime mismatch is reported, upgrade the NVIDIA driver to match the CUDA version of your PyTorch/sglang build"],"exampleFix":"// before\nlib.CudaRTLibrary().cudaSetDevice(7)  # only 4 GPUs visible\n\n// after\nlib = CudaRTLibrary()\nassert device_id < torch.cuda.device_count(), f\"device {device_id} not visible\"\nlib.cudaSetDevice(device_id)","handlingStrategy":"try-catch","validationCode":"import torch\ndef assert_device_ok(device_id: int) -> None:\n    n = torch.cuda.device_count()\n    if device_id < 0 or device_id >= n:\n        raise ValueError(f\"device {device_id} out of range (count={n})\")\n\ndef free_mem_gb() -> float:\n    free, _ = torch.cuda.mem_get_info()\n    return free / 1024**3","typeGuard":null,"tryCatchPattern":"try:\n    lib.cudaSetDevice(device_id)\nexcept RuntimeError as e:\n    if \"invalid device ordinal\" in str(e):\n        raise ValueError(f\"bad device id {device_id}; check CUDA_VISIBLE_DEVICES\") from e\n    raise  # OOM and driver errors are fatal; surface them","preventionTips":["Validate device ids against torch.cuda.device_count() before any CUDA call","Log free VRAM (torch.cuda.mem_get_info) before large cudaMalloc-driven allocations","Keep the NVIDIA driver version >= the CUDA version of your PyTorch build"],"tags":["cuda","gpu","out-of-memory","invalid-device","runtime"],"backgroundTag":"cuda-runtime-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}