{"record":{"id":"622c58650b2fa68f","repo":"vllm-project/vllm","slug":"cudart-error-error-str","errorCode":null,"errorMessage":"CUDART error: {error_str}","messagePattern":"CUDART error: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/device_communicators/cuda_wrapper.py","lineNumber":140,"sourceCode":"        if so_file not in CudaRTLibrary.path_to_dict_mapping:\n            _funcs = {}\n            for func in CudaRTLibrary.exported_functions:\n                f = getattr(\n                    self.lib,\n                    CudaRTLibrary.cuda_to_hip_mapping[func.name]\n                    if current_platform.is_rocm()\n                    else func.name,\n                )\n                f.restype = func.restype\n                f.argtypes = func.argtypes\n                _funcs[func.name] = f\n            CudaRTLibrary.path_to_dict_mapping[so_file] = _funcs\n        self.funcs = CudaRTLibrary.path_to_dict_mapping[so_file]\n\n    def CUDART_CHECK(self, result: cudaError_t) -> None:\n        if result != 0:\n            error_str = self.cudaGetErrorString(result)\n            raise RuntimeError(f\"CUDART error: {error_str}\")\n\n    def cudaGetErrorString(self, error: cudaError_t) -> str:\n        return self.funcs[\"cudaGetErrorString\"](error).decode(\"utf-8\")\n\n    def cudaSetDevice(self, device: int) -> None:\n        self.CUDART_CHECK(self.funcs[\"cudaSetDevice\"](device))\n\n    def cudaDeviceSynchronize(self) -> None:\n        self.CUDART_CHECK(self.funcs[\"cudaDeviceSynchronize\"]())\n\n    def cudaDeviceReset(self) -> None:\n        self.CUDART_CHECK(self.funcs[\"cudaDeviceReset\"]())\n\n    def cudaMalloc(self, size: int) -> ctypes.c_void_p:\n        devPtr = ctypes.c_void_p()\n        self.CUDART_CHECK(self.funcs[\"cudaMalloc\"](ctypes.byref(devPtr), size))\n        return devPtr\n","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/cuda_wrapper.py#L122-L158","documentation":"CudaRTLibrary loads libcudart via ctypes and wraps each CUDA Runtime call with CUDART_CHECK. Any non-zero cudaError_t returned by cudaSetDevice, cudaDeviceSynchronize, cudaDeviceReset, etc. is translated into a RuntimeError prefixed 'CUDART error:' with the text from cudaGetErrorString. The error text after the prefix is the upstream CUDA runtime error string, so the root cause is always a failed CUDA Runtime API call.","triggerScenarios":"Calling CudaRTLibrary.cudaSetDevice(device) with an ordinal outside the visible-device range (cudaErrorInvalidDevice), cudaDeviceSynchronize() after a device fault (cudaErrorDeviceUnsuitable / cudaErrorIllegalAddress surfacing at sync), or cudaDeviceReset() while a context/stream is still in use.","commonSituations":"CUDA_VISIBLE_DEVICES or device ordinal mismatch in multi-GPU workers (rank >= visible device count); ECC or thermal hardware fault that reset the GPU mid-run; driver/toolkit mismatch after an upgrade; a prior async CUDA kernel error that only surfaces at the next runtime call.","solutions":["Run nvidia-smi and confirm the GPU is healthy and the ordinal you pass is within torch.cuda.device_count() / CUDA_VISIBLE_DEVICES","Check dmesg for NVIDIA Xid errors (Xid 79, 63, 48...) indicating a hardware or illegal-memory fault, and reinitialize the process on a clean device","Verify driver version supports the CUDA runtime vLLM was built against (nvidia-smi vs nvcc --version) and reinstall the matching build if they diverged","Wrap initialization in try/except RuntimeError to capture the error string and re-run with CUDA_LAUNCH_BLOCKING=1 to find the true faulting kernel"],"exampleFix":"// before\nlib = CudaRTLibrary()\nlib.cudaSetDevice(rank)  # rank may exceed visible devices\n\n// after\nassert rank < torch.cuda.device_count(), f\"rank {rank} >= {torch.cuda.device_count()} visible devices\"\nlib = CudaRTLibrary()\nlib.cudaSetDevice(rank)","handlingStrategy":"validation","validationCode":"import torch\nassert torch.cuda.is_available(), \"CUDA unavailable\"\nassert device < torch.cuda.device_count(), (\n    f\"device {device} >= {torch.cuda.device_count()} visible devices; check CUDA_VISIBLE_DEVICES\")","typeGuard":null,"tryCatchPattern":"try:\n    lib.cudaSetDevice(device)\nexcept RuntimeError as e:\n    if \"CUDART error\" in str(e):\n        # inspect str(e) for the cudaError string, check dmesg/Xid before retrying\n        raise","preventionTips":["Validate device ordinals against torch.cuda.device_count() before init","Run nvidia-smi as a preflight in launcher scripts","Keep driver and CUDA runtime versions aligned"],"tags":["cuda","distributed","native-bindings","runtime"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}