{"record":{"id":"c5f074aafdedafc2","repo":"sgl-project/sglang","slug":"cuda-error-err","errorCode":null,"errorMessage":"CUDA error: {err}","messagePattern":"CUDA error: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/common.py","lineNumber":1098,"sourceCode":"                \"--format=csv,noheader,nounits\",\n            ],\n            capture_output=True,\n            text=True,\n            check=True,\n            timeout=10,\n        )\n        version_str = result.stdout.strip().split(\"\\n\")[0].strip()\n        return version_str if version_str else None\n    except (subprocess.CalledProcessError, FileNotFoundError, ValueError):\n        return None\n\n\ndef check_cuda_result(raw_output):\n    import cuda.bindings.runtime as cuda_rt\n\n    err, *results = raw_output\n    if err != cuda_rt.cudaError_t.cudaSuccess:\n        raise Exception(f\"CUDA error: {err}\")\n\n    return results\n\n\ndef get_cuda_driver_bindings():\n    try:\n        from cuda.bindings import driver as cuda_driver\n    except ImportError:\n        from cuda import cuda as cuda_driver\n\n    return cuda_driver\n\n\ndef get_physical_device_id(pytorch_device_id: int) -> int:\n    \"\"\"\n    Convert PyTorch logical device ID to physical device ID.\n\n    When CUDA_VISIBLE_DEVICES is set, maps the logical device ID (as seen by PyTorch)","sourceCodeStart":1080,"sourceCodeEnd":1116,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/common.py#L1080-L1116","documentation":"Raised after a CUDA Runtime API call when the returned error code is not cudaSuccess. This is a generic wrapper used by SGLang's raw CUDA memory allocation helpers that call driver/runtime APIs directly via cuda.bindings, converting the numeric cudaError_t into an exception.","triggerScenarios":"_malloc_raw calls a cuda.bindings.runtime function (e.g. cudaMalloc) whose return is passed to check_cuda_result; any non-zero status such as cudaErrorMemoryAllocation (out of memory), cudaErrorInitializationError, or device context errors triggers it.","commonSituations":"GPU out of memory during raw buffer allocation; CUDA context not initialized or device lost (previous async error); mismatched driver/CUDA toolkit versions when using cuda.bindings.","solutions":["Decode the numeric err against cudaError_t to find the root cause (2 = out of memory, 3 = driver init failure, etc.)","If OOM: reduce tensor sizes / max-running-requests / KV cache size, or free GPU memory","Check nvidia-smi and dmesg for Xid errors / device fall-off (may need to restart the process or machine)","Verify CUDA driver version supports the runtime version bundled with cuda.bindings"],"exampleFix":"# before\nraw = cuda_rt.cudaMalloc(nbytes)  # result unchecked downstream -> opaque failure\n# after\nbuf, = check_cuda_result(cuda_rt.cudaMalloc(nbytes))  # raises 'CUDA error: 2' on OOM -> catch and retry smaller","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    buf, = check_cuda_result(cuda_rt.cudaMalloc(nbytes))\nexcept Exception as e:\n    if 'CUDA error: 2' in str(e) or 'MEMORY' in str(e).upper():\n        nbytes //= 2\n        retry_allocation(nbytes)\n    else:\n        raise","preventionTips":["Pre-check free memory with torch.cuda.mem_get_info() before large raw allocations","Retry with smaller sizes on OOM error codes","Monitor nvidia-smi/dmesg for Xid errors before they surface as opaque runtime errors"],"tags":["cuda","gpu","memory-allocation","driver"],"backgroundTag":"cuda-runtime-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}