{"record":{"id":"dad33b6f9b731fb4","repo":"sgl-project/sglang","slug":"label-err","errorCode":null,"errorMessage":"{label}: {err}","messagePattern":"\\{label\\}: \\{err\\}","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":68,"sourceCode":"# NVML_GPU_FABRIC_STATE_COMPLETED: the GPU has joined its NVLink fabric clique.\n_NVML_GPU_FABRIC_STATE_COMPLETED = 3\n\n\ndef _get_cuda_driver():\n    \"\"\"Return the imported CUDA driver bindings.\"\"\"\n    if _drv is None:\n        raise ImportError(\"cuda.bindings.driver is required for CUDA VMM operations\")\n    return _drv\n\n\ndef check_drv(result_tuple, label):\n    \"\"\"Check a cuda.bindings driver call result and return the value.\"\"\"\n    if not isinstance(result_tuple, tuple):\n        result_tuple = (result_tuple,)\n    err = result_tuple[0]\n    drv = _get_cuda_driver()\n    if err != drv.CUresult.CUDA_SUCCESS:\n        raise RuntimeError(f\"{label}: {err}\")\n    return result_tuple[1] if len(result_tuple) > 1 else None\n\n\ndef tensor_from_pointer(\n    pointer: int,\n    nbytes: int,\n    *,\n    shape=None,\n    dtype: torch.dtype = torch.uint8,\n    device_id: int,\n) -> torch.Tensor:\n    \"\"\"Use non-owning storage; the caller controls the underlying pages' lifetime.\"\"\"\n    device = torch.device(\"cuda\", device_id)\n    storage = torch._C._construct_storage_from_data_pointer(pointer, device, nbytes)\n    if shape is None:\n        shape = (nbytes,)\n    return torch.empty(0, dtype=dtype, device=device).set_(storage, 0, shape)\n","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L50-L86","documentation":"check_drv wraps every cuda.bindings.driver call: when the returned CUresult is not CUDA_SUCCESS it raises RuntimeError with the operation label and the raw CUresult code. This is the generic 'CUDA driver API call failed' surface for all VMM operations (map/unmap, release, stream memops, handle export).","triggerScenarios":"Any driver call failing: cuMemRelease on an already-freed handle (CUDA_ERROR_INVALID_HANDLE), cuStreamWaitValue32 on an invalid stream, allocation on an out-of-memory device, fabric handle export on unsupported fabric.","commonSituations":"Driver/context teardown order bugs (releasing after context destroy), OOM during VMM allocation, unsupported platform for the requested handle type, driver version too old for VMM APIs.","solutions":["Decode the CUresult in the message (e.g. CUDA_ERROR_INVALID_VALUE=1, CUDA_ERROR_OUT_OF_MEMORY=2, CUDA_ERROR_INVALID_HANDLE=400) to find the specific cause","Check driver version with nvidia-smi — VMM (cuMem*) requires a reasonably recent driver","For handle errors, audit object lifetimes: ensure the handle/pointer isn't released twice or used after free"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from cuda.bindings import driver as drv\ntry:\n    handle = check_drv(drv.cuMemRelease(ptr), \"cuMemRelease\")\nexcept RuntimeError as e:\n    if \"CUDA_ERROR_INVALID_HANDLE\" in str(e):\n        pass  # already released\n    else:\n        raise","preventionTips":["Check driver version before using VMM features","Never use handles after release; audit lifetimes on teardown changes"],"tags":["cuda","driver-api","vmm","error-code"],"backgroundTag":"cuda-driver-api-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}