sgl-project/sglang · error · RuntimeError

{label}: {err}

Error message

{label}: {err}

What it means

check_drv wraps every cuda.bindings.driver call: when the returned CUresult is not CUDA_SUCCESS it raises RuntimeError with the operation label and the raw CUresult code. This is the generic 'CUDA driver API call failed' surface for all VMM operations (map/unmap, release, stream memops, handle export).

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:68

# NVML_GPU_FABRIC_STATE_COMPLETED: the GPU has joined its NVLink fabric clique.
_NVML_GPU_FABRIC_STATE_COMPLETED = 3


def _get_cuda_driver():
    """Return the imported CUDA driver bindings."""
    if _drv is None:
        raise ImportError("cuda.bindings.driver is required for CUDA VMM operations")
    return _drv


def check_drv(result_tuple, label):
    """Check a cuda.bindings driver call result and return the value."""
    if not isinstance(result_tuple, tuple):
        result_tuple = (result_tuple,)
    err = result_tuple[0]
    drv = _get_cuda_driver()
    if err != drv.CUresult.CUDA_SUCCESS:
        raise RuntimeError(f"{label}: {err}")
    return result_tuple[1] if len(result_tuple) > 1 else None


def tensor_from_pointer(
    pointer: int,
    nbytes: int,
    *,
    shape=None,
    dtype: torch.dtype = torch.uint8,
    device_id: int,
) -> torch.Tensor:
    """Use non-owning storage; the caller controls the underlying pages' lifetime."""
    device = torch.device("cuda", device_id)
    storage = torch._C._construct_storage_from_data_pointer(pointer, device, nbytes)
    if shape is None:
        shape = (nbytes,)
    return torch.empty(0, dtype=dtype, device=device).set_(storage, 0, shape)

View on GitHub (pinned to 0132848349)

Solutions

1. Decode the CUresult in the message (e.g. CUDA_ERROR_INVALID_VALUE=1, CUDA_ERROR_OUT_OF_MEMORY=2, CUDA_ERROR_INVALID_HANDLE=400) to find the specific cause
2. Check driver version with nvidia-smi — VMM (cuMem*) requires a reasonably recent driver
3. For handle errors, audit object lifetimes: ensure the handle/pointer isn't released twice or used after free

from cuda.bindings import driver as drv
try:
    handle = check_drv(drv.cuMemRelease(ptr), "cuMemRelease")
except RuntimeError as e:
    if "CUDA_ERROR_INVALID_HANDLE" in str(e):
        pass  # already released
    else:
        raise

When it happens

Trigger: Any driver call failing: cuMemRelease on an already-freed handle (CUDA_ERROR_INVALID_HANDLE), cuStreamWaitValue32 on an invalid stream, allocation on an out-of-memory device, fabric handle export on unsupported fabric.

Common situations: Driver/context teardown order bugs (releasing after context destroy), OOM during VMM allocation, unsupported platform for the requested handle type, driver version too old for VMM APIs.

Related errors

• CUDA VMM POSIX FD broker failed
• CUDA VMM POSIX FD broker returned no file descriptor
• memory_size must be positive
• consumer_count must be positive
• recycle_interval must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/dad33b6f9b731fb4. Report an issue: GitHub.