sgl-project/sglang · critical · Exception

CUDA error: {err}

Error message

CUDA error: {err}

What it means

Raised after a CUDA Runtime API call when the returned error code is not cudaSuccess. This is a generic wrapper used by SGLang's raw CUDA memory allocation helpers that call driver/runtime APIs directly via cuda.bindings, converting the numeric cudaError_t into an exception.

Source

Thrown at python/sglang/srt/utils/common.py:1098

                "--format=csv,noheader,nounits",
            ],
            capture_output=True,
            text=True,
            check=True,
            timeout=10,
        )
        version_str = result.stdout.strip().split("\n")[0].strip()
        return version_str if version_str else None
    except (subprocess.CalledProcessError, FileNotFoundError, ValueError):
        return None


def check_cuda_result(raw_output):
    import cuda.bindings.runtime as cuda_rt

    err, *results = raw_output
    if err != cuda_rt.cudaError_t.cudaSuccess:
        raise Exception(f"CUDA error: {err}")

    return results


def get_cuda_driver_bindings():
    try:
        from cuda.bindings import driver as cuda_driver
    except ImportError:
        from cuda import cuda as cuda_driver

    return cuda_driver


def get_physical_device_id(pytorch_device_id: int) -> int:
    """
    Convert PyTorch logical device ID to physical device ID.

    When CUDA_VISIBLE_DEVICES is set, maps the logical device ID (as seen by PyTorch)

View on GitHub (pinned to 0132848349)

Solutions

1. Decode the numeric err against cudaError_t to find the root cause (2 = out of memory, 3 = driver init failure, etc.)
2. If OOM: reduce tensor sizes / max-running-requests / KV cache size, or free GPU memory
3. Check nvidia-smi and dmesg for Xid errors / device fall-off (may need to restart the process or machine)
4. Verify CUDA driver version supports the runtime version bundled with cuda.bindings

Example fix

# before
raw = cuda_rt.cudaMalloc(nbytes)  # result unchecked downstream -> opaque failure
# after
buf, = check_cuda_result(cuda_rt.cudaMalloc(nbytes))  # raises 'CUDA error: 2' on OOM -> catch and retry smaller

try:
    buf, = check_cuda_result(cuda_rt.cudaMalloc(nbytes))
except Exception as e:
    if 'CUDA error: 2' in str(e) or 'MEMORY' in str(e).upper():
        nbytes //= 2
        retry_allocation(nbytes)
    else:
        raise

When it happens

Trigger: _malloc_raw calls a cuda.bindings.runtime function (e.g. cudaMalloc) whose return is passed to check_cuda_result; any non-zero status such as cudaErrorMemoryAllocation (out of memory), cudaErrorInitializationError, or device context errors triggers it.

Common situations: GPU out of memory during raw buffer allocation; CUDA context not initialized or device lost (previous async error); mismatched driver/CUDA toolkit versions when using cuda.bindings.

Related errors

• CUDA error {int(result[0])}({_cudaGetErrorString(result[0])}
• CUDART error: {error_str}
• memory_size={memory_size} is smaller than CUDA VMM granulari
• Invalid device_uuid=
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c5f074aafdedafc2. Report an issue: GitHub.