sgl-project/sglang · critical · RuntimeError

CUDA error {int(result[0])}({_cudaGetErrorString(result[0])}

Error message

CUDA error {int(result[0])}({_cudaGetErrorString(result[0])})

What it means

Raised by checkCudaErrors in the breakable CUDA graph runner after a cuda-python call returns a code other than cudaSuccess. It wraps the raw CUDA error code and the string from cudaGetErrorString, surfacing driver/runtime failures that occurred during graph capture, node payload setup, instantiation, or destruction. It usually indicates an underlying CUDA problem (invalid context, OOM, illegal address, driver mismatch) rather than a bug in the wrapper itself.

Source

Thrown at python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/cuda_utils.py:40

def _cudaGetErrorString(error):
    if rt is None:
        return "<cuda.bindings not available>"
    err, msg = rt.cudaGetErrorString(error)
    if err != rt.cudaError_t.cudaSuccess:
        return "<unknown>"
    if isinstance(msg, bytes):
        return msg.decode("utf-8", "replace")
    return str(msg)


def checkCudaErrors(result):
    if rt is None:
        raise RuntimeError(
            "cuda.bindings is not available. "
            "Install it with: pip install cuda-python"
        )
    if result[0] != rt.cudaError_t.cudaSuccess:
        raise RuntimeError(
            f"CUDA error {int(result[0])}({_cudaGetErrorString(result[0])})"
        )
    if len(result) == 1:
        return None
    elif len(result) == 2:
        return result[1]
    else:
        return result[1:]

View on GitHub (pinned to 0132848349)

Solutions

1. Decode the numeric code against cudaError_t (e.g. 2=OOM, 700=illegal address, 701=launch timeout) and fix the root cause (reduce mem usage, fix kernel, etc.)
2. Reproduce with --disable-cuda-graph to see if the failure is graph-capture specific
3. Verify cuda-python version matches the installed CUDA driver/toolkit (pip show cuda-python; nvidia-smi)
4. Check dmesg / Xid errors and GPU health if code is 700/701/Xid-style
5. Set CUDA_LAUNCH_BLOCKING=1 to get the failure closer to the faulting kernel

import torch
assert torch.cuda.is_available()
free, total = torch.cuda.mem_get_info()
needed = graph_mem_estimate_bytes
assert free > needed, f'insufficient GPU memory for graph capture: {free} < {needed}'

try:
    runner = BreakableCudaGraphRunner(...)
except RuntimeError as e:
    if 'CUDA error' in str(e):
        torch.cuda.synchronize()  # surface any pending async error
        log.exception('CUDA graph capture failed; disabling cuda graph')
        server_args.disable_cuda_graph = True
        runner = None
    else:
        raise

When it happens

Trigger: Calling any of maybe_cuda_result, kernel_node_payload, graph_node_payload, graph_signature, instantiate, or destroy_exec when the underlying cuda.bindings call fails; e.g. cudaGraphInstantiate returning an error during breakable CUDA graph capture, or a CUDA async error surfacing at the next checked call.

Common situations: Running with CUDA graphs enabled on a GPU/driver that doesn't support needed graph features, CUDA OOM during capture, illegal memory access from a kernel captured into the graph, mismatched cuda-python vs CUDA toolkit/driver versions, or stale CUDA context after a prior error.

Related errors

• CUDA error: {err}
• CUDART error: {error_str}
• Invalid graph capture input size: {nbytes}
• cuMemGetAddressRange: {err}
• graph capture input at {ptr} is outside VMM allocation [base

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/39a497fdfe17c71f. Report an issue: GitHub.