{"record":{"id":"39a497fdfe17c71f","repo":"sgl-project/sglang","slug":"cuda-error-int-result-0-cudageterrorstring-r","errorCode":null,"errorMessage":"CUDA error {int(result[0])}({_cudaGetErrorString(result[0])})","messagePattern":"CUDA error (.+?)\\((.+?)\\)","errorType":"panic","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/cuda_utils.py","lineNumber":40,"sourceCode":"def _cudaGetErrorString(error):\n    if rt is None:\n        return \"<cuda.bindings not available>\"\n    err, msg = rt.cudaGetErrorString(error)\n    if err != rt.cudaError_t.cudaSuccess:\n        return \"<unknown>\"\n    if isinstance(msg, bytes):\n        return msg.decode(\"utf-8\", \"replace\")\n    return str(msg)\n\n\ndef checkCudaErrors(result):\n    if rt is None:\n        raise RuntimeError(\n            \"cuda.bindings is not available. \"\n            \"Install it with: pip install cuda-python\"\n        )\n    if result[0] != rt.cudaError_t.cudaSuccess:\n        raise RuntimeError(\n            f\"CUDA error {int(result[0])}({_cudaGetErrorString(result[0])})\"\n        )\n    if len(result) == 1:\n        return None\n    elif len(result) == 2:\n        return result[1]\n    else:\n        return result[1:]\n","sourceCodeStart":22,"sourceCodeEnd":49,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/cuda_utils.py#L22-L49","documentation":"Raised by checkCudaErrors in the breakable CUDA graph runner after a cuda-python call returns a code other than cudaSuccess. It wraps the raw CUDA error code and the string from cudaGetErrorString, surfacing driver/runtime failures that occurred during graph capture, node payload setup, instantiation, or destruction. It usually indicates an underlying CUDA problem (invalid context, OOM, illegal address, driver mismatch) rather than a bug in the wrapper itself.","triggerScenarios":"Calling any of maybe_cuda_result, kernel_node_payload, graph_node_payload, graph_signature, instantiate, or destroy_exec when the underlying cuda.bindings call fails; e.g. cudaGraphInstantiate returning an error during breakable CUDA graph capture, or a CUDA async error surfacing at the next checked call.","commonSituations":"Running with CUDA graphs enabled on a GPU/driver that doesn't support needed graph features, CUDA OOM during capture, illegal memory access from a kernel captured into the graph, mismatched cuda-python vs CUDA toolkit/driver versions, or stale CUDA context after a prior error.","solutions":["Decode the numeric code against cudaError_t (e.g. 2=OOM, 700=illegal address, 701=launch timeout) and fix the root cause (reduce mem usage, fix kernel, etc.)","Reproduce with --disable-cuda-graph to see if the failure is graph-capture specific","Verify cuda-python version matches the installed CUDA driver/toolkit (pip show cuda-python; nvidia-smi)","Check dmesg / Xid errors and GPU health if code is 700/701/Xid-style","Set CUDA_LAUNCH_BLOCKING=1 to get the failure closer to the faulting kernel"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import torch\nassert torch.cuda.is_available()\nfree, total = torch.cuda.mem_get_info()\nneeded = graph_mem_estimate_bytes\nassert free > needed, f'insufficient GPU memory for graph capture: {free} < {needed}'","typeGuard":null,"tryCatchPattern":"try:\n    runner = BreakableCudaGraphRunner(...)\nexcept RuntimeError as e:\n    if 'CUDA error' in str(e):\n        torch.cuda.synchronize()  # surface any pending async error\n        log.exception('CUDA graph capture failed; disabling cuda graph')\n        server_args.disable_cuda_graph = True\n        runner = None\n    else:\n        raise","preventionTips":["Validate GPU memory and driver/CUDA compatibility before enabling CUDA graphs","Pin matching cuda-python and CUDA toolkit versions in the environment","Run a small smoke capture before long jobs to fail fast","Keep a --disable-cuda-graph fallback path in launch scripts"],"tags":["cuda","cuda-graph","gpu","driver"],"backgroundTag":"cuda-runtime-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}