{"record":{"id":"b14501453a6296cb","repo":"sgl-project/sglang","slug":"invalid-device-uuid","errorCode":null,"errorMessage":"Invalid device_uuid=","messagePattern":"Invalid device_uuid=","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/patch_torch.py","lineNumber":97,"sourceCode":"\ndef _rebuild_cuda_tensor_modified(*args):\n    args = _modify_tuple(args, _REDUCE_TENSOR_ARG_DEVICE_INDEX, _device_from_maybe_uuid)\n    return reductions._rebuild_cuda_tensor_original(*args)\n\n\ndef _device_to_uuid(device: int) -> str:\n    return str(torch.cuda.get_device_properties(device).uuid)\n\n\ndef _device_from_maybe_uuid(device_maybe_uuid: Union[int, str]) -> int:\n    if isinstance(device_maybe_uuid, int):\n        return device_maybe_uuid\n\n    if isinstance(device_maybe_uuid, str):\n        for device in range(torch.cuda.device_count()):\n            if str(torch.cuda.get_device_properties(device).uuid) == device_maybe_uuid:\n                return device\n        raise Exception(\"Invalid device_uuid=\" + device_maybe_uuid)\n\n    raise Exception(f\"Unknown type: {device_maybe_uuid=}\")\n\n\ndef _modify_tuple(t, index: int, modifier: Callable):\n    return *t[:index], modifier(t[index]), *t[index + 1 :]\n\n\ndef monkey_patch_torch_compile():\n    if torch_release < (2, 8):\n        # These things are cacheable by torch.compile. torch.compile just doesn't know it.\n        # This was fixed in PyTorch 2.8, but until then, we monkey patch.\n        import torch._higher_order_ops.auto_functionalize as af\n\n        af.auto_functionalized_v2._cacheable = True\n        af.auto_functionalized._cacheable = True\n\n","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/patch_torch.py#L79-L115","documentation":"Raised by the torch patching layer when a string passed as a device identifier does not match any visible CUDA device's UUID. The patch iterates torch.cuda.device_count() devices comparing str(props.uuid) to the input; no match means the UUID is malformed or the GPU is not visible to this process.","triggerScenarios":"Passing device_uuid='GPU-<wrong-or-clipped-uuid>' via config (e.g. CUDA_VISIBLE_DEVICES UUID form or a server arg) while the referenced GPU is masked by CUDA_VISIBLE_DEVICES by index, or the UUID string is truncated/malformed.","commonSituations":"Multi-node clusters addressing GPUs by UUID; NVIDIA MIG instances; typo'd or copy-truncated UUIDs; GPUs hidden by cgroup/device visibility.","solutions":["Verify the UUID with nvidia-smi -L and paste the full 'GPU-xxxxxxxx-...' string","Ensure the target GPU is visible: unset/fix CUDA_VISIBLE_DEVICES for the process","Use a plain integer device index instead of a UUID"],"exampleFix":"# before\ndevice_uuid='GPU-abc123'  # typo/truncated\n# after\ndevice_uuid='GPU-3d2f7a1b-09c4-11ee-be56-0242ac120002'  # full UUID from nvidia-smi -L","handlingStrategy":"validation","validationCode":"import subprocess\ndef uuid_exists(uuid: str) -> bool:\n    out = subprocess.run(['nvidia-smi','-L'], capture_output=True, text=True).stdout\n    return uuid in out","typeGuard":"def is_valid_device_spec(v) -> bool:\n    return isinstance(v, (int, str)) or v is None","tryCatchPattern":"try:\n    idx = _device_from_maybe_uuid(uuid)\nexcept Exception as e:\n    if 'Invalid device_uuid' in str(e):\n        idx = 0  # fallback device","preventionTips":["Copy full GPU-... UUIDs from nvidia-smi -L","Verify GPU visibility (CUDA_VISIBLE_DEVICES) before using UUID addressing","Prefer integer device indices in scripts"],"tags":["cuda","device-uuid","gpu","torch-patch","sglang"],"backgroundTag":"invalid-device-identifier","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}