{"record":{"id":"0fdb60950b3588f4","repo":"sgl-project/sglang","slug":"name-must-be-on-device-device-got-t-device","errorCode":null,"errorMessage":"{name} must be on device {device}, got {t.device}","messagePattern":"(.+?) must be on device (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":118,"sourceCode":"# id and must not be used here.\n_get_current_stream_raw = torch._C._cuda_getCurrentRawStream\n\n\n# Module-level cache for kernel-write-only output tensors. The active s_q rows\n# are overwritten every call; buffers grow monotonically by device/head shape.\ndef _check_out_buffer(\n    t: torch.Tensor,\n    name: str,\n    shape: tuple,\n    dtype: torch.dtype,\n    device: torch.device,\n) -> None:\n    if tuple(t.shape) != tuple(shape):\n        raise ValueError(f\"{name} must have shape {tuple(shape)}, got {tuple(t.shape)}\")\n    if t.dtype != dtype:\n        raise ValueError(f\"{name} must have dtype {dtype}, got {t.dtype}\")\n    if t.device != device:\n        raise ValueError(f\"{name} must be on device {device}, got {t.device}\")\n    if not t.is_contiguous():\n        raise ValueError(f\"{name} must be contiguous\")\n\n\n# Internal custom-op wrappers so the JIT kernel calls participate in\n# torch.library / torch.compile tracing and kernel-API debug logging.\n# The dispatch_full variant carries the optional attn_sink / topk_length\n# tensors as required args; the public API chooses which op to call.\n@register_custom_op(\n    op_name=\"sparse_mla_q8kv8_prefill\",\n    mutates_args=[\"out\", \"max_logits\", \"lse\"],\n)\ndef _sparse_mla_q8kv8_prefill_op(\n    q: torch.Tensor,\n    kv: torch.Tensor,\n    indices: torch.Tensor,\n    q_scale: torch.Tensor,\n    kv_scale: torch.Tensor,","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L100-L136","documentation":"All output buffers passed to sparse_mla_q8kv8_prefill_fwd must live on the same CUDA device the kernel launches on. _check_out_buffer compares t.device against the expected device and raises ValueError on mismatch.","triggerScenarios":"Passing out tensors allocated on a different GPU (or on CPU) than the one q/kv/indices reside on — common with cuda:0 vs cuda:1 mismatches in multi-GPU (TP/EP) inference.","commonSituations":"Tensor-parallel serving where buffers are allocated with a device string instead of the input tensors' device; multi-process rank code assuming device 0.","solutions":["Allocate output buffers with device=q.device (derive from the inputs)","Never hardcode 'cuda' or 'cuda:0' in TP/EP rank code; use the rank's device or set_device"],"exampleFix":"# before\nout = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device='cuda')  # maybe cuda:0\n# after\nout = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=q.device)","handlingStrategy":"validation","validationCode":"for buf in (out, max_logits, lse):\n    assert buf.device == q.device, (buf.device, q.device)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always allocate with device=q.device","Call torch.cuda.set_device(rank) before allocation in TP workers"],"tags":["device-validation","multi-gpu","output-buffer","sparse-mla"],"backgroundTag":"tensor-on-wrong-device","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}