{"record":{"id":"3822ffccee4659e0","repo":"sgl-project/sglang","slug":"kv-must-be-on-q-s-device-device-got-kv-device","errorCode":null,"errorMessage":"kv must be on q's device {device}, got {kv.device}","messagePattern":"kv must be on q's device (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":324,"sourceCode":"            \"indices must have shape (s_q, h_kv, topk), \" f\"got {tuple(indices.shape)}\"\n        )\n\n    s_q, h_q, d_qk = q.shape\n    s_kv, h_kv, kv_d_qk = kv.shape\n    topk = indices.shape[2]\n    device = q.device\n\n    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all\n    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():\n        raise ValueError(\"indices must be contiguous\")\n\n    if kv_d_qk != d_qk:","sourceCodeStart":306,"sourceCodeEnd":342,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L306-L342","documentation":"Beyond being CUDA tensors, kv and indices must be on the same device as q, because entry.cuh launches on q's device. A kv on a different GPU raises ValueError with both devices named.","triggerScenarios":"q on cuda:1 (e.g. TP rank 1) while kv was allocated on cuda:0 or with device='cuda' defaulting to the current device in a multi-GPU process.","commonSituations":"Tensor/data-parallel inference where buffers are allocated before torch.cuda.set_device(rank); code that hardcodes 'cuda:0'; NCCL pipelines copying inputs to the wrong rank's GPU.","solutions":["Allocate/derive every input from q.device: kv = kv.to(q.device)","Call torch.cuda.set_device(rank) before any tensor allocation in multi-GPU workers"],"exampleFix":"# before\nkv = torch.empty(..., device='cuda')  # defaults to cuda:0\n# after\nkv = torch.empty(..., device=q.device)  # matches q's GPU\nout = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)","handlingStrategy":"validation","validationCode":"assert kv.device == q.device and idx.device == q.device, \\\n    (q.device, kv.device, idx.device)","typeGuard":"def all_on_same_device(*ts: torch.Tensor) -> bool:\n    return len({t.device for t in ts}) == 1 and ts[0].is_cuda","tryCatchPattern":null,"preventionTips":["Derive all tensors from q.device, never hardcode 'cuda'/'cuda:0'","Call torch.cuda.set_device(rank) early in each TP/DP worker"],"tags":["device-validation","multi-gpu","tensor-parallel","sparse-mla"],"backgroundTag":"tensor-on-wrong-device","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}