{"record":{"id":"ed106d206424e315","repo":"sgl-project/sglang","slug":"indices-must-be-on-q-s-device-device-got-indic","errorCode":null,"errorMessage":"indices must be on q's device {device}, got {indices.device}","messagePattern":"indices must be on q's device (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":326,"sourceCode":"\n    s_q, h_q, d_qk = q.shape\n    s_kv, h_kv, kv_d_qk = kv.shape\n    topk = indices.shape[2]\n    device = q.device\n\n    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all\n    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():\n        raise ValueError(\"indices must be contiguous\")\n\n    if kv_d_qk != d_qk:\n        raise ValueError(f\"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}\")\n","sourceCodeStart":308,"sourceCodeEnd":344,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L308-L344","documentation":"Raised by sparse_mla_q8kv8_prefill_fwd when the indices tensor (top-k KV index tensor) lives on a different CUDA device than the query tensor q. The kernel requires all inputs on a single device because it launches a CUDA kernel directly using q's device stream, so a mismatched device would cause an illegal memory access.","triggerScenarios":"Calling sparse_mla_q8kv8_prefill_fwd(q, kv, indices, ...) where q is on cuda:0 but indices was produced or moved to cuda:1 (or CPU-then-moved incorrectly) in a multi-GPU TP setup.","commonSituations":"Tensor-parallel or pipeline-parallel pipelines where per-rank tensors are created on a fixed device while indices come from a cache on another device; device pinning via CUDA_VISIBLE_DEVICES mismatch; passing tensors from a different process/device context.","solutions":["Move indices to q's device: indices = indices.to(q.device) before the call","Verify all of q, kv, indices, topk_length, attn_sink are allocated on the same device in multi-GPU code","Check that your rank-to-device mapping (e.g. torch.cuda.set_device(local_rank)) is applied before building these tensors"],"exampleFix":"// before\nout = sparse_mla_q8kv8_prefill_fwd(q, kv, indices)  # indices on cuda:1\n// after\nindices = indices.to(q.device, non_blocking=True)\nout = sparse_mla_q8kv8_prefill_fwd(q, kv, indices)","handlingStrategy":"validation","validationCode":"assert indices.device == q.device, f\"indices on {indices.device}, q on {q.device}\"","typeGuard":"def indices_on_q_device(q: torch.Tensor, indices: torch.Tensor) -> bool:\n    return indices.is_cuda and indices.device == q.device","tryCatchPattern":"catch ValueError and re-raise with rank context: except ValueError as e: raise RuntimeError(f\"rank {rank}: {e}\") from e","preventionTips":["Standardize one device variable per rank and .to(device) every input","Set torch.cuda.set_device(local_rank) at worker startup"],"tags":["cuda","device-mismatch","sparse-attention","sglang"],"backgroundTag":"cuda-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}