{"record":{"id":"b6f2d9481a5070d6","repo":"sgl-project/sglang","slug":"indices-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"indices must be a CUDA tensor","messagePattern":"indices must be a CUDA tensor","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":321,"sourceCode":"        )\n    if indices.ndim != 3:\n        raise ValueError(\n            \"indices must have shape (s_q, h_kv, topk), \" f\"got {tuple(indices.shape)}\"\n        )\n\n    s_q, h_q, d_qk = q.shape\n    s_kv, h_kv, kv_d_qk = kv.shape\n    topk = indices.shape[2]\n    device = q.device\n\n    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all\n    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():","sourceCodeStart":303,"sourceCodeEnd":339,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L303-L339","documentation":"indices must also be a CUDA tensor; the kernel reads selected KV offsets directly from GPU memory, so CPU indices are rejected with ValueError before launch.","triggerScenarios":"q/kv on GPU but the top-k indices tensor still on CPU — e.g. indices computed by a numpy/scipy selection step and not moved to GPU.","commonSituations":"CPU-side top-k heuristics (numpy argsort) feeding the sparse prefill; exporting/importing indices from disk and loading without device pinning.","solutions":["Move indices to GPU: indices = indices.to(q.device, non_blocking=True)","Compute top-k selection with torch.topk on GPU so indices are already resident"],"exampleFix":"# before\nidx = np.argsort(scores)[:, -topk:]  # numpy -> CPU tensor\nidx = torch.from_numpy(idx)\n# after\nidx = torch.topk(scores_gpu, topk, dim=-1).indices  # already CUDA\nout = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)","handlingStrategy":"validation","validationCode":"idx = idx.to(q.device, non_blocking=True)\nassert idx.is_cuda and idx.device == q.device","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Compute top-k with torch.topk on GPU","Move any numpy-derived indices to the kernel device before launch"],"tags":["device-validation","cuda","indices","sparse-mla"],"backgroundTag":"expected-cuda-tensor","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}