{"record":{"id":"dfcaa269177af4bf","repo":"sgl-project/sglang","slug":"q-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"q must be a CUDA tensor","messagePattern":"q must be a CUDA tensor","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":317,"sourceCode":"        raise ValueError(f\"q must have shape (s_q, h_q, d_qk), got {tuple(q.shape)}\")\n    if kv.ndim != 3:\n        raise ValueError(\n            f\"kv must have shape (s_kv, h_kv, d_qk), got {tuple(kv.shape)}\"\n        )\n    if indices.ndim != 3:\n        raise ValueError(\n            \"indices must have shape (s_q, h_kv, topk), \" f\"got {tuple(indices.shape)}\"\n        )\n\n    s_q, h_q, d_qk = q.shape\n    s_kv, h_kv, kv_d_qk = kv.shape\n    topk = indices.shape[2]\n    device = q.device\n\n    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all\n    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():","sourceCodeStart":299,"sourceCodeEnd":335,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L299-L335","documentation":"The CUDA entry point launches all accesses on q's device, so sparse_mla_q8kv8_prefill_fwd requires q to be a CUDA tensor and rejects CPU (or other-backend) q with ValueError before launch.","triggerScenarios":"Calling the prefill op with q on CPU — e.g. constructing test tensors without device='cuda', or running the op in a CPU-only environment / before moving model outputs to GPU.","commonSituations":"Unit tests that build inputs on CPU (see test_q8kv8_sparse_prefill_rejects_* tests); weight-loading code that calls attention ops on CPU tensors; missing .cuda() in a prototype.","solutions":["Move q (and kv/indices) to the CUDA device before calling: q = q.cuda()","Skip/gate this op when torch.cuda.is_available() is False and use a CPU fallback"],"exampleFix":"# before\nq = torch.randn(s_q, h_q, d)  # CPU\nout = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)\n# after\nq, kv, idx = q.cuda(), kv.cuda(), idx.cuda()\nout = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)","handlingStrategy":"validation","validationCode":"q = q.cuda() if not q.is_cuda else q\nassert q.is_cuda","typeGuard":"def on_cuda(t: torch.Tensor) -> bool:\n    return t.is_cuda","tryCatchPattern":null,"preventionTips":["Build kernel inputs directly on the target device","Guard CUDA-only ops with torch.cuda.is_available() in tests"],"tags":["device-validation","cuda","cpu-tensor","sparse-mla"],"backgroundTag":"expected-cuda-tensor","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}