{"record":{"id":"bc386f2b8576305e","repo":"sgl-project/sglang","slug":"kv-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"kv must be a CUDA tensor","messagePattern":"kv must be a CUDA tensor","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":319,"sourceCode":"        raise ValueError(\n            f\"kv must have shape (s_kv, h_kv, d_qk), got {tuple(kv.shape)}\"\n        )\n    if indices.ndim != 3:\n        raise ValueError(\n            \"indices must have shape (s_q, h_kv, topk), \" f\"got {tuple(indices.shape)}\"\n        )\n\n    s_q, h_q, d_qk = q.shape\n    s_kv, h_kv, kv_d_qk = kv.shape\n    topk = indices.shape[2]\n    device = q.device\n\n    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all\n    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():","sourceCodeStart":301,"sourceCodeEnd":337,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L301-L337","documentation":"kv must be a CUDA tensor for the SM90 sparse prefill kernel; a CPU kv raises ValueError at the contract check (sparse_mla_q8kv8_prefill_sm90.py:319) before any launch.","triggerScenarios":"Mixing devices: q on GPU but kv still on CPU (or on a meta/other-backend tensor) when calling sparse_mla_q8kv8_prefill_fwd.","commonSituations":"Quantizing/preparing kv on CPU (quantization tooling) and forgetting the final .cuda(); partial .to(device) migrations of a pipeline.","solutions":["Move kv to the same CUDA device as q before the call","Ensure the fp8 quantization step for kv runs on GPU or ends with .to(q.device)"],"exampleFix":"# before\nkv_fp8 = quantize(kv_cpu)  # CPU\n# after\nkv_fp8 = quantize(kv_cpu).to(q.device)","handlingStrategy":"validation","validationCode":"kv = kv.to(q.device)\nassert kv.is_cuda","typeGuard":null,"tryCatchPattern":null,"preventionTips":["End every CPU preprocessing step with .to(q.device)","Run fp8 quantization for kv on the GPU"],"tags":["device-validation","cuda","mixed-device","sparse-mla"],"backgroundTag":"expected-cuda-tensor","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}