{"record":{"id":"5d9da259c68eb6c4","repo":"sgl-project/sglang","slug":"triton-sparse-attn-v4-paged-prefill-requires-cuda","errorCode":null,"errorMessage":"Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors","messagePattern":"Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py","lineNumber":229,"sourceCode":"        + d_offs[None, :] * out_stride_d,\n        out,\n        mask=h_mask[:, None] & d_mask[None, :],\n    )\n\n\ndef _sparse_attn_v4_paged_prefill_triton(\n    q: torch.Tensor,\n    unified_kv: torch.Tensor,\n    kv_indices_prefix: torch.Tensor,\n    kv_indptr_prefix: torch.Tensor,\n    kv: torch.Tensor,\n    kv_indices_extend: torch.Tensor,\n    kv_indptr_extend: torch.Tensor,\n    attn_sink: torch.Tensor,\n    softmax_scale: float,\n) -> torch.Tensor:\n    if not q.is_cuda:\n        raise RuntimeError(\n            \"Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors\"\n        )\n    if q.dtype not in (torch.bfloat16, torch.float16):\n        raise RuntimeError(\n            f\"sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dtype}\"\n        )\n    if unified_kv.dtype != q.dtype:\n        raise RuntimeError(\n            f\"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}\"\n        )\n    if kv.dtype != q.dtype:\n        raise RuntimeError(f\"kv dtype mismatch: kv={kv.dtype}, q={q.dtype}\")\n    if unified_kv.size(-1) != kv.size(-1):\n        raise RuntimeError(\n            f\"head_dim mismatch: unified_kv={unified_kv.size(-1)}, kv={kv.size(-1)}\"\n        )\n\n    T, H, D = q.shape","sourceCodeStart":211,"sourceCodeEnd":247,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py#L211-L247","documentation":"The Triton sparse DeepSeek-V4 paged prefill attention kernel is CUDA/HIP-only; the wrapper detects the device via q.is_cuda and refuses CPU tensors so it never reaches Triton, which cannot launch on CPU.","triggerScenarios":"Calling sparse_attn_v4_paged_prefill with any of the tensors (checked via q) on CPU, e.g. in a unit test or a CPU-only debug run.","commonSituations":"Unit tests with CPU fixtures; forgetting .cuda() after building tensors; running on a machine without a GPU; device mismatch where q is CPU but the cache is on GPU.","solutions":["Move all tensors to the CUDA device: q = q.cuda() (and likewise for unified_kv, kv, indices)","If you need a CPU reference path, branch to the torch reference implementation instead of the Triton kernel","Ensure the environment actually has a GPU (torch.cuda.is_available()) before dispatching"],"exampleFix":"// before\nout = sparse_attn_v4_paged_prefill(q, kv, ...)  # q on CPU\n// after\nout = sparse_attn_v4_paged_prefill(q.cuda(), kv.cuda(), ...)","handlingStrategy":"type-guard","validationCode":"assert q.is_cuda, \"sparse_attn_v4_paged_prefill requires CUDA tensors\"","typeGuard":"def on_gpu(*ts) -> bool:\n    return all(t.is_cuda for t in ts if isinstance(t, torch.Tensor))","tryCatchPattern":null,"preventionTips":["Skip GPU-kernel tests when torch.cuda.is_available() is False","Move every tensor to the same device at the start of the forward call"],"tags":["attention","triton","device","cpu-vs-gpu"],"backgroundTag":"tensor-not-on-gpu","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}