sgl-project/sglang · error · RuntimeError

Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensor

Error message

Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors

What it means

The Triton sparse DeepSeek-V4 paged prefill attention kernel is CUDA/HIP-only; the wrapper detects the device via q.is_cuda and refuses CPU tensors so it never reaches Triton, which cannot launch on CPU.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py:229

        + d_offs[None, :] * out_stride_d,
        out,
        mask=h_mask[:, None] & d_mask[None, :],
    )


def _sparse_attn_v4_paged_prefill_triton(
    q: torch.Tensor,
    unified_kv: torch.Tensor,
    kv_indices_prefix: torch.Tensor,
    kv_indptr_prefix: torch.Tensor,
    kv: torch.Tensor,
    kv_indices_extend: torch.Tensor,
    kv_indptr_extend: torch.Tensor,
    attn_sink: torch.Tensor,
    softmax_scale: float,
) -> torch.Tensor:
    if not q.is_cuda:
        raise RuntimeError(
            "Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dtype}"
        )
    if unified_kv.dtype != q.dtype:
        raise RuntimeError(
            f"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}"
        )
    if kv.dtype != q.dtype:
        raise RuntimeError(f"kv dtype mismatch: kv={kv.dtype}, q={q.dtype}")
    if unified_kv.size(-1) != kv.size(-1):
        raise RuntimeError(
            f"head_dim mismatch: unified_kv={unified_kv.size(-1)}, kv={kv.size(-1)}"
        )

    T, H, D = q.shape

View on GitHub (pinned to 0132848349)

Solutions

Move all tensors to the CUDA device: q = q.cuda() (and likewise for unified_kv, kv, indices)
If you need a CPU reference path, branch to the torch reference implementation instead of the Triton kernel
Ensure the environment actually has a GPU (torch.cuda.is_available()) before dispatching

Example fix

// before
out = sparse_attn_v4_paged_prefill(q, kv, ...)  # q on CPU
// after
out = sparse_attn_v4_paged_prefill(q.cuda(), kv.cuda(), ...)

Defensive patterns

Strategy: type-guard

Validate before calling

assert q.is_cuda, "sparse_attn_v4_paged_prefill requires CUDA tensors"

Type guard

def on_gpu(*ts) -> bool:
    return all(t.is_cuda for t in ts if isinstance(t, torch.Tensor))

Prevention

Skip GPU-kernel tests when torch.cuda.is_available() is False
Move every tensor to the same device at the start of the forward call

When it happens

Trigger: Calling sparse_attn_v4_paged_prefill with any of the tensors (checked via q) on CPU, e.g. in a unit test or a CPU-only debug run.

Common situations: Unit tests with CPU fixtures; forgetting .cuda() after building tensors; running on a machine without a GPU; device mismatch where q is CPU but the cache is on GPU.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5d9da259c68eb6c4. Report an issue: GitHub.