sgl-project/sglang · error · ValueError

q must be a CUDA tensor

Error message

q must be a CUDA tensor

What it means

The CUDA entry point launches all accesses on q's device, so sparse_mla_q8kv8_prefill_fwd requires q to be a CUDA tensor and rejects CPU (or other-backend) q with ValueError before launch.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:317

        raise ValueError(f"q must have shape (s_q, h_q, d_qk), got {tuple(q.shape)}")
    if kv.ndim != 3:
        raise ValueError(
            f"kv must have shape (s_kv, h_kv, d_qk), got {tuple(kv.shape)}"
        )
    if indices.ndim != 3:
        raise ValueError(
            "indices must have shape (s_q, h_kv, topk), " f"got {tuple(indices.shape)}"
        )

    s_q, h_q, d_qk = q.shape
    s_kv, h_kv, kv_d_qk = kv.shape
    topk = indices.shape[2]
    device = q.device

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():

View on GitHub (pinned to 0132848349)

Solutions

1. Move q (and kv/indices) to the CUDA device before calling: q = q.cuda()
2. Skip/gate this op when torch.cuda.is_available() is False and use a CPU fallback

Example fix

# before
q = torch.randn(s_q, h_q, d)  # CPU
out = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)
# after
q, kv, idx = q.cuda(), kv.cuda(), idx.cuda()
out = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)

q = q.cuda() if not q.is_cuda else q
assert q.is_cuda

def on_cuda(t: torch.Tensor) -> bool:
    return t.is_cuda

When it happens

Trigger: Calling the prefill op with q on CPU — e.g. constructing test tensors without device='cuda', or running the op in a CPU-only environment / before moving model outputs to GPU.

Common situations: Unit tests that build inputs on CPU (see test_q8kv8_sparse_prefill_rejects_* tests); weight-loading code that calls attention ops on CPU tensors; missing .cuda() in a prototype.

Related errors

• kv must be a CUDA tensor
• indices must be a CUDA tensor
• {name}_block tensors must live on CUDA
• {name} must live on CUDA
• {name} must be on device {device}, got {t.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/dfcaa269177af4bf. Report an issue: GitHub.