sgl-project/sglang · error · ValueError

kv must be a CUDA tensor

Error message

kv must be a CUDA tensor

What it means

kv must be a CUDA tensor for the SM90 sparse prefill kernel; a CPU kv raises ValueError at the contract check (sparse_mla_q8kv8_prefill_sm90.py:319) before any launch.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:319

        raise ValueError(
            f"kv must have shape (s_kv, h_kv, d_qk), got {tuple(kv.shape)}"
        )
    if indices.ndim != 3:
        raise ValueError(
            "indices must have shape (s_q, h_kv, topk), " f"got {tuple(indices.shape)}"
        )

    s_q, h_q, d_qk = q.shape
    s_kv, h_kv, kv_d_qk = kv.shape
    topk = indices.shape[2]
    device = q.device

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():

View on GitHub (pinned to 0132848349)

Solutions

1. Move kv to the same CUDA device as q before the call
2. Ensure the fp8 quantization step for kv runs on GPU or ends with .to(q.device)

Example fix

# before
kv_fp8 = quantize(kv_cpu)  # CPU
# after
kv_fp8 = quantize(kv_cpu).to(q.device)

kv = kv.to(q.device)
assert kv.is_cuda

When it happens

Trigger: Mixing devices: q on GPU but kv still on CPU (or on a meta/other-backend tensor) when calling sparse_mla_q8kv8_prefill_fwd.

Common situations: Quantizing/preparing kv on CPU (quantization tooling) and forgetting the final .cuda(); partial .to(device) migrations of a pipeline.

Related errors

• q must be a CUDA tensor
• indices must be a CUDA tensor
• {name} must be on device {device}, got {t.device}
• kv must be on q's device {device}, got {kv.device}
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bc386f2b8576305e. Report an issue: GitHub.