sgl-project/sglang · error · ValueError

indices must be a CUDA tensor

Error message

indices must be a CUDA tensor

What it means

indices must also be a CUDA tensor; the kernel reads selected KV offsets directly from GPU memory, so CPU indices are rejected with ValueError before launch.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:321

        )
    if indices.ndim != 3:
        raise ValueError(
            "indices must have shape (s_q, h_kv, topk), " f"got {tuple(indices.shape)}"
        )

    s_q, h_q, d_qk = q.shape
    s_kv, h_kv, kv_d_qk = kv.shape
    topk = indices.shape[2]
    device = q.device

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():

View on GitHub (pinned to 0132848349)

Solutions

1. Move indices to GPU: indices = indices.to(q.device, non_blocking=True)
2. Compute top-k selection with torch.topk on GPU so indices are already resident

Example fix

# before
idx = np.argsort(scores)[:, -topk:]  # numpy -> CPU tensor
idx = torch.from_numpy(idx)
# after
idx = torch.topk(scores_gpu, topk, dim=-1).indices  # already CUDA
out = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)

idx = idx.to(q.device, non_blocking=True)
assert idx.is_cuda and idx.device == q.device

When it happens

Trigger: q/kv on GPU but the top-k indices tensor still on CPU — e.g. indices computed by a numpy/scipy selection step and not moved to GPU.

Common situations: CPU-side top-k heuristics (numpy argsort) feeding the sparse prefill; exporting/importing indices from disk and loading without device pinning.

Related errors

• q must be a CUDA tensor
• kv must be a CUDA tensor
• {name} must be on device {device}, got {t.device}
• indices must have shape (s_q, h_kv, topk), got {tuple(indice
• kv must be on q's device {device}, got {kv.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b6f2d9481a5070d6. Report an issue: GitHub.