sgl-project/sglang · error · ValueError

indices must be on q's device {device}, got {indices.device}

Error message

indices must be on q's device {device}, got {indices.device}

What it means

Raised by sparse_mla_q8kv8_prefill_fwd when the indices tensor (top-k KV index tensor) lives on a different CUDA device than the query tensor q. The kernel requires all inputs on a single device because it launches a CUDA kernel directly using q's device stream, so a mismatched device would cause an illegal memory access.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:326

    s_q, h_q, d_qk = q.shape
    s_kv, h_kv, kv_d_qk = kv.shape
    topk = indices.shape[2]
    device = q.device

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():
        raise ValueError("indices must be contiguous")

    if kv_d_qk != d_qk:
        raise ValueError(f"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}")

View on GitHub (pinned to 0132848349)

Solutions

1. Move indices to q's device: indices = indices.to(q.device) before the call
2. Verify all of q, kv, indices, topk_length, attn_sink are allocated on the same device in multi-GPU code
3. Check that your rank-to-device mapping (e.g. torch.cuda.set_device(local_rank)) is applied before building these tensors

Example fix

// before
out = sparse_mla_q8kv8_prefill_fwd(q, kv, indices)  # indices on cuda:1
// after
indices = indices.to(q.device, non_blocking=True)
out = sparse_mla_q8kv8_prefill_fwd(q, kv, indices)

assert indices.device == q.device, f"indices on {indices.device}, q on {q.device}"

def indices_on_q_device(q: torch.Tensor, indices: torch.Tensor) -> bool:
    return indices.is_cuda and indices.device == q.device

catch ValueError and re-raise with rank context: except ValueError as e: raise RuntimeError(f"rank {rank}: {e}") from e

When it happens

Trigger: Calling sparse_mla_q8kv8_prefill_fwd(q, kv, indices, ...) where q is on cuda:0 but indices was produced or moved to cuda:1 (or CPU-then-moved incorrectly) in a multi-GPU TP setup.

Common situations: Tensor-parallel or pipeline-parallel pipelines where per-rank tensors are created on a fixed device while indices come from a cache on another device; device pinning via CUDA_VISIBLE_DEVICES mismatch; passing tensors from a different process/device context.

Related errors

• candidates and next_token_logits must be on the same device,
• {name}_block_cnt and {name}_block_idx must be on the same de
• q must be torch.float8_e4m3fn, got {q.dtype}
• q must be contiguous
• topk_length must be a CUDA tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ed106d206424e315. Report an issue: GitHub.