sgl-project/sglang · error · ValueError

kv must be on q's device {device}, got {kv.device}

Error message

kv must be on q's device {device}, got {kv.device}

What it means

Beyond being CUDA tensors, kv and indices must be on the same device as q, because entry.cuh launches on q's device. A kv on a different GPU raises ValueError with both devices named.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:324

            "indices must have shape (s_q, h_kv, topk), " f"got {tuple(indices.shape)}"
        )

    s_q, h_q, d_qk = q.shape
    s_kv, h_kv, kv_d_qk = kv.shape
    topk = indices.shape[2]
    device = q.device

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():
        raise ValueError("indices must be contiguous")

    if kv_d_qk != d_qk:

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate/derive every input from q.device: kv = kv.to(q.device)
2. Call torch.cuda.set_device(rank) before any tensor allocation in multi-GPU workers

Example fix

# before
kv = torch.empty(..., device='cuda')  # defaults to cuda:0
# after
kv = torch.empty(..., device=q.device)  # matches q's GPU
out = sparse_mla_q8kv8_prefill_fwd(q, kv, idx, ...)

assert kv.device == q.device and idx.device == q.device, \
    (q.device, kv.device, idx.device)

def all_on_same_device(*ts: torch.Tensor) -> bool:
    return len({t.device for t in ts}) == 1 and ts[0].is_cuda

When it happens

Trigger: q on cuda:1 (e.g. TP rank 1) while kv was allocated on cuda:0 or with device='cuda' defaulting to the current device in a multi-GPU process.

Common situations: Tensor/data-parallel inference where buffers are allocated before torch.cuda.set_device(rank); code that hardcodes 'cuda:0'; NCCL pipelines copying inputs to the wrong rank's GPU.

Related errors

• {name} must be on device {device}, got {t.device}
• q must be a CUDA tensor
• kv must be a CUDA tensor
• indices must be a CUDA tensor
• Role {role_type.value} rank {rank_idx} failed to initialize.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3822ffccee4659e0. Report an issue: GitHub.