sgl-project/sglang · error · ValueError

QKV tensors must be on the same CUDA device

Error message

QKV tensors must be on the same CUDA device

What it means

All six QKV tensors must live on the same CUDA device as img_q; multi-device or CPU/CUDA mixes are rejected because the kernel launches on img_q's stream/device.

Source

Thrown at python/sglang/kernels/ops/diffusion/rope/hunyuan_qkv_pack_triton.py:167

def hunyuan_qkv_rope_pack(
    img_q: torch.Tensor,
    img_k: torch.Tensor,
    img_v: torch.Tensor,
    txt_q: torch.Tensor,
    txt_k: torch.Tensor,
    txt_v: torch.Tensor,
    cos: torch.Tensor,
    sin: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    tensors = (img_q, img_k, img_v, txt_q, txt_k, txt_v)
    if any(x.ndim != 4 for x in tensors):
        raise ValueError("QKV tensors must have shape [B, S, H, D]")
    if any(not x.is_cuda or x.dtype != torch.bfloat16 for x in tensors):
        raise ValueError("QKV tensors must be CUDA bfloat16 tensors")
    if any(x.device != img_q.device for x in tensors):
        raise ValueError("QKV tensors must be on the same CUDA device")
    batch, img_tokens, num_heads, head_dim = img_q.shape
    txt_tokens = txt_q.shape[1]
    expected_img = (batch, img_tokens, num_heads, head_dim)
    expected_txt = (batch, txt_tokens, num_heads, head_dim)
    if any(tuple(x.shape) != expected_img for x in (img_q, img_k, img_v)):
        raise ValueError("image QKV shapes must match")
    if any(tuple(x.shape) != expected_txt for x in (txt_q, txt_k, txt_v)):
        raise ValueError("text QKV shapes must match")
    if any(x.stride(-1) != 1 for x in tensors):
        raise ValueError("QKV last dimensions must be contiguous")
    if head_dim <= 0 or head_dim > 128 or head_dim % 2:
        raise ValueError("head_dim must be positive, even, and <= 128")
    if cos.ndim != 2 or sin.ndim != 2 or cos.shape != sin.shape:
        raise ValueError("cos and sin must have matching [S, D/2] shapes")
    if cos.shape[0] < img_tokens or cos.shape[1] != head_dim // 2:
        raise ValueError("cos/sin shape does not cover image tokens and head_dim")
    if not cos.is_cuda or not sin.is_cuda or cos.stride(-1) != 1 or sin.stride(-1) != 1:
        raise ValueError("cos and sin must be CUDA and last-dim contiguous")

View on GitHub (pinned to 0132848349)

Solutions

1. Move all tensors to one device: t = t.to(img_q.device)
2. Verify the whole model (weights and activations) is on a single device or properly sharded per rank

Example fix

# before
out = hunyuan_qkv_rope_pack(img_q, img_k, img_v, txt_q, txt_k, txt_v, cos, sin)  # txt on cuda:1
# after
dev = img_q.device
out = hunyuan_qkv_rope_pack(img_q, img_k, img_v, txt_q.to(dev), txt_k.to(dev), txt_v.to(dev), cos.to(dev), sin.to(dev))

dev = img_q.device
tensors = [t.to(dev) for t in (img_k, img_v, txt_q, txt_k, txt_v, cos, sin)]

def same_device(t: torch.Tensor, ref: torch.Tensor) -> bool:
    return t.device == ref.device

When it happens

Trigger: Passing txt_q/k/v (or img tensors) allocated on a different GPU or on CPU than img_q in hunyuan_qkv_rope_pack.

Common situations: Tensor-parallel setups where some projections got sharded to another rank's device, or a partially-moved model where the txt stream was never transferred.

Related errors

• QKV tensors must be CUDA bfloat16 tensors
• QKV tensors must have shape [B, S, H, D]
• image QKV shapes must match
• text QKV shapes must match
• QKV last dimensions must be contiguous

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7cd58a20b0eb39d3. Report an issue: GitHub.