sgl-project/sglang · error · ValueError

QKV tensors must be on the same CUDA device

Error message

QKV tensors must be on the same CUDA device

What it means

Device consistency check for Hunyuan QKV-pack RoPE: q, k, and v must live on the same CUDA device, since the kernel packs them in one launch. Typically violated under multi-GPU sharding when one projection lands on another rank.

Solutions

1. Move all tensors to one device: t = t.to(img_q.device)
2. Verify the whole model (weights and activations) is on a single device or properly sharded per rank

Example fix

# before
out = hunyuan_qkv_rope_pack(img_q, img_k, img_v, txt_q, txt_k, txt_v, cos, sin)  # txt on cuda:1
# after
dev = img_q.device
out = hunyuan_qkv_rope_pack(img_q, img_k, img_v, txt_q.to(dev), txt_k.to(dev), txt_v.to(dev), cos.to(dev), sin.to(dev))

dev = img_q.device
tensors = [t.to(dev) for t in (img_k, img_v, txt_q, txt_k, txt_v, cos, sin)]

def same_device(t: torch.Tensor, ref: torch.Tensor) -> bool:
    return t.device == ref.device

When it happens

Trigger: Passing txt_q/k/v (or img tensors) allocated on a different GPU or on CPU than img_q in hunyuan_qkv_rope_pack.

Common situations: Tensor-parallel setups where some projections got sharded to another rank's device, or a partially-moved model where the txt stream was never transferred.

Related errors

• QKV tensors must be CUDA bfloat16 tensors
• head_dim must be positive, even, and <= 128
• image QKV shapes must match
• QKV and cos/sin tensors must be on the same CUDA device
• QKV last dimensions must be contiguous

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7cd58a20b0eb39d3. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/diffusion/rope/hunyuan_qkv_pack_triton.py:167

def hunyuan_qkv_rope_pack(
    img_q: torch.Tensor,
    img_k: torch.Tensor,
    img_v: torch.Tensor,
    txt_q: torch.Tensor,
    txt_k: torch.Tensor,
    txt_v: torch.Tensor,
    cos: torch.Tensor,
    sin: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    tensors = (img_q, img_k, img_v, txt_q, txt_k, txt_v)
    if any(x.ndim != 4 for x in tensors):
        raise ValueError("QKV tensors must have shape [B, S, H, D]")
    if any(not x.is_cuda or x.dtype != torch.bfloat16 for x in tensors):
        raise ValueError("QKV tensors must be CUDA bfloat16 tensors")
    if any(x.device != img_q.device for x in tensors):
        raise ValueError("QKV tensors must be on the same CUDA device")
    batch, img_tokens, num_heads, head_dim = img_q.shape
    txt_tokens = txt_q.shape[1]
    expected_img = (batch, img_tokens, num_heads, head_dim)
    expected_txt = (batch, txt_tokens, num_heads, head_dim)
    if any(tuple(x.shape) != expected_img for x in (img_q, img_k, img_v)):
        raise ValueError("image QKV shapes must match")
    if any(tuple(x.shape) != expected_txt for x in (txt_q, txt_k, txt_v)):
        raise ValueError("text QKV shapes must match")
    if any(x.stride(-1) != 1 for x in tensors):
        raise ValueError("QKV last dimensions must be contiguous")
    if head_dim <= 0 or head_dim > 128 or head_dim % 2:
        raise ValueError("head_dim must be positive, even, and <= 128")
    if cos.ndim != 2 or sin.ndim != 2 or cos.shape != sin.shape:
        raise ValueError("cos and sin must have matching [S, D/2] shapes")
    if cos.shape[0] < img_tokens or cos.shape[1] != head_dim // 2:
        raise ValueError("cos/sin shape does not cover image tokens and head_dim")
    if not cos.is_cuda or not sin.is_cuda or cos.stride(-1) != 1 or sin.stride(-1) != 1:
        raise ValueError("cos and sin must be CUDA and last-dim contiguous")

View on GitHub (pinned to 0132848349)