{"record":{"id":"7cd58a20b0eb39d3","repo":"sgl-project/sglang","slug":"qkv-tensors-must-be-on-the-same-cuda-device","errorCode":null,"errorMessage":"QKV tensors must be on the same CUDA device","messagePattern":"QKV tensors must be on the same CUDA device","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/diffusion/rope/hunyuan_qkv_pack_triton.py","lineNumber":167,"sourceCode":"\n\ndef hunyuan_qkv_rope_pack(\n    img_q: torch.Tensor,\n    img_k: torch.Tensor,\n    img_v: torch.Tensor,\n    txt_q: torch.Tensor,\n    txt_k: torch.Tensor,\n    txt_v: torch.Tensor,\n    cos: torch.Tensor,\n    sin: torch.Tensor,\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:\n    tensors = (img_q, img_k, img_v, txt_q, txt_k, txt_v)\n    if any(x.ndim != 4 for x in tensors):\n        raise ValueError(\"QKV tensors must have shape [B, S, H, D]\")\n    if any(not x.is_cuda or x.dtype != torch.bfloat16 for x in tensors):\n        raise ValueError(\"QKV tensors must be CUDA bfloat16 tensors\")\n    if any(x.device != img_q.device for x in tensors):\n        raise ValueError(\"QKV tensors must be on the same CUDA device\")\n    batch, img_tokens, num_heads, head_dim = img_q.shape\n    txt_tokens = txt_q.shape[1]\n    expected_img = (batch, img_tokens, num_heads, head_dim)\n    expected_txt = (batch, txt_tokens, num_heads, head_dim)\n    if any(tuple(x.shape) != expected_img for x in (img_q, img_k, img_v)):\n        raise ValueError(\"image QKV shapes must match\")\n    if any(tuple(x.shape) != expected_txt for x in (txt_q, txt_k, txt_v)):\n        raise ValueError(\"text QKV shapes must match\")\n    if any(x.stride(-1) != 1 for x in tensors):\n        raise ValueError(\"QKV last dimensions must be contiguous\")\n    if head_dim <= 0 or head_dim > 128 or head_dim % 2:\n        raise ValueError(\"head_dim must be positive, even, and <= 128\")\n    if cos.ndim != 2 or sin.ndim != 2 or cos.shape != sin.shape:\n        raise ValueError(\"cos and sin must have matching [S, D/2] shapes\")\n    if cos.shape[0] < img_tokens or cos.shape[1] != head_dim // 2:\n        raise ValueError(\"cos/sin shape does not cover image tokens and head_dim\")\n    if not cos.is_cuda or not sin.is_cuda or cos.stride(-1) != 1 or sin.stride(-1) != 1:\n        raise ValueError(\"cos and sin must be CUDA and last-dim contiguous\")","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/diffusion/rope/hunyuan_qkv_pack_triton.py#L149-L185","documentation":"All six QKV tensors must live on the same CUDA device as img_q; multi-device or CPU/CUDA mixes are rejected because the kernel launches on img_q's stream/device.","triggerScenarios":"Passing txt_q/k/v (or img tensors) allocated on a different GPU or on CPU than img_q in hunyuan_qkv_rope_pack.","commonSituations":"Tensor-parallel setups where some projections got sharded to another rank's device, or a partially-moved model where the txt stream was never transferred.","solutions":["Move all tensors to one device: t = t.to(img_q.device)","Verify the whole model (weights and activations) is on a single device or properly sharded per rank"],"exampleFix":"# before\nout = hunyuan_qkv_rope_pack(img_q, img_k, img_v, txt_q, txt_k, txt_v, cos, sin)  # txt on cuda:1\n# after\ndev = img_q.device\nout = hunyuan_qkv_rope_pack(img_q, img_k, img_v, txt_q.to(dev), txt_k.to(dev), txt_v.to(dev), cos.to(dev), sin.to(dev))","handlingStrategy":"validation","validationCode":"dev = img_q.device\ntensors = [t.to(dev) for t in (img_k, img_v, txt_q, txt_k, txt_v, cos, sin)]","typeGuard":"def same_device(t: torch.Tensor, ref: torch.Tensor) -> bool:\n    return t.device == ref.device","tryCatchPattern":null,"preventionTips":["Pin one device per rank and route all streams through it","Assert device equality before multi-tensor kernel calls in TP setups"],"tags":["device","multi-gpu","rope","hunyuan"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}