{"record":{"id":"5cc08fee06d069c9","repo":"sgl-project/sglang","slug":"attn-sink-must-be-on-q-s-device-device-got-att","errorCode":null,"errorMessage":"attn_sink must be on q's device {device}, got {attn_sink.device}","messagePattern":"attn_sink must be on q's device (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":415,"sourceCode":"\n    if d_v != 512:\n        raise ValueError(\n            f\"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}\"\n        )\n\n    if attn_sink is not None and topk_length is None:\n        raise ValueError(\"attn_sink requires topk_length to be provided as well\")\n\n    if attn_sink is not None:\n        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:\n            raise ValueError(\n                f\"attn_sink must be float32 with shape ({h_q},), got \"\n                f\"{tuple(attn_sink.shape)}/{attn_sink.dtype}\"\n            )\n        if not attn_sink.is_cuda:\n            raise ValueError(\"attn_sink must be a CUDA tensor\")\n        if attn_sink.device != device:\n            raise ValueError(\n                f\"attn_sink must be on q's device {device}, got {attn_sink.device}\"\n            )\n        if not attn_sink.is_contiguous():\n            raise ValueError(\"attn_sink must be contiguous\")\n\n    for name, scale in ((\"q_scale\", q_scale), (\"kv_scale\", kv_scale)):\n        if not isinstance(scale, torch.Tensor):\n            raise ValueError(f\"{name} must be a torch.Tensor\")\n        if not scale.is_cuda:\n            raise ValueError(f\"{name} must be a CUDA tensor\")\n        if scale.device != device:\n            raise ValueError(\n                f\"{name} must be on q's device {device}, got {scale.device}\"\n            )\n        if scale.dtype != torch.float32:\n            raise ValueError(f\"{name} must be float32, got {scale.dtype}\")\n        if scale.numel() != 1:\n            raise ValueError(","sourceCodeStart":397,"sourceCodeEnd":433,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L397-L433","documentation":"sparse_mla_q8kv8_prefill_fwd requires attn_sink to sit on the same CUDA device as the q tensor (multi-GPU safety). Even a CUDA tensor on a different GPU is rejected, since the kernel would otherwise read cross-device memory and either crash or silently corrupt results.","triggerScenarios":"Running with CUDA_VISIBLE_DEVICES spanning multiple GPUs and passing an attn_sink allocated on cuda:0 while q lives on cuda:1 (device mismatch).","commonSituations":"Tensor-parallel or multi-GPU inference where a sink buffer is allocated once on the default device but reused by workers pinned to other devices.","solutions":["Allocate or move the sink with device=q.device explicitly","In TP workers, create per-rank sink buffers instead of sharing one global buffer"],"exampleFix":"// before\nattn_sink = torch.zeros(h_q, dtype=torch.float32, device='cuda:0')\n// after\nattn_sink = torch.zeros(h_q, dtype=torch.float32, device=q.device)","handlingStrategy":"validation","validationCode":"assert attn_sink.device == q.device, f'sink on {attn_sink.device}, q on {q.device}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["In TP workers create per-rank buffers with device=q.device rather than sharing one","Set the CUDA device (torch.cuda.set_device) before allocating any per-rank tensors"],"tags":["mla","sparse-attention","multi-gpu","device-mismatch"],"backgroundTag":"tensor-on-wrong-device","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}