{"record":{"id":"f8d9aefdae04a856","repo":"sgl-project/sglang","slug":"name-must-be-on-q-s-device-device-got-scale","errorCode":null,"errorMessage":"{name} must be on q's device {device}, got {scale.device}","messagePattern":"(.+?) must be on q's device (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":427,"sourceCode":"                f\"attn_sink must be float32 with shape ({h_q},), got \"\n                f\"{tuple(attn_sink.shape)}/{attn_sink.dtype}\"\n            )\n        if not attn_sink.is_cuda:\n            raise ValueError(\"attn_sink must be a CUDA tensor\")\n        if attn_sink.device != device:\n            raise ValueError(\n                f\"attn_sink must be on q's device {device}, got {attn_sink.device}\"\n            )\n        if not attn_sink.is_contiguous():\n            raise ValueError(\"attn_sink must be contiguous\")\n\n    for name, scale in ((\"q_scale\", q_scale), (\"kv_scale\", kv_scale)):\n        if not isinstance(scale, torch.Tensor):\n            raise ValueError(f\"{name} must be a torch.Tensor\")\n        if not scale.is_cuda:\n            raise ValueError(f\"{name} must be a CUDA tensor\")\n        if scale.device != device:\n            raise ValueError(\n                f\"{name} must be on q's device {device}, got {scale.device}\"\n            )\n        if scale.dtype != torch.float32:\n            raise ValueError(f\"{name} must be float32, got {scale.dtype}\")\n        if scale.numel() != 1:\n            raise ValueError(\n                f\"{name} must be a scalar tensor, got shape {tuple(scale.shape)}\"\n            )\n        if not scale.is_contiguous():\n            raise ValueError(f\"{name} must be contiguous\")\n\n    if out is None:\n        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)\n    else:\n        _check_out_buffer(out, \"out\", (s_q, h_q, d_v), torch.bfloat16, device)\n\n    if max_logits is None:\n        max_logits = torch.empty(s_q, h_q, dtype=torch.float32, device=device)","sourceCodeStart":409,"sourceCodeEnd":445,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L409-L445","documentation":"q_scale and kv_scale must reside on the same CUDA device as q. In multi-GPU runs a scale tensor on another GPU is rejected to prevent cross-device pointer dereference inside the kernel.","triggerScenarios":"Tensor-parallel inference where scales are allocated once on cuda:0 but the worker's q tensor is on cuda:N (N != 0).","commonSituations":"Sharing a single global scale buffer across TP ranks, or initializing scales before torch.cuda.set_device is called for the worker.","solutions":["Allocate scales with device=q.device per rank","Move shared buffers: kv_scale = kv_scale.to(q.device) at forward time"],"exampleFix":"// before\nq_scale = GLOBAL_SCALES['q']  # on cuda:0\n// after\nq_scale = GLOBAL_SCALES['q'].to(q.device)","handlingStrategy":"validation","validationCode":"q_scale = q_scale.to(q.device); kv_scale = kv_scale.to(q.device)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check scale.device == q.device in debug asserts during TP bring-up","Avoid global shared buffers across ranks; allocate per-device"],"tags":["mla","quantization","multi-gpu","device-mismatch"],"backgroundTag":"tensor-on-wrong-device","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}