{"record":{"id":"8caa99387313e65b","repo":"sgl-project/sglang","slug":"topk-length-must-be-on-q-s-device-device-got-t","errorCode":null,"errorMessage":"topk_length must be on q's device {device}, got {topk_length.device}","messagePattern":"topk_length must be on q's device (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":387,"sourceCode":"    if indices.dtype != torch.int32:\n        raise ValueError(f\"indices must be int32, got {indices.dtype}\")\n\n    if topk == 0 or topk % 128 != 0:\n        raise ValueError(\n            \"Q8KV8 sparse-prefill topk width must be a positive multiple of 128, \"\n            f\"got {topk}\"\n        )\n\n    if topk_length is not None:\n        if topk_length.shape != (s_q,) or topk_length.dtype != torch.int32:\n            raise ValueError(\n                f\"topk_length must be int32 with shape ({s_q},), got \"\n                f\"{tuple(topk_length.shape)}/{topk_length.dtype}\"\n            )\n        if not topk_length.is_cuda:\n            raise ValueError(\"topk_length must be a CUDA tensor\")\n        if topk_length.device != device:\n            raise ValueError(\n                \"topk_length must be on q's device \"\n                f\"{device}, got {topk_length.device}\"\n            )\n        if not topk_length.is_contiguous():\n            raise ValueError(\"topk_length must be contiguous\")\n        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():\n            raise ValueError(\n                \"topk_length values must satisfy \" f\"0 <= topk_length <= topk ({topk})\"\n            )\n\n    if d_v != 512:\n        raise ValueError(\n            f\"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}\"\n        )\n\n    if attn_sink is not None and topk_length is None:\n        raise ValueError(\"attn_sink requires topk_length to be provided as well\")\n","sourceCodeStart":369,"sourceCodeEnd":405,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L369-L405","documentation":"Even when topk_length is a CUDA tensor, it must be on the same device as q, since the kernel launches on q's device and stream. A length tensor on another GPU triggers this error.","triggerScenarios":"q on cuda:0 with topk_length on cuda:1 in a TP worker; lengths moved to the wrong rank's device during broadcast.","commonSituations":"Multi-GPU pipelines where metadata tensors are gathered/broadcast across ranks; hard-coded .cuda() (device 0) in multi-GPU processes.","solutions":["topk_length = topk_length.to(q.device)","Use explicit device indices (torch.device('cuda', local_rank)) instead of bare .cuda()","Verify rank-local device assignment before constructing any metadata tensors"],"exampleFix":"// before\ntopk_length = topk_length.cuda()  # lands on cuda:0 in every rank\n// after\ntopk_length = topk_length.to(q.device)","handlingStrategy":"validation","validationCode":"assert topk_length.device == q.device","typeGuard":"def lengths_on_device(q: torch.Tensor, tl: torch.Tensor) -> bool:\n    return tl.is_cuda and tl.device == q.device","tryCatchPattern":null,"preventionTips":["Use .to(q.device) instead of bare .cuda() in multi-GPU code","Broadcast metadata per rank after device assignment"],"tags":["cuda","device-mismatch","topk"],"backgroundTag":"cuda-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}