{"record":{"id":"f2fbfb11228a860c","repo":"sgl-project/sglang","slug":"npu-packed-attention-requires-q-k-and-v-on-the-s","errorCode":null,"errorMessage":"NPU packed attention requires q, k, and v on the same NPU; invalid tensors: {', '.join(invalid_devices)}","messagePattern":"NPU packed attention requires q, k, and v on the same NPU; invalid tensors: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":78,"sourceCode":"    cu_seqlens_q_host: Sequence[int] | None = None,\n    cu_seqlens_k_host: Sequence[int] | None = None,\n    softmax_scale: float | None = None,\n    return_softmax_lse: bool = False,\n) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:\n    tensors = {\"q\": q, \"k\": k, \"v\": v}\n    invalid_layouts = [name for name, tensor in tensors.items() if tensor.ndim != 3]\n    if invalid_layouts:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v in [T, N, D] layout; \"\n            f\"invalid tensors: {', '.join(invalid_layouts)}\"\n        )\n    invalid_devices = [\n        name\n        for name, tensor in tensors.items()\n        if tensor.device.type != \"npu\" or tensor.device != q.device\n    ]\n    if invalid_devices:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v on the same NPU; \"\n            f\"invalid tensors: {', '.join(invalid_devices)}\"\n        )\n    if not (q.dtype == k.dtype == v.dtype):\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v with the same dtype\"\n        )\n    if k.shape[:2] != v.shape[:2]:\n        raise ValueError(\n            \"NPU packed attention requires matching K/V token and head counts\"\n        )\n    if q.shape[-1] != k.shape[-1]:\n        raise ValueError(\"NPU packed attention requires matching Q/K head dimensions\")\n\n    q_boundaries = _packed_boundaries(\n        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], \"cu_seqlens_q\"\n    )\n    k_boundaries = _packed_boundaries(","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L60-L96","documentation":"All of q, k, v must be on device type 'npu' and on the same NPU device as q. The kernel torch.ops.npu.npu_fused_infer_attention_score is device-specific; CPU tensors or tensors scattered across different Ascend devices are rejected, with the offending names listed.","triggerScenarios":"Passing CPU tensors (e.g. never moved after creation), mixing a KV cache on 'npu:1' with q on 'npu:0', or tensors left on 'cuda' after copying code from a GPU path.","commonSituations":"Initial bring-up of Ascend inference where tensors were created without device=; tensor-parallel code copying between devices; unit tests running on CPU by default.","solutions":["Move all three to the same NPU: q, k, v = q.to('npu'), k.to('npu'), v.to('npu')","In tensor-parallel setups, gather/replicate KV to the same device as q before the call","Add an assert q.device == k.device == v.device and q.device.type == 'npu' before invoking"],"exampleFix":"# before\nq = torch.randn(T, N, D)  # CPU\n# after\nq = torch.randn(T, N, D, device=\"npu\")\nk = k.to(q.device)\nv = v.to(q.device)","handlingStrategy":"validation","validationCode":"assert q.device.type == \"npu\" and q.device == k.device == v.device","typeGuard":"def on_same_npu(*ts: torch.Tensor) -> bool:\n    return all(t.device.type == \"npu\" and t.device == ts[0].device for t in ts)","tryCatchPattern":null,"preventionTips":["Allocate tensors directly on the target NPU device","In TP code, explicitly move KV shards to q's device"],"tags":["npu","ascend","device-placement","validation"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}