{"record":{"id":"89626810c370b3f9","repo":"sgl-project/sglang","slug":"npu-packed-attention-requires-q-k-and-v-in-t-n","errorCode":null,"errorMessage":"NPU packed attention requires q, k, and v in [T, N, D] layout; invalid tensors: {', '.join(invalid_layouts)}","messagePattern":"NPU packed attention requires q, k, and v in \\[T, N, D\\] layout; invalid tensors: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":68,"sourceCode":"    return boundaries\n\n\ndef fused_infer_attention_varlen(\n    q: torch.Tensor,\n    k: torch.Tensor,\n    v: torch.Tensor,\n    cu_seqlens_q: torch.Tensor,\n    cu_seqlens_k: torch.Tensor,\n    *,\n    cu_seqlens_q_host: Sequence[int] | None = None,\n    cu_seqlens_k_host: Sequence[int] | None = None,\n    softmax_scale: float | None = None,\n    return_softmax_lse: bool = False,\n) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:\n    tensors = {\"q\": q, \"k\": k, \"v\": v}\n    invalid_layouts = [name for name, tensor in tensors.items() if tensor.ndim != 3]\n    if invalid_layouts:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v in [T, N, D] layout; \"\n            f\"invalid tensors: {', '.join(invalid_layouts)}\"\n        )\n    invalid_devices = [\n        name\n        for name, tensor in tensors.items()\n        if tensor.device.type != \"npu\" or tensor.device != q.device\n    ]\n    if invalid_devices:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v on the same NPU; \"\n            f\"invalid tensors: {', '.join(invalid_devices)}\"\n        )\n    if not (q.dtype == k.dtype == v.dtype):\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v with the same dtype\"\n        )\n    if k.shape[:2] != v.shape[:2]:","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L50-L86","documentation":"fused_infer_attention_varlen (NPU packed attention) requires q, k, v as rank-3 tensors in [T, N, D] (tokens, heads, head_dim) layout. Any of the three with ndim != 3 is rejected, and the message names the offending tensors.","triggerScenarios":"Passing [B, S, N, D] batched tensors, [B, N, S, D] transposed attention-layout tensors, or 2D/4D projections; invalid_layouts lists which of q/k/v are wrong.","commonSituations":"Porting code from a backend that takes [B, S, N, D] (like the base AITerImpl.forward or flash_attn batched API); forgetting to flatten a batched tensor to packed tokens before the varlen call.","solutions":["Reshape to packed layout: q = q.reshape(-1, num_heads, head_dim) (same for k, v) with cu_seqlens covering the packed T dimension","If your tensors are [B, S, N, D], flatten batch and sequence dims: q.reshape(q.shape[0]*q.shape[1], q.shape[2], q.shape[3])","Check the error's invalid list to see which tensor(s) came in wrong and fix those call sites"],"exampleFix":"# before: [B, S, N, D]\nout = fused_infer_attention_varlen(q, k, v, cu_q, cu_k)\n# after: packed [T, N, D]\nq3 = q.reshape(-1, q.shape[-2], q.shape[-1])\nk3 = k.reshape(-1, k.shape[-2], k.shape[-1])\nv3 = v.reshape(-1, v.shape[-2], v.shape[-1])\nout = fused_infer_attention_varlen(q3, k3, v3, cu_q, cu_k)","handlingStrategy":"type-guard","validationCode":"q3 = q.reshape(-1, num_heads, head_dim)\nk3 = k.reshape(-1, num_kv_heads, head_dim)\nv3 = v.reshape(-1, num_kv_heads, head_dim)","typeGuard":"def is_tnd(t: torch.Tensor) -> bool:\n    return t.ndim == 3","tryCatchPattern":null,"preventionTips":["Standardize on packed [T, N, D] layout at your attention wrapper boundary","Flatten [B, S, N, D] -> [B*S, N, D] before varlen calls"],"tags":["npu","ascend","varlen","tensor-layout","shape-validation"],"backgroundTag":"wrong-tensor-layout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}