{"record":{"id":"5a32e337b8b4429a","repo":"sgl-project/sglang","slug":"npu-packed-attention-requires-q-k-and-v-with-the","errorCode":null,"errorMessage":"NPU packed attention requires q, k, and v with the same dtype","messagePattern":"NPU packed attention requires q, k, and v with the same dtype","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":83,"sourceCode":"    tensors = {\"q\": q, \"k\": k, \"v\": v}\n    invalid_layouts = [name for name, tensor in tensors.items() if tensor.ndim != 3]\n    if invalid_layouts:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v in [T, N, D] layout; \"\n            f\"invalid tensors: {', '.join(invalid_layouts)}\"\n        )\n    invalid_devices = [\n        name\n        for name, tensor in tensors.items()\n        if tensor.device.type != \"npu\" or tensor.device != q.device\n    ]\n    if invalid_devices:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v on the same NPU; \"\n            f\"invalid tensors: {', '.join(invalid_devices)}\"\n        )\n    if not (q.dtype == k.dtype == v.dtype):\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v with the same dtype\"\n        )\n    if k.shape[:2] != v.shape[:2]:\n        raise ValueError(\n            \"NPU packed attention requires matching K/V token and head counts\"\n        )\n    if q.shape[-1] != k.shape[-1]:\n        raise ValueError(\"NPU packed attention requires matching Q/K head dimensions\")\n\n    q_boundaries = _packed_boundaries(\n        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], \"cu_seqlens_q\"\n    )\n    k_boundaries = _packed_boundaries(\n        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], \"cu_seqlens_k\"\n    )\n    if len(q_boundaries) != len(k_boundaries):\n        raise ValueError(\"cu_seqlens_q and cu_seqlens_k must describe the same batch\")\n","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L65-L101","documentation":"NPU packed attention requires q, k, and v to share the same dtype; the fused kernel does not support mixed-precision inputs. A common case is fp16 q with bf16 KV cache (or vice versa), or an unquantized tensor alongside a quantized one.","triggerScenarios":"Calling fused_infer_attention_varlen where q.dtype != k.dtype or k.dtype != v.dtype — e.g. model weights in bf16 but projections computed in fp16, or a KV cache stored in a different dtype than the query path.","commonSituations":"Autocast/AMP producing mixed dtypes across branches; loading a KV cache in fp16 while the model runs bf16; partial quantization (fp8 KV) not handled by this kernel.","solutions":["Cast to a single dtype before the call: k = k.to(q.dtype); v = v.to(q.dtype) (prefer the compute dtype, usually bf16 on Ascend)","Align KV cache dtype with the model dtype at allocation time","If you intended quantized KV attention, use the dedicated quantized path, not this kernel"],"exampleFix":"# before\nout = fused_infer_attention_varlen(q_bf16, k_fp16, v_fp16, cu_q, cu_k)\n# after\nk = k.to(q.dtype)\nv = v.to(q.dtype)\nout = fused_infer_attention_varlen(q, k, v, cu_q, cu_k)","handlingStrategy":"validation","validationCode":"if k.dtype != q.dtype:\n    k = k.to(q.dtype)\nif v.dtype != q.dtype:\n    v = v.to(q.dtype)","typeGuard":"def same_dtype(*ts: torch.Tensor) -> bool:\n    return all(t.dtype == ts[0].dtype for t in ts)","tryCatchPattern":null,"preventionTips":["Allocate the KV cache with the model compute dtype","Avoid autocast around attention calls on NPU"],"tags":["npu","ascend","dtype","mixed-precision","validation"],"backgroundTag":"invalid-tensor-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}