{"record":{"id":"f7daa8b7a9fcd2c8","repo":"sgl-project/sglang","slug":"npu-packed-attention-requires-matching-k-v-token-a","errorCode":null,"errorMessage":"NPU packed attention requires matching K/V token and head counts","messagePattern":"NPU packed attention requires matching K/V token and head counts","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":87,"sourceCode":"            \"NPU packed attention requires q, k, and v in [T, N, D] layout; \"\n            f\"invalid tensors: {', '.join(invalid_layouts)}\"\n        )\n    invalid_devices = [\n        name\n        for name, tensor in tensors.items()\n        if tensor.device.type != \"npu\" or tensor.device != q.device\n    ]\n    if invalid_devices:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v on the same NPU; \"\n            f\"invalid tensors: {', '.join(invalid_devices)}\"\n        )\n    if not (q.dtype == k.dtype == v.dtype):\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v with the same dtype\"\n        )\n    if k.shape[:2] != v.shape[:2]:\n        raise ValueError(\n            \"NPU packed attention requires matching K/V token and head counts\"\n        )\n    if q.shape[-1] != k.shape[-1]:\n        raise ValueError(\"NPU packed attention requires matching Q/K head dimensions\")\n\n    q_boundaries = _packed_boundaries(\n        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], \"cu_seqlens_q\"\n    )\n    k_boundaries = _packed_boundaries(\n        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], \"cu_seqlens_k\"\n    )\n    if len(q_boundaries) != len(k_boundaries):\n        raise ValueError(\"cu_seqlens_q and cu_seqlens_k must describe the same batch\")\n\n    q_nonempty = [\n        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])\n    ]\n    k_nonempty = [","sourceCodeStart":69,"sourceCodeEnd":105,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L69-L105","documentation":"k and v must agree on their first two dims in [T, N, D] layout: same packed token count and same number of KV heads. The kernel requires K and V to describe identical token/head structure; a mismatch means the KV pair is internally inconsistent.","triggerScenarios":"Passing k from a truncated KV cache but the full v (or different head counts, e.g. MQA k with N=1 but v with N=num_heads); slicing k along tokens without slicing v identically.","commonSituations":"KV cache paging bugs where K and V pools are indexed differently; ring-attention chunking that splits K but not V; weight-loading errors producing different head counts for k_proj vs v_proj.","solutions":["Ensure k.shape[:2] == v.shape[:2]; slice both identically when truncating/chunking","Verify num_key_value_heads is consistent between k and v at cache allocation","Add assert k.shape[:2] == v.shape[:2] before the call in your wrapper"],"exampleFix":"# before\nk_chunk = k[:t]  # only K truncated\n# after\nk_chunk = k[:t]\nv_chunk = v[:t]\nassert k_chunk.shape[:2] == v_chunk.shape[:2]","handlingStrategy":"validation","validationCode":"assert k.shape[:2] == v.shape[:2], \"K/V token and head counts must match\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always slice/truncate K and V together","Keep K and V in one stacked cache tensor so they cannot diverge"],"tags":["npu","ascend","varlen","shape-validation","kv-cache"],"backgroundTag":"head-dimension-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}