{"record":{"id":"002dcdec21fea142","repo":"sgl-project/sglang","slug":"npu-packed-attention-requires-matching-q-k-head-di","errorCode":null,"errorMessage":"NPU packed attention requires matching Q/K head dimensions","messagePattern":"NPU packed attention requires matching Q/K head dimensions","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":91,"sourceCode":"        name\n        for name, tensor in tensors.items()\n        if tensor.device.type != \"npu\" or tensor.device != q.device\n    ]\n    if invalid_devices:\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v on the same NPU; \"\n            f\"invalid tensors: {', '.join(invalid_devices)}\"\n        )\n    if not (q.dtype == k.dtype == v.dtype):\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v with the same dtype\"\n        )\n    if k.shape[:2] != v.shape[:2]:\n        raise ValueError(\n            \"NPU packed attention requires matching K/V token and head counts\"\n        )\n    if q.shape[-1] != k.shape[-1]:\n        raise ValueError(\"NPU packed attention requires matching Q/K head dimensions\")\n\n    q_boundaries = _packed_boundaries(\n        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], \"cu_seqlens_q\"\n    )\n    k_boundaries = _packed_boundaries(\n        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], \"cu_seqlens_k\"\n    )\n    if len(q_boundaries) != len(k_boundaries):\n        raise ValueError(\"cu_seqlens_q and cu_seqlens_k must describe the same batch\")\n\n    q_nonempty = [\n        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])\n    ]\n    k_nonempty = [\n        stop > start for start, stop in zip(k_boundaries[:-1], k_boundaries[1:])\n    ]\n    if q_nonempty != k_nonempty:\n        raise NotImplementedError(","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L73-L109","documentation":"q.shape[-1] (query head dim) must equal k.shape[-1] (key head dim); the NPU fused attention kernel does not support different query and key head dimensions (no MQA-style head-dim projection inside the kernel).","triggerScenarios":"Passing q with head_dim 128 and k with head_dim 64 (e.g. a model with asymmetric q/k head dims), which some architectures and some backends support but this kernel does not.","commonSituations":"Multi-modal/diffusion models with per-layer head-dim tweaks; loading only some projections at a different head dim due to a config mismatch; porting a model that relied on a GPU backend's support for unequal head dims.","solutions":["Check the model config: num_head_dims for query and key must match when using this backend","If the model genuinely has unequal q/k head dims, project q to k's head dim before the call or use a different backend","Fix accidental head-dim mismatches from a wrong checkpoint/config (e.g. wrong rope or qk layer weights)"],"exampleFix":"# before: q head_dim=128, k head_dim=64\nout = fused_infer_attention_varlen(q, k, v, cu_q, cu_k)\n# after\nq_proj = q[..., : k.shape[-1]]  # only if truncation is semantically valid\n# better: use a backend that supports asymmetric head dims","handlingStrategy":"validation","validationCode":"assert q.shape[-1] == k.shape[-1], \"Q/K head dims must match on NPU kernel\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Verify q/k head_dim equality when loading model configs","Choose a different backend for architectures with asymmetric head dims"],"tags":["npu","ascend","head-dim","shape-validation"],"backgroundTag":"head-dimension-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}