{"record":{"id":"54371b8d85109a75","repo":"sgl-project/sglang","slug":"npu-packed-attention-does-not-support-a-sequence-t","errorCode":null,"errorMessage":"NPU packed attention does not support a sequence that is empty only on the query or key/value side","messagePattern":"NPU packed attention does not support a sequence that is empty only on the query or key/value side","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":109,"sourceCode":"        raise ValueError(\"NPU packed attention requires matching Q/K head dimensions\")\n\n    q_boundaries = _packed_boundaries(\n        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], \"cu_seqlens_q\"\n    )\n    k_boundaries = _packed_boundaries(\n        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], \"cu_seqlens_k\"\n    )\n    if len(q_boundaries) != len(k_boundaries):\n        raise ValueError(\"cu_seqlens_q and cu_seqlens_k must describe the same batch\")\n\n    q_nonempty = [\n        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])\n    ]\n    k_nonempty = [\n        stop > start for start, stop in zip(k_boundaries[:-1], k_boundaries[1:])\n    ]\n    if q_nonempty != k_nonempty:\n        raise NotImplementedError(\n            \"NPU packed attention does not support a sequence that is empty only \"\n            \"on the query or key/value side\"\n        )\n    actual_seq_lengths = [\n        stop for stop, nonempty in zip(q_boundaries[1:], q_nonempty) if nonempty\n    ]\n    actual_seq_lengths_kv = [\n        stop for stop, nonempty in zip(k_boundaries[1:], k_nonempty) if nonempty\n    ]\n    if not actual_seq_lengths:\n        output = torch.empty_like(q)\n        if return_softmax_lse:\n            lse = torch.empty(\n                (q.shape[1], q.shape[0]), dtype=torch.float32, device=q.device\n            )\n            return output, lse\n        return output\n","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L91-L127","documentation":"The kernel cannot handle a sequence that has zero query tokens but nonzero KV tokens (or the reverse). q_nonempty and k_nonempty (per-sequence emptiness derived from the boundaries) must match element-wise; this is a NotImplementedError because the underlying npu_fused_infer_attention_score has no representation for such sequences.","triggerScenarios":"A boundary sequence where some entry has q_len==0 but k_len>0 — e.g. chunked prefill/ring attention producing a KV-only chunk with no new queries, or prefill sequences with empty query prefixes. Note: sequences empty on BOTH sides are fine (they are filtered into actual_seq_lengths).","commonSituations":"Ring-attention KV chunking where a rank holds KV but no queries for some sequence; speculative/prefill schedulers producing zero-length query segments; naive boundary construction that inserts a 0-length q entry for a KV-only sequence.","solutions":["Drop or merge KV-only sequences from the batch before calling: remove boundary entries where q_len==0 and k_len>0 (and process those KV chunks separately)","Ensure both sides use zero-length entries for the same sequences — an empty sequence must be empty on both q and k","Restructure ring-KV chunking so each chunk's queries are nonempty whenever its KV is"],"exampleFix":"# before: seq 2 has q_len=0 but k_len=8\ncu_q = [0, 5, 5, 12]; cu_k = [0, 9, 17, 25]\n# after: drop the KV-only sequence\ncu_q = [0, 5, 12]; cu_k = [0, 9, 25]  # handle seq 2's KV elsewhere","handlingStrategy":"validation","validationCode":"q_ne = [b > a for a, b in zip(cu_q[:-1], cu_q[1:])]\nk_ne = [b > a for a, b in zip(cu_k[:-1], cu_k[1:])]\nassert q_ne == k_ne, \"sequence empty on only one side is unsupported\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Drop KV-only or query-only sequences before calling the kernel","Keep emptiness symmetric: a sequence is either empty on both sides or neither"],"tags":["npu","ascend","varlen","ring-attention","not-implemented"],"backgroundTag":"empty-sequence-in-batch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}