{"record":{"id":"96ac66f040f18c17","repo":"sgl-project/sglang","slug":"cu-seqlens-q-and-cu-seqlens-k-must-describe-the-sa","errorCode":null,"errorMessage":"cu_seqlens_q and cu_seqlens_k must describe the same batch","messagePattern":"cu_seqlens_q and cu_seqlens_k must describe the same batch","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py","lineNumber":100,"sourceCode":"    if not (q.dtype == k.dtype == v.dtype):\n        raise ValueError(\n            \"NPU packed attention requires q, k, and v with the same dtype\"\n        )\n    if k.shape[:2] != v.shape[:2]:\n        raise ValueError(\n            \"NPU packed attention requires matching K/V token and head counts\"\n        )\n    if q.shape[-1] != k.shape[-1]:\n        raise ValueError(\"NPU packed attention requires matching Q/K head dimensions\")\n\n    q_boundaries = _packed_boundaries(\n        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], \"cu_seqlens_q\"\n    )\n    k_boundaries = _packed_boundaries(\n        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], \"cu_seqlens_k\"\n    )\n    if len(q_boundaries) != len(k_boundaries):\n        raise ValueError(\"cu_seqlens_q and cu_seqlens_k must describe the same batch\")\n\n    q_nonempty = [\n        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])\n    ]\n    k_nonempty = [\n        stop > start for start, stop in zip(k_boundaries[:-1], k_boundaries[1:])\n    ]\n    if q_nonempty != k_nonempty:\n        raise NotImplementedError(\n            \"NPU packed attention does not support a sequence that is empty only \"\n            \"on the query or key/value side\"\n        )\n    actual_seq_lengths = [\n        stop for stop, nonempty in zip(q_boundaries[1:], q_nonempty) if nonempty\n    ]\n    actual_seq_lengths_kv = [\n        stop for stop, nonempty in zip(k_boundaries[1:], k_nonempty) if nonempty\n    ]","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py#L82-L118","documentation":"After validating q and k boundaries independently, the function requires len(cu_seqlens_q) == len(cu_seqlens_k): both must describe the same number of sequences (B+1 entries each). Varlen self-attention needs per-sequence pairs; different batch sizes cannot be paired.","triggerScenarios":"Passing cu_seqlens_q for 4 sequences (5 entries) and cu_seqlens_k for 3 sequences (4 entries); typically when q and k boundaries are built from different batch metadata (e.g. chunked prefill splitting one side).","commonSituations":"Cross-attention varlen where q batch and kv batch legitimately differ (unsupported here); ring-KV chunk code that rebuilds k boundaries per chunk but reuses stale q boundaries; filtering empty sequences from one list but not the other.","solutions":["Build both cu_seqlens from the same per-sequence metadata: same list of (q_len, k_len) pairs, cumsum each side","For cross-attention with different q/kv batch structures, use a kernel that supports it — this one is self-attention-shaped","When chunking KV, keep one boundary entry per sequence on both sides (padding zero-length entries) rather than dropping entries"],"exampleFix":"# before\nseqs = [(5, 9), (7, 12), (3, 4)]\ncu_q = cumsum([0, 5, 7, 3]); cu_k = cumsum([0, 9, 12])  # dropped one\n# after\ncu_q = torch.tensor([0, 5, 12, 15], dtype=torch.int32)\ncu_k = torch.tensor([0, 9, 21, 25], dtype=torch.int32)\nassert cu_q.numel() == cu_k.numel()","handlingStrategy":"validation","validationCode":"assert cu_q.numel() == cu_k.numel(), \"q/k boundaries must describe the same batch\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Build both cu_seqlens from one list of (q_len, k_len) pairs","Don't filter entries from one side only"],"tags":["npu","ascend","varlen","batch-mismatch","validation"],"backgroundTag":"batch-size-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}