sgl-project/sglang · error · ValueError

cu_seqlens_q and cu_seqlens_k must describe the same batch

Error message

cu_seqlens_q and cu_seqlens_k must describe the same batch

What it means

After validating q and k boundaries independently, the function requires len(cu_seqlens_q) == len(cu_seqlens_k): both must describe the same number of sequences (B+1 entries each). Varlen self-attention needs per-sequence pairs; different batch sizes cannot be paired.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:100

    if not (q.dtype == k.dtype == v.dtype):
        raise ValueError(
            "NPU packed attention requires q, k, and v with the same dtype"
        )
    if k.shape[:2] != v.shape[:2]:
        raise ValueError(
            "NPU packed attention requires matching K/V token and head counts"
        )
    if q.shape[-1] != k.shape[-1]:
        raise ValueError("NPU packed attention requires matching Q/K head dimensions")

    q_boundaries = _packed_boundaries(
        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], "cu_seqlens_q"
    )
    k_boundaries = _packed_boundaries(
        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], "cu_seqlens_k"
    )
    if len(q_boundaries) != len(k_boundaries):
        raise ValueError("cu_seqlens_q and cu_seqlens_k must describe the same batch")

    q_nonempty = [
        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])
    ]
    k_nonempty = [
        stop > start for start, stop in zip(k_boundaries[:-1], k_boundaries[1:])
    ]
    if q_nonempty != k_nonempty:
        raise NotImplementedError(
            "NPU packed attention does not support a sequence that is empty only "
            "on the query or key/value side"
        )
    actual_seq_lengths = [
        stop for stop, nonempty in zip(q_boundaries[1:], q_nonempty) if nonempty
    ]
    actual_seq_lengths_kv = [
        stop for stop, nonempty in zip(k_boundaries[1:], k_nonempty) if nonempty
    ]

View on GitHub (pinned to 0132848349)

Solutions

1. Build both cu_seqlens from the same per-sequence metadata: same list of (q_len, k_len) pairs, cumsum each side
2. For cross-attention with different q/kv batch structures, use a kernel that supports it — this one is self-attention-shaped
3. When chunking KV, keep one boundary entry per sequence on both sides (padding zero-length entries) rather than dropping entries

Example fix

# before
seqs = [(5, 9), (7, 12), (3, 4)]
cu_q = cumsum([0, 5, 7, 3]); cu_k = cumsum([0, 9, 12])  # dropped one
# after
cu_q = torch.tensor([0, 5, 12, 15], dtype=torch.int32)
cu_k = torch.tensor([0, 9, 21, 25], dtype=torch.int32)
assert cu_q.numel() == cu_k.numel()

assert cu_q.numel() == cu_k.numel(), "q/k boundaries must describe the same batch"

When it happens

Trigger: Passing cu_seqlens_q for 4 sequences (5 entries) and cu_seqlens_k for 3 sequences (4 entries); typically when q and k boundaries are built from different batch metadata (e.g. chunked prefill splitting one side).

Common situations: Cross-attention varlen where q batch and kv batch legitimately differ (unsupported here); ring-KV chunk code that rebuilds k boundaries per chunk but reuses stale q boundaries; filtering empty sequences from one list but not the other.

Related errors

• {name} and its host copy must have the same length
• {name} must start with 0 and contain at least one sequence
• {name} must be non-decreasing
• {name} is required for NPU packed attention
• {name} must be a 1D int32 or int64 tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/96ac66f040f18c17. Report an issue: GitHub.