sgl-project/sglang · error · NotImplementedError

NPU packed attention does not support a sequence that is emp

Error message

NPU packed attention does not support a sequence that is empty only on the query or key/value side

What it means

The kernel cannot handle a sequence that has zero query tokens but nonzero KV tokens (or the reverse). q_nonempty and k_nonempty (per-sequence emptiness derived from the boundaries) must match element-wise; this is a NotImplementedError because the underlying npu_fused_infer_attention_score has no representation for such sequences.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:109

        raise ValueError("NPU packed attention requires matching Q/K head dimensions")

    q_boundaries = _packed_boundaries(
        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], "cu_seqlens_q"
    )
    k_boundaries = _packed_boundaries(
        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], "cu_seqlens_k"
    )
    if len(q_boundaries) != len(k_boundaries):
        raise ValueError("cu_seqlens_q and cu_seqlens_k must describe the same batch")

    q_nonempty = [
        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])
    ]
    k_nonempty = [
        stop > start for start, stop in zip(k_boundaries[:-1], k_boundaries[1:])
    ]
    if q_nonempty != k_nonempty:
        raise NotImplementedError(
            "NPU packed attention does not support a sequence that is empty only "
            "on the query or key/value side"
        )
    actual_seq_lengths = [
        stop for stop, nonempty in zip(q_boundaries[1:], q_nonempty) if nonempty
    ]
    actual_seq_lengths_kv = [
        stop for stop, nonempty in zip(k_boundaries[1:], k_nonempty) if nonempty
    ]
    if not actual_seq_lengths:
        output = torch.empty_like(q)
        if return_softmax_lse:
            lse = torch.empty(
                (q.shape[1], q.shape[0]), dtype=torch.float32, device=q.device
            )
            return output, lse
        return output

View on GitHub (pinned to 0132848349)

Solutions

1. Drop or merge KV-only sequences from the batch before calling: remove boundary entries where q_len==0 and k_len>0 (and process those KV chunks separately)
2. Ensure both sides use zero-length entries for the same sequences — an empty sequence must be empty on both q and k
3. Restructure ring-KV chunking so each chunk's queries are nonempty whenever its KV is

Example fix

# before: seq 2 has q_len=0 but k_len=8
cu_q = [0, 5, 5, 12]; cu_k = [0, 9, 17, 25]
# after: drop the KV-only sequence
cu_q = [0, 5, 12]; cu_k = [0, 9, 25]  # handle seq 2's KV elsewhere

q_ne = [b > a for a, b in zip(cu_q[:-1], cu_q[1:])]
k_ne = [b > a for a, b in zip(cu_k[:-1], cu_k[1:])]
assert q_ne == k_ne, "sequence empty on only one side is unsupported"

When it happens

Trigger: A boundary sequence where some entry has q_len==0 but k_len>0 — e.g. chunked prefill/ring attention producing a KV-only chunk with no new queries, or prefill sequences with empty query prefixes. Note: sequences empty on BOTH sides are fine (they are filtered into actual_seq_lengths).

Common situations: Ring-attention KV chunking where a rank holds KV but no queries for some sequence; speculative/prefill schedulers producing zero-length query segments; naive boundary construction that inserts a 0-length q entry for a KV-only sequence.

Related errors

• {name} is required for NPU packed attention
• {name} must be a 1D int32 or int64 tensor
• {name} and its host copy must have the same length
• {name} must start with 0 and contain at least one sequence
• {name} must end at the packed token count {total_tokens}, go

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/54371b8d85109a75. Report an issue: GitHub.