sgl-project/sglang · error · ValueError

NPU packed attention requires matching K/V token and head co

Error message

NPU packed attention requires matching K/V token and head counts

What it means

k and v must agree on their first two dims in [T, N, D] layout: same packed token count and same number of KV heads. The kernel requires K and V to describe identical token/head structure; a mismatch means the KV pair is internally inconsistent.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:87

            "NPU packed attention requires q, k, and v in [T, N, D] layout; "
            f"invalid tensors: {', '.join(invalid_layouts)}"
        )
    invalid_devices = [
        name
        for name, tensor in tensors.items()
        if tensor.device.type != "npu" or tensor.device != q.device
    ]
    if invalid_devices:
        raise ValueError(
            "NPU packed attention requires q, k, and v on the same NPU; "
            f"invalid tensors: {', '.join(invalid_devices)}"
        )
    if not (q.dtype == k.dtype == v.dtype):
        raise ValueError(
            "NPU packed attention requires q, k, and v with the same dtype"
        )
    if k.shape[:2] != v.shape[:2]:
        raise ValueError(
            "NPU packed attention requires matching K/V token and head counts"
        )
    if q.shape[-1] != k.shape[-1]:
        raise ValueError("NPU packed attention requires matching Q/K head dimensions")

    q_boundaries = _packed_boundaries(
        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], "cu_seqlens_q"
    )
    k_boundaries = _packed_boundaries(
        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], "cu_seqlens_k"
    )
    if len(q_boundaries) != len(k_boundaries):
        raise ValueError("cu_seqlens_q and cu_seqlens_k must describe the same batch")

    q_nonempty = [
        stop > start for start, stop in zip(q_boundaries[:-1], q_boundaries[1:])
    ]
    k_nonempty = [

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure k.shape[:2] == v.shape[:2]; slice both identically when truncating/chunking
2. Verify num_key_value_heads is consistent between k and v at cache allocation
3. Add assert k.shape[:2] == v.shape[:2] before the call in your wrapper

Example fix

# before
k_chunk = k[:t]  # only K truncated
# after
k_chunk = k[:t]
v_chunk = v[:t]
assert k_chunk.shape[:2] == v_chunk.shape[:2]

assert k.shape[:2] == v.shape[:2], "K/V token and head counts must match"

When it happens

Trigger: Passing k from a truncated KV cache but the full v (or different head counts, e.g. MQA k with N=1 but v with N=num_heads); slicing k along tokens without slicing v identically.

Common situations: KV cache paging bugs where K and V pools are indexed differently; ring-attention chunking that splits K but not V; weight-loading errors producing different head counts for k_proj vs v_proj.

Related errors

• {name} must end at the packed token count {total_tokens}, go
• NPU packed attention requires q, k, and v in [T, N, D] layou
• {name} is required for NPU packed attention
• {name} must be a 1D int32 or int64 tensor
• {name} and its host copy must have the same length

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f7daa8b7a9fcd2c8. Report an issue: GitHub.