sgl-project/sglang · error · ValueError

NPU packed attention requires q, k, and v with the same dtyp

Error message

NPU packed attention requires q, k, and v with the same dtype

What it means

NPU packed attention requires q, k, and v to share the same dtype; the fused kernel does not support mixed-precision inputs. A common case is fp16 q with bf16 KV cache (or vice versa), or an unquantized tensor alongside a quantized one.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:83

    tensors = {"q": q, "k": k, "v": v}
    invalid_layouts = [name for name, tensor in tensors.items() if tensor.ndim != 3]
    if invalid_layouts:
        raise ValueError(
            "NPU packed attention requires q, k, and v in [T, N, D] layout; "
            f"invalid tensors: {', '.join(invalid_layouts)}"
        )
    invalid_devices = [
        name
        for name, tensor in tensors.items()
        if tensor.device.type != "npu" or tensor.device != q.device
    ]
    if invalid_devices:
        raise ValueError(
            "NPU packed attention requires q, k, and v on the same NPU; "
            f"invalid tensors: {', '.join(invalid_devices)}"
        )
    if not (q.dtype == k.dtype == v.dtype):
        raise ValueError(
            "NPU packed attention requires q, k, and v with the same dtype"
        )
    if k.shape[:2] != v.shape[:2]:
        raise ValueError(
            "NPU packed attention requires matching K/V token and head counts"
        )
    if q.shape[-1] != k.shape[-1]:
        raise ValueError("NPU packed attention requires matching Q/K head dimensions")

    q_boundaries = _packed_boundaries(
        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], "cu_seqlens_q"
    )
    k_boundaries = _packed_boundaries(
        cu_seqlens_k, cu_seqlens_k_host, k.shape[0], "cu_seqlens_k"
    )
    if len(q_boundaries) != len(k_boundaries):
        raise ValueError("cu_seqlens_q and cu_seqlens_k must describe the same batch")

View on GitHub (pinned to 0132848349)

Solutions

1. Cast to a single dtype before the call: k = k.to(q.dtype); v = v.to(q.dtype) (prefer the compute dtype, usually bf16 on Ascend)
2. Align KV cache dtype with the model dtype at allocation time
3. If you intended quantized KV attention, use the dedicated quantized path, not this kernel

Example fix

# before
out = fused_infer_attention_varlen(q_bf16, k_fp16, v_fp16, cu_q, cu_k)
# after
k = k.to(q.dtype)
v = v.to(q.dtype)
out = fused_infer_attention_varlen(q, k, v, cu_q, cu_k)

if k.dtype != q.dtype:
    k = k.to(q.dtype)
if v.dtype != q.dtype:
    v = v.to(q.dtype)

def same_dtype(*ts: torch.Tensor) -> bool:
    return all(t.dtype == ts[0].dtype for t in ts)

When it happens

Trigger: Calling fused_infer_attention_varlen where q.dtype != k.dtype or k.dtype != v.dtype — e.g. model weights in bf16 but projections computed in fp16, or a KV cache stored in a different dtype than the query path.

Common situations: Autocast/AMP producing mixed dtypes across branches; loading a KV cache in fp16 while the model runs bf16; partial quantization (fp8 KV) not handled by this kernel.

Related errors

• {name} and its host copy must have the same length
• {name} must start with 0 and contain at least one sequence
• {name} must be non-decreasing
• NPU packed attention requires q, k, and v on the same NPU; i
• cu_seqlens_q and cu_seqlens_k must describe the same batch

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5a32e337b8b4429a. Report an issue: GitHub.