sgl-project/sglang · error · ValueError

NPU packed attention requires q, k, and v on the same NPU; i

Error message

NPU packed attention requires q, k, and v on the same NPU; invalid tensors: {', '.join(invalid_devices)}

What it means

All of q, k, v must be on device type 'npu' and on the same NPU device as q. The kernel torch.ops.npu.npu_fused_infer_attention_score is device-specific; CPU tensors or tensors scattered across different Ascend devices are rejected, with the offending names listed.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:78

    cu_seqlens_q_host: Sequence[int] | None = None,
    cu_seqlens_k_host: Sequence[int] | None = None,
    softmax_scale: float | None = None,
    return_softmax_lse: bool = False,
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
    tensors = {"q": q, "k": k, "v": v}
    invalid_layouts = [name for name, tensor in tensors.items() if tensor.ndim != 3]
    if invalid_layouts:
        raise ValueError(
            "NPU packed attention requires q, k, and v in [T, N, D] layout; "
            f"invalid tensors: {', '.join(invalid_layouts)}"
        )
    invalid_devices = [
        name
        for name, tensor in tensors.items()
        if tensor.device.type != "npu" or tensor.device != q.device
    ]
    if invalid_devices:
        raise ValueError(
            "NPU packed attention requires q, k, and v on the same NPU; "
            f"invalid tensors: {', '.join(invalid_devices)}"
        )
    if not (q.dtype == k.dtype == v.dtype):
        raise ValueError(
            "NPU packed attention requires q, k, and v with the same dtype"
        )
    if k.shape[:2] != v.shape[:2]:
        raise ValueError(
            "NPU packed attention requires matching K/V token and head counts"
        )
    if q.shape[-1] != k.shape[-1]:
        raise ValueError("NPU packed attention requires matching Q/K head dimensions")

    q_boundaries = _packed_boundaries(
        cu_seqlens_q, cu_seqlens_q_host, q.shape[0], "cu_seqlens_q"
    )
    k_boundaries = _packed_boundaries(

View on GitHub (pinned to 0132848349)

Solutions

1. Move all three to the same NPU: q, k, v = q.to('npu'), k.to('npu'), v.to('npu')
2. In tensor-parallel setups, gather/replicate KV to the same device as q before the call
3. Add an assert q.device == k.device == v.device and q.device.type == 'npu' before invoking

Example fix

# before
q = torch.randn(T, N, D)  # CPU
# after
q = torch.randn(T, N, D, device="npu")
k = k.to(q.device)
v = v.to(q.device)

assert q.device.type == "npu" and q.device == k.device == v.device

def on_same_npu(*ts: torch.Tensor) -> bool:
    return all(t.device.type == "npu" and t.device == ts[0].device for t in ts)

When it happens

Trigger: Passing CPU tensors (e.g. never moved after creation), mixing a KV cache on 'npu:1' with q on 'npu:0', or tensors left on 'cuda' after copying code from a GPU path.

Common situations: Initial bring-up of Ascend inference where tensors were created without device=; tensor-parallel code copying between devices; unit tests running on CPU by default.

Related errors

• {name} and its host copy must have the same length
• {name} must start with 0 and contain at least one sequence
• {name} must be non-decreasing
• NPU packed attention requires q, k, and v with the same dtyp
• cu_seqlens_q and cu_seqlens_k must describe the same batch

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f2fbfb11228a860c. Report an issue: GitHub.