sgl-project/sglang · error · ValueError

attn_sink must be on q's device {device}, got {attn_sink.dev

Error message

attn_sink must be on q's device {device}, got {attn_sink.device}

What it means

sparse_mla_q8kv8_prefill_fwd requires attn_sink to sit on the same CUDA device as the q tensor (multi-GPU safety). Even a CUDA tensor on a different GPU is rejected, since the kernel would otherwise read cross-device memory and either crash or silently corrupt results.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:415

    if d_v != 512:
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}"
        )

    if attn_sink is not None and topk_length is None:
        raise ValueError("attn_sink requires topk_length to be provided as well")

    if attn_sink is not None:
        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:
            raise ValueError(
                f"attn_sink must be float32 with shape ({h_q},), got "
                f"{tuple(attn_sink.shape)}/{attn_sink.dtype}"
            )
        if not attn_sink.is_cuda:
            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):
            raise ValueError(f"{name} must be a torch.Tensor")
        if not scale.is_cuda:
            raise ValueError(f"{name} must be a CUDA tensor")
        if scale.device != device:
            raise ValueError(
                f"{name} must be on q's device {device}, got {scale.device}"
            )
        if scale.dtype != torch.float32:
            raise ValueError(f"{name} must be float32, got {scale.dtype}")
        if scale.numel() != 1:
            raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate or move the sink with device=q.device explicitly
2. In TP workers, create per-rank sink buffers instead of sharing one global buffer

Example fix

// before
attn_sink = torch.zeros(h_q, dtype=torch.float32, device='cuda:0')
// after
attn_sink = torch.zeros(h_q, dtype=torch.float32, device=q.device)

assert attn_sink.device == q.device, f'sink on {attn_sink.device}, q on {q.device}'

When it happens

Trigger: Running with CUDA_VISIBLE_DEVICES spanning multiple GPUs and passing an attn_sink allocated on cuda:0 while q lives on cuda:1 (device mismatch).

Common situations: Tensor-parallel or multi-GPU inference where a sink buffer is allocated once on the default device but reused by workers pinned to other devices.

Related errors

• {name} must be on q's device {device}, got {scale.device}
• All inputs must be on the same device.
• All inputs must be on the same device.
• indices must be on q's device {device}, got {indices.device}
• sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5cc08fee06d069c9. Report an issue: GitHub.