sgl-project/sglang · error · ValueError

{name} must be on q's device {device}, got {scale.device}

Error message

{name} must be on q's device {device}, got {scale.device}

What it means

q_scale and kv_scale must reside on the same CUDA device as q. In multi-GPU runs a scale tensor on another GPU is rejected to prevent cross-device pointer dereference inside the kernel.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:427

                f"attn_sink must be float32 with shape ({h_q},), got "
                f"{tuple(attn_sink.shape)}/{attn_sink.dtype}"
            )
        if not attn_sink.is_cuda:
            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):
            raise ValueError(f"{name} must be a torch.Tensor")
        if not scale.is_cuda:
            raise ValueError(f"{name} must be a CUDA tensor")
        if scale.device != device:
            raise ValueError(
                f"{name} must be on q's device {device}, got {scale.device}"
            )
        if scale.dtype != torch.float32:
            raise ValueError(f"{name} must be float32, got {scale.dtype}")
        if scale.numel() != 1:
            raise ValueError(
                f"{name} must be a scalar tensor, got shape {tuple(scale.shape)}"
            )
        if not scale.is_contiguous():
            raise ValueError(f"{name} must be contiguous")

    if out is None:
        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)
    else:
        _check_out_buffer(out, "out", (s_q, h_q, d_v), torch.bfloat16, device)

    if max_logits is None:
        max_logits = torch.empty(s_q, h_q, dtype=torch.float32, device=device)

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate scales with device=q.device per rank
2. Move shared buffers: kv_scale = kv_scale.to(q.device) at forward time

Example fix

// before
q_scale = GLOBAL_SCALES['q']  # on cuda:0
// after
q_scale = GLOBAL_SCALES['q'].to(q.device)

q_scale = q_scale.to(q.device); kv_scale = kv_scale.to(q.device)

When it happens

Trigger: Tensor-parallel inference where scales are allocated once on cuda:0 but the worker's q tensor is on cuda:N (N != 0).

Common situations: Sharing a single global scale buffer across TP ranks, or initializing scales before torch.cuda.set_device is called for the worker.

Related errors

• attn_sink must be on q's device {device}, got {attn_sink.dev
• All inputs must be on the same device.
• All inputs must be on the same device.
• {name} must be a torch.Tensor
• {name} must be a CUDA tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f8d9aefdae04a856. Report an issue: GitHub.