sgl-project/sglang · error · ValueError

must be on q's device , got

Error message

{name} must be on q's device {device}, got {scale.device}

What it means

The scale (logit softcap or similar) tensor must live on q's device. Move the named tensor to the same CUDA device as q.

Solutions

1. Allocate scales with device=q.device per rank
2. Move shared buffers: kv_scale = kv_scale.to(q.device) at forward time

Example fix

// before
q_scale = GLOBAL_SCALES['q']  # on cuda:0
// after
q_scale = GLOBAL_SCALES['q'].to(q.device)

q_scale = q_scale.to(q.device); kv_scale = kv_scale.to(q.device)

When it happens

Trigger: Tensor-parallel inference where scales are allocated once on cuda:0 but the worker's q tensor is on cuda:N (N != 0).

Common situations: Sharing a single global scale buffer across TP ranks, or initializing scales before torch.cuda.set_device is called for the worker.

Related errors

• attn_sink must be on q's device
• All inputs must be on the same device.
• All inputs must be on the same device.
• Kimi-K3 MLA K projection must remain GGUF Q4_0
• Kimi-K3 MLA V projection must remain GGUF Q2_K

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f8d9aefdae04a856. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:427

                f"attn_sink must be float32 with shape ({h_q},), got "
                f"{tuple(attn_sink.shape)}/{attn_sink.dtype}"
            )
        if not attn_sink.is_cuda:
            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):
            raise ValueError(f"{name} must be a torch.Tensor")
        if not scale.is_cuda:
            raise ValueError(f"{name} must be a CUDA tensor")
        if scale.device != device:
            raise ValueError(
                f"{name} must be on q's device {device}, got {scale.device}"
            )
        if scale.dtype != torch.float32:
            raise ValueError(f"{name} must be float32, got {scale.dtype}")
        if scale.numel() != 1:
            raise ValueError(
                f"{name} must be a scalar tensor, got shape {tuple(scale.shape)}"
            )
        if not scale.is_contiguous():
            raise ValueError(f"{name} must be contiguous")

    if out is None:
        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)
    else:
        _check_out_buffer(out, "out", (s_q, h_q, d_v), torch.bfloat16, device)

    if max_logits is None:
        max_logits = torch.empty(s_q, h_q, dtype=torch.float32, device=device)

View on GitHub (pinned to 0132848349)