sgl-project/sglang · error · ValueError

topk_length must be on q's device {device}, got {topk_length

Error message

topk_length must be on q's device {device}, got {topk_length.device}

What it means

Even when topk_length is a CUDA tensor, it must be on the same device as q, since the kernel launches on q's device and stream. A length tensor on another GPU triggers this error.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:387

    if indices.dtype != torch.int32:
        raise ValueError(f"indices must be int32, got {indices.dtype}")

    if topk == 0 or topk % 128 != 0:
        raise ValueError(
            "Q8KV8 sparse-prefill topk width must be a positive multiple of 128, "
            f"got {topk}"
        )

    if topk_length is not None:
        if topk_length.shape != (s_q,) or topk_length.dtype != torch.int32:
            raise ValueError(
                f"topk_length must be int32 with shape ({s_q},), got "
                f"{tuple(topk_length.shape)}/{topk_length.dtype}"
            )
        if not topk_length.is_cuda:
            raise ValueError("topk_length must be a CUDA tensor")
        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():
            raise ValueError("topk_length must be contiguous")
        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():
            raise ValueError(
                "topk_length values must satisfy " f"0 <= topk_length <= topk ({topk})"
            )

    if d_v != 512:
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}"
        )

    if attn_sink is not None and topk_length is None:
        raise ValueError("attn_sink requires topk_length to be provided as well")

View on GitHub (pinned to 0132848349)

Solutions

1. topk_length = topk_length.to(q.device)
2. Use explicit device indices (torch.device('cuda', local_rank)) instead of bare .cuda()
3. Verify rank-local device assignment before constructing any metadata tensors

Example fix

// before
topk_length = topk_length.cuda()  # lands on cuda:0 in every rank
// after
topk_length = topk_length.to(q.device)

assert topk_length.device == q.device

def lengths_on_device(q: torch.Tensor, tl: torch.Tensor) -> bool:
    return tl.is_cuda and tl.device == q.device

When it happens

Trigger: q on cuda:0 with topk_length on cuda:1 in a TP worker; lengths moved to the wrong rank's device during broadcast.

Common situations: Multi-GPU pipelines where metadata tensors are gathered/broadcast across ranks; hard-coded .cuda() (device 0) in multi-GPU processes.

Related errors

• topk_length must be a CUDA tensor
• {name}_block_cnt and {name}_block_idx must be on the same de
• indices must be on q's device {device}, got {indices.device}
• QKV and cos/sin tensors must be on the same CUDA device
• LPLB fused solver requires CUDA tensors; got A on {A.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8caa99387313e65b. Report an issue: GitHub.