sgl-project/sglang · error · ValueError

topk_length must be a CUDA tensor

Error message

topk_length must be a CUDA tensor

What it means

topk_length is optional, but if provided it must be a CUDA tensor: the kernel reads it on-device to apply per-token variable lengths. A CPU tensor (e.g. a list converted with torch.tensor on CPU) is rejected.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:385

        )

    if indices.dtype != torch.int32:
        raise ValueError(f"indices must be int32, got {indices.dtype}")

    if topk == 0 or topk % 128 != 0:
        raise ValueError(
            "Q8KV8 sparse-prefill topk width must be a positive multiple of 128, "
            f"got {topk}"
        )

    if topk_length is not None:
        if topk_length.shape != (s_q,) or topk_length.dtype != torch.int32:
            raise ValueError(
                f"topk_length must be int32 with shape ({s_q},), got "
                f"{tuple(topk_length.shape)}/{topk_length.dtype}"
            )
        if not topk_length.is_cuda:
            raise ValueError("topk_length must be a CUDA tensor")
        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():
            raise ValueError("topk_length must be contiguous")
        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():
            raise ValueError(
                "topk_length values must satisfy " f"0 <= topk_length <= topk ({topk})"
            )

    if d_v != 512:
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}"
        )

    if attn_sink is not None and topk_length is None:

View on GitHub (pinned to 0132848349)

Solutions

1. Move to the right GPU: topk_length = topk_length.to(q.device)
2. Create it directly on device: torch.full((s_q,), k, dtype=torch.int32, device=q.device)
3. Set torch.cuda.set_device(local_rank) so default constructions land on the right device

Example fix

// before
topk_length = torch.tensor(lengths, dtype=torch.int32)
// after
topk_length = torch.tensor(lengths, dtype=torch.int32, device=q.device)

assert topk_length.is_cuda, "topk_length must be a CUDA tensor"

def cuda_int32_lengths(tl: torch.Tensor) -> bool:
    return tl.is_cuda and tl.dtype == torch.int32

When it happens

Trigger: Passing torch.tensor([128]*s_q) (defaults to CPU) or lengths computed with numpy and converted without .cuda().

Common situations: Scheduler-computed lengths starting on CPU; debugging with CPU-constructed dummy tensors; multi-GPU runs where the default device is not set.

Related errors

• topk_length must be on q's device {device}, got {topk_length
• {name}_block_cnt and {name}_block_idx must be on the same de
• indices must be on q's device {device}, got {indices.device}
• QKV and cos/sin tensors must be on the same CUDA device
• LPLB fused solver requires CUDA tensors; got A on {A.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4be712f1fd4b74dc. Report an issue: GitHub.