sgl-project/sglang · error · ValueError

topk_length must be int32 with shape ({s_q},), got {tuple(to

Error message

topk_length must be int32 with shape ({s_q},), got {tuple(topk_length.shape)}/{topk_length.dtype}

What it means

When the optional variable-length topk mode is used, topk_length must be an int32 tensor of shape (s_q,) holding the effective number of selected KV entries per query token. This check enforces both shape and dtype before the values are validated.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:380

    if indices.shape[:2] != (s_q, h_kv):
        raise ValueError(
            "indices must have shape "
            f"({s_q}, {h_kv}, topk), got {tuple(indices.shape)}"
        )

    if indices.dtype != torch.int32:
        raise ValueError(f"indices must be int32, got {indices.dtype}")

    if topk == 0 or topk % 128 != 0:
        raise ValueError(
            "Q8KV8 sparse-prefill topk width must be a positive multiple of 128, "
            f"got {topk}"
        )

    if topk_length is not None:
        if topk_length.shape != (s_q,) or topk_length.dtype != torch.int32:
            raise ValueError(
                f"topk_length must be int32 with shape ({s_q},), got "
                f"{tuple(topk_length.shape)}/{topk_length.dtype}"
            )
        if not topk_length.is_cuda:
            raise ValueError("topk_length must be a CUDA tensor")
        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():
            raise ValueError("topk_length must be contiguous")
        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():
            raise ValueError(
                "topk_length values must satisfy " f"0 <= topk_length <= topk ({topk})"
            )

    if d_v != 512:

View on GitHub (pinned to 0132848349)

Solutions

1. Reshape to (s_q,) matching q's token count: topk_length = topk_length.reshape(-1).to(torch.int32)
2. Ensure the producer writes int32 per flattened prefill token
3. Confirm s_q (q.shape[0]) equals len(topk_length)

Example fix

// before
topk_length = lengths_per_batch.to(torch.int64)  # wrong dtype/shape
// after
topk_length = lengths_per_batch.reshape(-1).to(torch.int32)

s_q = q.shape[0]
assert topk_length.shape == (s_q,) and topk_length.dtype == torch.int32

def topk_length_ok(q: torch.Tensor, tl: torch.Tensor) -> bool:
    return tl.shape == (q.shape[0],) and tl.dtype == torch.int32

When it happens

Trigger: Passing topk_length as int64, scalar, or shaped (bs, seq_len) instead of the flattened (s_q,) int32 vector.

Common situations: Variable topk per request from a scheduler; forgetting to flatten a batched tensor; dtype promotion from int64 counters in a producer kernel.

Related errors

• Reserved serve backend names cannot be used: {names}
• Unsupported type {type(data)}
• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp
• kv_scales must be fp32, got {kv_scales.dtype}
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/67e42e4cfa04f9cb. Report an issue: GitHub.