sgl-project/sglang · error · ValueError

Q8KV8 sparse-prefill topk width must be a positive multiple

Error message

Q8KV8 sparse-prefill topk width must be a positive multiple of 128, got {topk}

What it means

The kernel's tile size for the sparse selection is 128, so the topk width (indices.shape[-1] or indices.shape[2]) must be a positive multiple of 128 (e.g. 128, 256, 512...). Values like 64 or 300 cannot be tiled and are rejected.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:373

    if h_kv != 1:
        raise ValueError(f"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}")

    if d_qk not in (512, 576):
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}"
        )

    if indices.shape[:2] != (s_q, h_kv):
        raise ValueError(
            "indices must have shape "
            f"({s_q}, {h_kv}, topk), got {tuple(indices.shape)}"
        )

    if indices.dtype != torch.int32:
        raise ValueError(f"indices must be int32, got {indices.dtype}")

    if topk == 0 or topk % 128 != 0:
        raise ValueError(
            "Q8KV8 sparse-prefill topk width must be a positive multiple of 128, "
            f"got {topk}"
        )

    if topk_length is not None:
        if topk_length.shape != (s_q,) or topk_length.dtype != torch.int32:
            raise ValueError(
                f"topk_length must be int32 with shape ({s_q},), got "
                f"{tuple(topk_length.shape)}/{topk_length.dtype}"
            )
        if not topk_length.is_cuda:
            raise ValueError("topk_length must be a CUDA tensor")
        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():

View on GitHub (pinned to 0132848349)

Solutions

1. Round topk up to the next multiple of 128 (e.g. 64 -> 128) and pad indices with dummy/valid indices
2. Only enable this q8kv8 sparse path when the configured topk is 128, 256, 384, ...
3. Use variable-length mode (topk_length) with padded width a multiple of 128 for intermediate effective topk values

Example fix

# before
topk = 64  # raises
# after
topk = 128
indices = torch.cat([indices, dummy_idx], dim=-1)  # pad width to 128

topk = indices.shape[-1]
assert topk > 0 and topk % 128 == 0, f"topk={topk} must be a positive multiple of 128"

def topk_width_ok(indices: torch.Tensor) -> bool:
    t = indices.shape[-1]
    return t > 0 and t % 128 == 0

When it happens

Trigger: Passing indices with last dim 64 (e.g. MLP-sparsity-style top-64), or an odd topk like 204 from a heuristic.

Common situations: Configuring sparse attention topk to a small value (64) or a non-multiple value expecting FlashMLA-style semantics; migrating configs from a kernel that allowed topk=64; per-request variable topk padded to a wrong width.

Related errors

• sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positi
• sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}
• topk_length values must satisfy 0 <= topk_length <= topk ({t
• The pointers must be multiple of 16 bytes.
• The last dimension ({input.shape[-1]}) x itemsize ({input.dt

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/04688db7f17c92ee. Report an issue: GitHub.