sgl-project/sglang · error · ValueError

Q8KV8 sparse-prefill topk width must be a positive multiple…

Error message

Q8KV8 sparse-prefill topk width must be a positive multiple of 128, got {topk}

What it means

The kernel's tile size for the sparse selection is 128, so the topk width (indices.shape[-1] or indices.shape[2]) must be a positive multiple of 128 (e.g. 128, 256, 512...). Values like 64 or 300 cannot be tiled and are rejected.

Solutions

  1. Round topk up to the next multiple of 128 (e.g. 64 -> 128) and pad indices with dummy/valid indices
  2. Only enable this q8kv8 sparse path when the configured topk is 128, 256, 384, ...
  3. Use variable-length mode (topk_length) with padded width a multiple of 128 for intermediate effective topk values

Example fix

# before
topk = 64  # raises
# after
topk = 128
indices = torch.cat([indices, dummy_idx], dim=-1)  # pad width to 128
Defensive patterns

Strategy: validation

Validate before calling

topk = indices.shape[-1]
assert topk > 0 and topk % 128 == 0, f"topk={topk} must be a positive multiple of 128"

Type guard

def topk_width_ok(indices: torch.Tensor) -> bool:
    t = indices.shape[-1]
    return t > 0 and t % 128 == 0

Prevention

When it happens

Trigger: Passing indices with last dim 64 (e.g. MLP-sparsity-style top-64), or an odd topk like 204 from a heuristic.

Common situations: Configuring sparse attention topk to a small value (64) or a non-multiple value expecting FlashMLA-style semantics; migrating configs from a kernel that allowed topk=64; per-request variable topk padded to a wrong width.

Related errors


AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/04688db7f17c92ee. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:373

    if h_kv != 1:
        raise ValueError(f"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}")

    if d_qk not in (512, 576):
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}"
        )

    if indices.shape[:2] != (s_q, h_kv):
        raise ValueError(
            "indices must have shape "
            f"({s_q}, {h_kv}, topk), got {tuple(indices.shape)}"
        )

    if indices.dtype != torch.int32:
        raise ValueError(f"indices must be int32, got {indices.dtype}")

    if topk == 0 or topk % 128 != 0:
        raise ValueError(
            "Q8KV8 sparse-prefill topk width must be a positive multiple of 128, "
            f"got {topk}"
        )

    if topk_length is not None:
        if topk_length.shape != (s_q,) or topk_length.dtype != torch.int32:
            raise ValueError(
                f"topk_length must be int32 with shape ({s_q},), got "
                f"{tuple(topk_length.shape)}/{topk_length.dtype}"
            )
        if not topk_length.is_cuda:
            raise ValueError("topk_length must be a CUDA tensor")
        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():

View on GitHub (pinned to 0132848349)