sgl-project/sglang · error · ValueError

sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positi

Error message

sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive multiple of 64, got {h_q}

What it means

The CUDA kernel uses a block size B_H=64 and launches h_q/64 CTAs; it requires the TP-local query head count h_q to be a positive multiple of 64. Zero or non-multiple head counts would launch zero CTAs and return uninitialized outputs, so they are rejected explicitly (comment notes this could otherwise look like a hang).

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:350

    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():
        raise ValueError("indices must be contiguous")

    if kv_d_qk != d_qk:
        raise ValueError(f"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}")

    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.
    # Reject unpadded TP-local head counts instead of launching zero CTAs and
    # returning uninitialized outputs, which can appear to callers as a hang or
    # a later collective failure.
    if h_q == 0 or h_q % 64 != 0:
        raise ValueError(
            "sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive "
            f"multiple of 64, got {h_q}"
        )

    if h_kv != 1:
        raise ValueError(f"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}")

    if d_qk not in (512, 576):
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}"
        )

    if indices.shape[:2] != (s_q, h_kv):
        raise ValueError(
            "indices must have shape "
            f"({s_q}, {h_kv}, topk), got {tuple(indices.shape)}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Choose a TP degree such that num_attention_heads / tp_size is a multiple of 64 (e.g. TP in {1,2,4,8} for 128 heads)
2. Pad h_q to the next multiple of 64 if your model supports head padding
3. Verify h_q is computed from the correct config (num_attention_heads, not num_kv_heads) and is nonzero

Example fix

# before: 128 heads, tp=3 -> h_q=43
# after: use tp=4 -> h_q=32? no -> use tp=2 -> h_q=64 (multiple of 64)
server_args.tensor_parallel_size = 2

h_q = q.shape[1]
assert h_q > 0 and h_q % 64 == 0, f"h_q={h_q} must be a positive multiple of 64"

def head_count_ok(q: torch.Tensor) -> bool:
    h = q.shape[1]
    return h > 0 and h % 64 == 0

When it happens

Trigger: Running with a TP degree that produces h_q not divisible by 64 (e.g. 128 total heads with TP=3 gives h_q≈43), or h_q==0 due to a misconfigured head allocation.

Common situations: Unusual tensor-parallel sizes; small models with head counts like 32; head-padding step skipped when building the per-rank q tensor.

Related errors

• sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}
• Q8KV8 sparse-prefill topk width must be a positive multiple
• PD disagg: heterogeneous TP not supported for MiniMax sparse
• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8780904c0e559cf2. Report an issue: GitHub.