sgl-project/sglang · error · ValueError

sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_q

Error message

sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}

What it means

The FP8 sparse prefill kernel is templated only for QK head dimensions 512 and 576 (the DeepSeek-MLA latent dims: 512 + optional 64 rope carriers). Any other d_qk is rejected because no compiled kernel specialization exists.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:359

    if kv_d_qk != d_qk:
        raise ValueError(f"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}")

    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.
    # Reject unpadded TP-local head counts instead of launching zero CTAs and
    # returning uninitialized outputs, which can appear to callers as a hang or
    # a later collective failure.
    if h_q == 0 or h_q % 64 != 0:
        raise ValueError(
            "sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive "
            f"multiple of 64, got {h_q}"
        )

    if h_kv != 1:
        raise ValueError(f"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}")

    if d_qk not in (512, 576):
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}"
        )

    if indices.shape[:2] != (s_q, h_kv):
        raise ValueError(
            "indices must have shape "
            f"({s_q}, {h_kv}, topk), got {tuple(indices.shape)}"
        )

    if indices.dtype != torch.int32:
        raise ValueError(f"indices must be int32, got {indices.dtype}")

    if topk == 0 or topk % 128 != 0:
        raise ValueError(
            "Q8KV8 sparse-prefill topk width must be a positive multiple of 128, "
            f"got {topk}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Confirm the model's qk_nope_head_dim + qk_rope_head_dim equals 512 or 576
2. If not, this kernel is not applicable; fall back to FlashMLA/FlashInfer/triton prefill for that model
3. Check that d_qk is derived from q.shape (last dim), not from a stale config value

Example fix

# before
model has qk dims 128+64=192 -> kernel raises
# after: route to a generic backend
attn_backend = 'fa3'  # instead of q8kv8 sparse prefill

d_qk = q.shape[-1]
assert d_qk in (512, 576), f"unsupported d_qk {d_qk}"

def supported_d_qk(q: torch.Tensor) -> bool:
    return q.shape[-1] in (512, 576)

When it happens

Trigger: q with per-head QK dim 128, 256, or 1024 passed to sparse_mla_q8kv8_prefill_fwd.

Common situations: Trying to reuse the SM90 sparse MLA kernel with a non-DeepSeek model; config picking up v_head_dim or kv_lora_rank incorrectly as the head dim.

Related errors

• sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v
• sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}
• The pointers must be multiple of 16 bytes.
• The last dimension ({input.shape[-1]}) x itemsize ({input.dt
• rope_pool_fused expects q/k/v to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6357225f152c6626. Report an issue: GitHub.