sgl-project/sglang · error · ValueError

sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v

Error message

sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}

What it means

This sparse MLA kernel only supports value head dim d_v=512 (the DeepSeek latent value width). Unlike d_qk (512 or 576), the value projection width is fixed at 512, and other values are rejected.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:399

                f"topk_length must be int32 with shape ({s_q},), got "
                f"{tuple(topk_length.shape)}/{topk_length.dtype}"
            )
        if not topk_length.is_cuda:
            raise ValueError("topk_length must be a CUDA tensor")
        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():
            raise ValueError("topk_length must be contiguous")
        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():
            raise ValueError(
                "topk_length values must satisfy " f"0 <= topk_length <= topk ({topk})"
            )

    if d_v != 512:
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}"
        )

    if attn_sink is not None and topk_length is None:
        raise ValueError("attn_sink requires topk_length to be provided as well")

    if attn_sink is not None:
        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:
            raise ValueError(
                f"attn_sink must be float32 with shape ({h_q},), got "
                f"{tuple(attn_sink.shape)}/{attn_sink.dtype}"
            )
        if not attn_sink.is_cuda:
            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the value (output) head dim passed to the kernel is exactly 512
2. For DeepSeek-style models, split q into qk (512/576) and keep v/latent at 512
3. Use a different backend if your model genuinely has v_head_dim != 512

Example fix

# before
q layout implies d_v=576 (rope carriers included)
# after
q_nope_rope: d_qk=576; value path uses d_v=512 latent

assert d_v == 512, f"d_v must be 512, got {d_v}"

def supported_d_v(d_v: int) -> bool:
    return d_v == 512

When it happens

Trigger: Passing a q/o layout where the value dim derived from the tensors is 576 (e.g. including rope carriers in d_v) or 256/1024.

Common situations: Splitting heads incorrectly so rope dims leak into d_v; non-DeepSeek models with different v_head_dim; config using qk dims for the value path.

Related errors

• sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_q
• sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}
• The pointers must be multiple of 16 bytes.
• The last dimension ({input.shape[-1]}) x itemsize ({input.dt
• rope_pool_fused expects q/k/v to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f9df9cd748354eeb. Report an issue: GitHub.