sgl-project/sglang · error · ValueError

DeepSeek-V4 flashmla_sparse_q8 prefill requires d_v=512, got

Error message

DeepSeek-V4 flashmla_sparse_q8 prefill requires d_v=512, got {self.head_dim_v}.

What it means

The flashmla_sparse_q8 prefill kernels for DeepSeek-V4 are hardcoded for a value-head dimension of 512; the backend validates self.head_dim_v at init and raises if the model config deviates.

Source

Thrown at python/sglang/srt/layers/attention/deepseek_v4_backend.py:582

        assert isinstance(self.token_to_kv_pool, DeepSeekV4TokenToKVPool)
        self.c4_topk = getattr(
            model_runner.model_config.hf_text_config, "index_topk", C4_TOPK
        )

        self.enable_deepseek_v4_fp4_indexer: bool = (
            model_runner.server_args.enable_deepseek_v4_fp4_indexer
        )
        self.dsa_topk_backend: DSATopKBackend = DSATopKBackend.resolve(model_runner)
        self.dsv4_prefill_backend: str = getattr(
            model_runner.server_args, "dsv4_prefill_backend", "auto"
        )
        if use_dsv4_q8kv8_sparse_prefill(self.dsv4_prefill_backend):
            if not is_sm90_supported():
                raise ValueError(
                    "DeepSeek-V4 flashmla_sparse_q8 prefill requires SM90 CUDA GPUs."
                )
            if self.head_dim_v != 512:
                raise ValueError(
                    "DeepSeek-V4 flashmla_sparse_q8 prefill requires d_v=512, "
                    f"got {self.head_dim_v}."
                )
        self._q8kv8_qpad_buf = None
        self._q8kv8_attn_sink_pad = None
        self._q8kv8_identity_scale = None
        self.topk = get_spec().speculative_eagle_topk or 0
        assert self.topk in [0, 1], "MTP Topk > 1 not supported for DeepSeek V4"
        self.mtp_enabled = self.topk > 0
        self.speculative_num_steps = speculative_num_steps
        self.speculative_num_draft_tokens: int = get_spec().speculative_num_draft_tokens
        if self.speculative_num_draft_tokens is not None:
            # Persistent target-verify metadata buffers. Allocated here (not
            # lazily) so they are ordinary tensors: the first touch of a lazy
            # buffer would inherit the caller's context, and a creation inside
            # an inference_mode forward would forbid the in-place updates the
            # graph-capture path performs outside inference mode.
            num_reqs = self.req_to_token.shape[0]

View on GitHub (pinned to 0132848349)

Solutions

Use the default prefill backend for non-512 head_dim_v checkpoints
Verify the checkpoint's config (kv_lora_rank / rope dims) matches stock DeepSeek-V4 with d_v=512
Switch to a stock DeepSeek-V4 model when you need flashmla_sparse_q8

Example fix

# before
--dsv4-prefill-backend flashmla_sparse_q8  # head_dim_v=256 checkpoint
# after
--dsv4-prefill-backend auto

Defensive patterns

Strategy: validation

Validate before calling

if use_dsv4_q8kv8_sparse_prefill(prefill_backend) and head_dim_v != 512:
    prefill_backend = "auto"  # don't request q8 sparse kernels

Type guard

def dsv4_q8_sparse_compatible(head_dim_v) -> bool:
    return head_dim_v == 512

Prevention

Validate checkpoint latent dims against backend requirements in a config lint
Don't assume V4 backend flags work on resized/fine-tuned checkpoints

When it happens

Trigger: Enabling the q8kv8 sparse prefill backend while the model config's head_dim_v (kv_lora_rank + qk_rope_head_dim composition) is not 512 — e.g. a DeepSeek-V4 variant or fine-tune with modified latent dimensions.

Common situations: Custom DeepSeek-V4 checkpoints with resized latent dims; mixing a V4 backend flag with a V3.x checkpoint whose head_dim_v differs.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1b8d72f6ce7cebaa. Report an issue: GitHub.