sgl-project/sglang · error · ValueError

DeepSeek-V4 flashmla_sparse_q8 prefill requires SM90 CUDA GP

Error message

DeepSeek-V4 flashmla_sparse_q8 prefill requires SM90 CUDA GPUs.

What it means

The DeepSeek-V4 flashmla_sparse_q8 prefill path (quantized KV, sparse MLA) is built on SM90 (Hopper) kernels. The backend __init__ checks is_sm90_supported() and rejects the configuration on non-SM90 GPUs before any capture or inference.

Source

Thrown at python/sglang/srt/layers/attention/deepseek_v4_backend.py:578

        self.hisparse_coordinator = model_runner.hisparse_coordinator
        self.req_to_token = model_runner.req_to_token_pool.req_to_token
        self.MAX_SEQ_LEN_FOR_CAPTURE = self.req_to_token.shape[1]

        assert isinstance(self.token_to_kv_pool, DeepSeekV4TokenToKVPool)
        self.c4_topk = getattr(
            model_runner.model_config.hf_text_config, "index_topk", C4_TOPK
        )

        self.enable_deepseek_v4_fp4_indexer: bool = (
            model_runner.server_args.enable_deepseek_v4_fp4_indexer
        )
        self.dsa_topk_backend: DSATopKBackend = DSATopKBackend.resolve(model_runner)
        self.dsv4_prefill_backend: str = getattr(
            model_runner.server_args, "dsv4_prefill_backend", "auto"
        )
        if use_dsv4_q8kv8_sparse_prefill(self.dsv4_prefill_backend):
            if not is_sm90_supported():
                raise ValueError(
                    "DeepSeek-V4 flashmla_sparse_q8 prefill requires SM90 CUDA GPUs."
                )
            if self.head_dim_v != 512:
                raise ValueError(
                    "DeepSeek-V4 flashmla_sparse_q8 prefill requires d_v=512, "
                    f"got {self.head_dim_v}."
                )
        self._q8kv8_qpad_buf = None
        self._q8kv8_attn_sink_pad = None
        self._q8kv8_identity_scale = None
        self.topk = get_spec().speculative_eagle_topk or 0
        assert self.topk in [0, 1], "MTP Topk > 1 not supported for DeepSeek V4"
        self.mtp_enabled = self.topk > 0
        self.speculative_num_steps = speculative_num_steps
        self.speculative_num_draft_tokens: int = get_spec().speculative_num_draft_tokens
        if self.speculative_num_draft_tokens is not None:
            # Persistent target-verify metadata buffers. Allocated here (not
            # lazily) so they are ordinary tensors: the first touch of a lazy

View on GitHub (pinned to 0132848349)

Solutions

  1. Remove or change the dsv4_prefill_backend override to a backend supported on your GPU (e.g. default/auto non-q8 path)
  2. Run on an SM90 (H100/H200) GPU to use flashmla_sparse_q8

Example fix

# before
--dsv4-prefill-backend flashmla_sparse_q8  # on A100
# after
--dsv4-prefill-backend auto
Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.utils import is_sm90_supported
if use_dsv4_q8kv8_sparse_prefill(getattr(server_args, "dsv4_prefill_backend", "auto")) and not is_sm90_supported():
    server_args.dsv4_prefill_backend = "auto"  # downgrade before launch

Type guard

def supports_q8_sparse_prefill() -> bool:
    import torch
    return torch.cuda.is_available() and torch.cuda.get_device_capability()[0] == 9

Prevention

When it happens

Trigger: Setting server_args.dsv4_prefill_backend to the flashmla_sparse_q8 (q8kv8 sparse) mode — or resolving to it via 'auto' on a qualifying config — on a GPU that is not compute capability 9.0 (e.g. A100 SM80, Ada SM89, or non-CUDA devices).

Common situations: Borrowing a DeepSeek-V4 launch config tuned on H100 and running it on A100/L40S; enabling the q8 sparse prefill backend flag for memory savings on older hardware.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cd10285b9d5dc8a4. Report an issue: GitHub.