sgl-project/sglang · error · RuntimeError

--kv-cache-dtype=nvfp4 requires Blackwell SM100 or SM120. Us

Error message

--kv-cache-dtype=nvfp4 requires Blackwell SM100 or SM120. Use --kv-cache-dtype=fp4_mx_block16 for the block-size-16 FP4 recipe.

What it means

--kv-cache-dtype nvfp4 uses the NVFP4 (FP4 with per-block FP8 scales) representation whose kernels require Blackwell SM100 or SM120. On other CUDA GPUs the resolution pipeline raises a RuntimeError pointing users to the fp4_mx_block16 recipe, which works on a wider set of hardware.

Source

Thrown at python/sglang/srt/server_args.py:6658

                "block-scaled operands used by the FA4 MXFP8 attention path."
            )

    def _handle_kv4_compatibility(self):
        """Check FP4 KV cache compatibility with the attention backend"""
        cfg = resolving_view(self)

        if cfg.kv_cache_dtype not in ("nvfp4", "fp4_mx_block16"):
            return

        use_mla_backend = self.use_mla_backend()
        prefill_backend, decode_backend = self._resolved_attention_backends()
        attention_backend = resolved_view(self).attention_backend

        if is_cuda():
            if cfg.kv_cache_dtype == "nvfp4" and not (
                is_sm100_supported() or is_sm120_supported()
            ):
                raise RuntimeError(
                    "--kv-cache-dtype=nvfp4 requires Blackwell SM100 or SM120. "
                    "Use --kv-cache-dtype=fp4_mx_block16 for the block-size-16 FP4 recipe."
                )
            if (
                prefill_backend != decode_backend and prefill_backend != "fa4"
            ):  # Take care of prefill=fa4 later
                logger.warning(
                    f"Attention: Using KV4 with PREFILL = {prefill_backend} "
                    f"and DECODE = {decode_backend}. "
                    f"Compatibility issues are unlikely, but may occur in rare edge cases."
                )
            else:
                if prefill_backend == "fa4":
                    if use_mla_backend:  # FA4 + MLA
                        KV4_FA4_MLA_BACKEND_CHOICES = [
                            "cutlass_mla",
                            "flashinfer",
                            "trtllm_mla",

View on GitHub (pinned to 0132848349)

Solutions

Use --kv-cache-dtype fp4_mx_block16 (block-size-16 FP4 recipe) instead
Run on SM100/SM120 hardware if nvfp4 is required
Fall back to fp8 or bf16 KV cache on non-Blackwell GPUs

Example fix

# before
python -m sglang.launch_server --model M --kv-cache-dtype nvfp4   # on H100
# after
python -m sglang.launch_server --model M --kv-cache-dtype fp4_mx_block16

Defensive patterns

Strategy: validation

Validate before calling

import torch
cap = torch.cuda.get_device_capability(0)
sm = cap[0] * 10 + cap[1]
if args.kv_cache_dtype == "nvfp4" and sm not in (100, 120):
    args.kv_cache_dtype = "fp4_mx_block16"  # portable block-16 FP4 recipe

Try / catch

try:
    ServerArgs(**kwargs)
except RuntimeError as e:
    if "nvfp4" in str(e):
        kwargs["kv_cache_dtype"] = "fp4_mx_block16"
        ServerArgs(**kwargs)
    else:
        raise

Prevention

Know the two FP4 recipes: nvfp4 (SM100/120 only) vs fp4_mx_block16 (portable)
Select kv-cache dtype from a hardware capability lookup, not a hardcoded string
Document per-cluster supported KV dtypes to avoid config drift

When it happens

Trigger: On a CUDA system where is_sm100_supported() and is_sm120_supported() are both false, passing --kv-cache-dtype nvfp4.

Common situations: Migrating FP4 KV-cache experimentation from a Blackwell box to Hopper; confusing the two FP4 recipes (nvfp4 vs fp4_mx_block16) when copying configs between clusters.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/598c23b2449d9366. Report an issue: GitHub.