sgl-project/sglang · error · ValueError

--quantization nvfp4_online is supported only on NVIDIA Blac

Error message

--quantization nvfp4_online is supported only on NVIDIA Blackwell SM100/SM103 GPUs.

What it means

Online NVFP4 quantization (--quantization nvfp4_online) relies on Blackwell SM100/SM103 hardware instructions (FP4 tensor cores and block-scaled conversion), so SGLang rejects it at startup on older architectures.

Source

Thrown at python/sglang/srt/arg_groups/overrides.py:2739

            f"dp_size, got tp_size={view.tp_size}, dp_size={view.dp_size}."
        )
        assert view.attn_cp_size == 1, (
            "--enable-tp-lm-head-all-to-all currently requires "
            f"attn_cp_size == 1, got {view.attn_cp_size}."
        )
    return {}


@register_post_process
def _moe_runner_backend_quant_constraints(view: Any) -> dict:
    """The quantization-driven moe_runner_backend resolutions at the head of
    _handle_moe_kernel_config. The backend-compatibility asserts and the
    disable_shared_experts_fusion writes (post-publish writers exist for that
    field) stay in the handler."""
    moe_runner_backend = view.moe_runner_backend
    if view.quantization == "nvfp4_online":
        if not is_sm100_supported():
            raise ValueError(
                "--quantization nvfp4_online is supported only on "
                "NVIDIA Blackwell SM100/SM103 GPUs."
            )
        if moe_runner_backend == "auto":
            moe_runner_backend = "flashinfer_trtllm"
        elif moe_runner_backend not in [
            "flashinfer_trtllm",
            "flashinfer_trtllm_routed",
            "flashinfer_cutedsl",
        ]:
            raise ValueError(
                "--quantization nvfp4_online supports only "
                "--moe-runner-backend flashinfer_trtllm or "
                "flashinfer_trtllm_routed, or flashinfer_cutedsl."
            )
    # Ascend runs MXFP8 MoE on the Ascend runner; every backend selected below is
    # CUDA/ROCm-only. Forcing one here would not merely pick the wrong runner:
    # FusedMoE keys its w1/w3 shard swap ("flashinfer assumes w31") and its

View on GitHub (pinned to 0132848349)

Solutions

1. Run on an SM100/SM103 Blackwell GPU (B200/GB200/B300)
2. Use a quantization supported by your hardware (e.g. fp8 on Hopper, or a pre-quantized checkpoint with a matching scheme)

Example fix

# before (on H100)
--quantization nvfp4_online
# after
--quantization fp8_dynauto_float

import torch
if args.quantization == "nvfp4_online":
    assert torch.cuda.get_device_capability() in ((10, 0), (10, 3)), "nvfp4_online needs SM100/SM103"

When it happens

Trigger: Launching with --quantization nvfp4_online on a GPU that is not SM100/SM103 (H100, A100, AMD, SM12x consumer Blackwell).

Common situations: Trying NVFP4 online quantization on an H100 dev machine or in CI without Blackwell runners.

Related errors

• tokenspeed_mla backend is only supported on Blackwell GPUs (
• --quantization nvfp4_online supports only --moe-runner-backe
• Current platform does not support NVFP4 quantization. Please
• --kv-cache-dtype=nvfp4 requires Blackwell SM100 or SM120. Us
• Type must match: {self.a_dtype} != {self.b_dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6d8c3fd0d2685ca2. Report an issue: GitHub.