sgl-project/sglang · error · ValueError

--quantization nvfp4_online supports only --moe-runner-backe

Error message

--quantization nvfp4_online supports only --moe-runner-backend flashinfer_trtllm or flashinfer_trtllm_routed, or flashinfer_cutedsl.

What it means

When nvfp4_online quantization is active, the MoE runner must be one of the FlashInfer TRTLLM FP4 kernels: flashinfer_trtllm, flashinfer_trtllm_routed, or flashinfer_cutedsl. Other MoE runners (triton, apex, etc.) have no NVFP4 online-quantized kernels.

Source

Thrown at python/sglang/srt/arg_groups/overrides.py:2750

    """The quantization-driven moe_runner_backend resolutions at the head of
    _handle_moe_kernel_config. The backend-compatibility asserts and the
    disable_shared_experts_fusion writes (post-publish writers exist for that
    field) stay in the handler."""
    moe_runner_backend = view.moe_runner_backend
    if view.quantization == "nvfp4_online":
        if not is_sm100_supported():
            raise ValueError(
                "--quantization nvfp4_online is supported only on "
                "NVIDIA Blackwell SM100/SM103 GPUs."
            )
        if moe_runner_backend == "auto":
            moe_runner_backend = "flashinfer_trtllm"
        elif moe_runner_backend not in [
            "flashinfer_trtllm",
            "flashinfer_trtllm_routed",
            "flashinfer_cutedsl",
        ]:
            raise ValueError(
                "--quantization nvfp4_online supports only "
                "--moe-runner-backend flashinfer_trtllm or "
                "flashinfer_trtllm_routed, or flashinfer_cutedsl."
            )
    # Ascend runs MXFP8 MoE on the Ascend runner; every backend selected below is
    # CUDA/ROCm-only. Forcing one here would not merely pick the wrong runner:
    # FusedMoE keys its w1/w3 shard swap ("flashinfer assumes w31") and its
    # 128-alignment round-up off flashinfer_trtllm, so the experts would silently
    # load with gate and up exchanged. Leave the backend at "auto" and let
    # create_moe_runner resolve it to ASCEND.
    if view.quantization == "mxfp8" and not is_npu():
        from sglang.srt.server_args import MXFP8_MOE_RUNNER_BACKEND_CHOICES

        is_gfx95_mxfp8 = is_hip() and is_gfx95_supported()
        allowed = list(MXFP8_MOE_RUNNER_BACKEND_CHOICES)
        if is_gfx95_mxfp8:
            allowed.append("triton")
        mxfp8_default = "triton" if is_gfx95_mxfp8 else "flashinfer_trtllm"

View on GitHub (pinned to 0132848349)

Solutions

1. Drop --moe-runner-backend to let auto resolve to flashinfer_trtllm
2. Or set --moe-runner-backend flashinfer_trtllm (or flashinfer_trtllm_routed / flashinfer_cutedsl) explicitly
3. Switch off nvfp4_online if you need a different MoE runner

Example fix

# before
--quantization nvfp4_online --moe-runner-backend triton
# after
--quantization nvfp4_online --moe-runner-backend flashinfer_trtllm

ok = {"flashinfer_trtllm","flashinfer_trtllm_routed","flashinfer_cutedsl","auto"}
if args.quantization == "nvfp4_online":
    assert args.moe_runner_backend in ok, "nvfp4_online requires flashinfer_trtllm family"

When it happens

Trigger: --quantization nvfp4_online combined with --moe-runner-backend set to a non-FlashInfer runner (e.g. triton or auto-resolved-to-anything-else after explicit user override).

Common situations: A global script pinning --moe-runner-backend triton for stability being reused when trying nvfp4_online quantization.

Related errors

• --quantization nvfp4_online is supported only on NVIDIA Blac
• Config list contains configs from 2 methods, must be only 1
• Type must match: {self.a_dtype} != {self.b_dtype}
• nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/51240749c376f000. Report an issue: GitHub.