{"record":{"id":"51240749c376f000","repo":"sgl-project/sglang","slug":"quantization-nvfp4-online-supports-only-moe-ru","errorCode":null,"errorMessage":"--quantization nvfp4_online supports only --moe-runner-backend flashinfer_trtllm or flashinfer_trtllm_routed, or flashinfer_cutedsl.","messagePattern":"--quantization nvfp4_online supports only --moe-runner-backend flashinfer_trtllm or flashinfer_trtllm_routed, or flashinfer_cutedsl\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/arg_groups/overrides.py","lineNumber":2750,"sourceCode":"    \"\"\"The quantization-driven moe_runner_backend resolutions at the head of\n    _handle_moe_kernel_config. The backend-compatibility asserts and the\n    disable_shared_experts_fusion writes (post-publish writers exist for that\n    field) stay in the handler.\"\"\"\n    moe_runner_backend = view.moe_runner_backend\n    if view.quantization == \"nvfp4_online\":\n        if not is_sm100_supported():\n            raise ValueError(\n                \"--quantization nvfp4_online is supported only on \"\n                \"NVIDIA Blackwell SM100/SM103 GPUs.\"\n            )\n        if moe_runner_backend == \"auto\":\n            moe_runner_backend = \"flashinfer_trtllm\"\n        elif moe_runner_backend not in [\n            \"flashinfer_trtllm\",\n            \"flashinfer_trtllm_routed\",\n            \"flashinfer_cutedsl\",\n        ]:\n            raise ValueError(\n                \"--quantization nvfp4_online supports only \"\n                \"--moe-runner-backend flashinfer_trtllm or \"\n                \"flashinfer_trtllm_routed, or flashinfer_cutedsl.\"\n            )\n    # Ascend runs MXFP8 MoE on the Ascend runner; every backend selected below is\n    # CUDA/ROCm-only. Forcing one here would not merely pick the wrong runner:\n    # FusedMoE keys its w1/w3 shard swap (\"flashinfer assumes w31\") and its\n    # 128-alignment round-up off flashinfer_trtllm, so the experts would silently\n    # load with gate and up exchanged. Leave the backend at \"auto\" and let\n    # create_moe_runner resolve it to ASCEND.\n    if view.quantization == \"mxfp8\" and not is_npu():\n        from sglang.srt.server_args import MXFP8_MOE_RUNNER_BACKEND_CHOICES\n\n        is_gfx95_mxfp8 = is_hip() and is_gfx95_supported()\n        allowed = list(MXFP8_MOE_RUNNER_BACKEND_CHOICES)\n        if is_gfx95_mxfp8:\n            allowed.append(\"triton\")\n        mxfp8_default = \"triton\" if is_gfx95_mxfp8 else \"flashinfer_trtllm\"","sourceCodeStart":2732,"sourceCodeEnd":2768,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/arg_groups/overrides.py#L2732-L2768","documentation":"When nvfp4_online quantization is active, the MoE runner must be one of the FlashInfer TRTLLM FP4 kernels: flashinfer_trtllm, flashinfer_trtllm_routed, or flashinfer_cutedsl. Other MoE runners (triton, apex, etc.) have no NVFP4 online-quantized kernels.","triggerScenarios":"--quantization nvfp4_online combined with --moe-runner-backend set to a non-FlashInfer runner (e.g. triton or auto-resolved-to-anything-else after explicit user override).","commonSituations":"A global script pinning --moe-runner-backend triton for stability being reused when trying nvfp4_online quantization.","solutions":["Drop --moe-runner-backend to let auto resolve to flashinfer_trtllm","Or set --moe-runner-backend flashinfer_trtllm (or flashinfer_trtllm_routed / flashinfer_cutedsl) explicitly","Switch off nvfp4_online if you need a different MoE runner"],"exampleFix":"# before\n--quantization nvfp4_online --moe-runner-backend triton\n# after\n--quantization nvfp4_online --moe-runner-backend flashinfer_trtllm","handlingStrategy":"validation","validationCode":"ok = {\"flashinfer_trtllm\",\"flashinfer_trtllm_routed\",\"flashinfer_cutedsl\",\"auto\"}\nif args.quantization == \"nvfp4_online\":\n    assert args.moe_runner_backend in ok, \"nvfp4_online requires flashinfer_trtllm family\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Let moe_runner_backend default to auto when quantization is exotic"],"tags":["sglang","nvfp4","moe-runner-backend","quantization","config-conflict"],"backgroundTag":"unsupported-argument-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}