sgl-project/sglang · error · ValueError

flashinfer_cutedsl FP4 MoE only supports DeepEP low_latency

Error message

flashinfer_cutedsl FP4 MoE only supports DeepEP low_latency dispatch (masked layout). DeepEP normal (prefill) dispatch has no CuteDSL FP4 handler. Pass --deepep-mode low_latency or auto.

What it means

Raised when --moe-runner-backend flashinfer_cutedsl FP4 MoE is combined with --deepep-mode normal. The CuteDSL FP4 kernels only implement the masked (low_latency) layout; DeepEP normal prefill dispatch would crash because no FP4 handler exists for it. The resolver auto-forces low_latency in 'auto' mode, so only an explicit 'normal' reaches this error.

Source

Thrown at python/sglang/srt/server_args.py:7677

        if cfg.enable_waterfill:
            self._declare("_handle_a2a_moe", enforce_shared_experts_fusion=True)
            logger.info(f"Waterfill is enabled with moe_a2a_backend='{a2a_backend}'.")

        if a2a_backend == "deepep":
            if cfg.moe_runner_backend == "flashinfer_cutedsl":
                if cfg.deepep_mode == "auto":
                    self._declare(
                        "_handle_a2a_moe",
                        deepep_mode="low_latency",
                    )
                    logger.warning(
                        "Forcing --deepep-mode low_latency: flashinfer_cutedsl "
                        "FP4 MoE has no DeepEP normal-dispatch handler, so "
                        "deepep auto mode would crash during prefill. "
                        "low_latency covers both prefill and decode."
                    )
                elif cfg.deepep_mode == "normal":
                    raise ValueError(
                        "flashinfer_cutedsl FP4 MoE only supports DeepEP "
                        "low_latency dispatch (masked layout). DeepEP normal "
                        "(prefill) dispatch has no CuteDSL FP4 handler. Pass "
                        "--deepep-mode low_latency or auto."
                    )
            if cfg.deepep_mode == "normal":
                logger.warning("Cuda graph is disabled because deepep_mode=`normal`")
                self._declare(
                    "_handle_a2a_moe",
                    cuda_graph_config=with_phase(
                        cfg.cuda_graph_config, Phase.DECODE, backend=Backend.DISABLED
                    ),
                )
                self._declare(
                    "_handle_a2a_moe",
                    cuda_graph_config=with_phase(
                        cfg.cuda_graph_config, Phase.PREFILL, backend=Backend.DISABLED
                    ),

View on GitHub (pinned to 0132848349)

Solutions

1. Change --deepep-mode to low_latency (covers both prefill and decode)
2. Use --deepep-mode auto, which the resolver coerces to low_latency for this combo
3. Switch --moe-runner-backend away from flashinfer_cutedsl if normal-mode dispatch is required

Example fix

# before
--moe-runner-backend flashinfer_cutedsl --deepep-mode normal
# after
--moe-runner-backend flashinfer_cutedsl --deepep-mode low_latency

if moe_runner_backend == "flashinfer_cutedsl" and deepep_mode == "normal":
    raise SystemExit("cutedsl FP4 requires low_latency dispatch; fix flags")
# or simply coerce:
deep_mode = "low_latency" if moe_runner_backend == "flashinfer_cutedsl" else deepep_mode

When it happens

Trigger: Launching with --moe-runner-backend flashinfer_cutedsl (FP4 quantized MoE) plus --deepep-mode normal.

Common situations: Copying a deepep normal-mode prefill config from an FP8/BF16 deployment onto an NVFP4 model; explicitly pinning --deepep-mode normal for prefill throughput on a cutedsl FP4 setup.

Related errors

• Unsupported cute dtype {input.dtype}
• CuteDSL masked MoE supports activation 'silu' (gated) or 're
• moe_a2a_backend='pplx' only supports low-latency mode; set -
• Can't import trtllm_fp8_block_scale_moe from flashinfer. Ple
• Can't import trtllm_fp8_block_scale_routed_moe from flashinf

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/70e072f5fc32f001. Report an issue: GitHub.