sgl-project/sglang · error · RuntimeError

Ascend A2/A3 NPU does not support nvfp4 deepep_dispatcher_ou

Error message

Ascend A2/A3 NPU does not support nvfp4 deepep_dispatcher_output_dtype.

What it means

When quantizing dispatch output, DeepEP-style dispatchers validate the requested deepep_dispatcher_output_dtype against the hardware: on Ascend A2/A3 NPUs fp8 silently downgrades to int8 with a warning, but NVFP4 is a hard RuntimeError — Ascend NPUs have no fp4 support. Raised from _validate_and_adjust_dtype via set_deepep_dispatcher_dtype.

Source

Thrown at python/sglang/srt/layers/moe/token_dispatcher/deepep.py:484

        config = config_map[self.deepep_output_dtype]
        self.use_fp8 = config["use_fp8"]
        self.use_nvfp4 = config["use_nvfp4"]

        # Handle environment variables
        if _is_npu:
            self._update_int8_quant_env()

    def _validate_and_adjust_dtype(self) -> None:
        """Validate dtype against hardware and adjust if necessary."""
        if _is_npu:
            if self.deepep_output_dtype == DispatcherOutputDtype.FP8:
                logger.warning_once(
                    "Ascend A2/A3 NPU does not support fp8 "
                    "deepep_dispatcher_output_dtype, switching to int8..."
                )
                self.deepep_output_dtype = DispatcherOutputDtype.INT8
            elif self.deepep_output_dtype == DispatcherOutputDtype.NVFP4:
                raise RuntimeError(
                    "Ascend A2/A3 NPU does not support nvfp4 deepep_dispatcher_output_dtype."
                )
        else:
            if self.deepep_output_dtype == DispatcherOutputDtype.INT8:
                logger.warning_once(
                    "GPU does not support int8 "
                    "deepep_dispatcher_output_dtype, switching to fp8..."
                )
                self.deepep_output_dtype = DispatcherOutputDtype.FP8
            # NVFP4 is supported on GPU, no adjustment needed

    def _update_int8_quant_env(self) -> None:
        """TODO adapt different quantization schemes for base model and draft model on NPU"""
        pass

    def set_overlap_args(
        self, combine_overlap_args: CombineOverlapArgs, meta_overlap_args: dict
    ) -> None:

View on GitHub (pinned to 0132848349)

Solutions

1. Use a supported dtype on Ascend: bf16, fp8 (auto-switches to int8 with warning), or int8
2. Don't run nvfp4-quantized MoE checkpoints on A2/A3 NPUs; use an fp8/int8-quantized checkpoint instead
3. Move the workload to supported NVIDIA hardware if nvfp4 is a hard requirement

Example fix

# before
--deepep-dispatcher-output-dtype nvfp4  # on Ascend -> RuntimeError

# after
--deepep-dispatcher-output-dtype int8  # or fp8/bf16 on Ascend

import torch
if getattr(torch, "npu", None) is not None and dtype == "nvfp4":
    raise SystemExit("nvfp4 dispatch dtype unsupported on Ascend A2/A3; use int8/fp8/bf16")

When it happens

Trigger: Configuring deepep_dispatcher_output_dtype to nvfp4 (e.g. --deepep-dispatcher-output-dtype nvfp4, common for NVFP4-quantized models like some Blackwell recipes) while running on an Ascend A2/A3 NPU.

Common situations: Taking a GPU (Blackwell) nvfp4 serving recipe and running it on Huawei Ascend hardware; config templates copied across platforms without checking NPU support matrix.

Related errors

• Unsupported ascend_dispatcher_output_dtype: {self.ascend_dis
• The W8A8Int8 Fused MoE scheme is implemented only for NPU fo
• The W4A8Int8 Fused MoE scheme is implemented only for NPU fo
• Type must match: {self.a_dtype} != {self.b_dtype}
• nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fb83c84be8af7a73. Report an issue: GitHub.