sgl-project/sglang · warning

HIP does not support fused_marlin_moe currently.

Error message

HIP does not support fused_marlin_moe currently.

What it means

Warning from AWQMarlinConfig.__init__: on HIP (ROCm) GPUs the fused_marlin_moe kernel is not supported. The config still initializes, but MoE layers will not use the fused Marlin path and may fall back to slower execution or fail later.

Source

Thrown at python/sglang/srt/layers/quantization/awq/awq.py:265

    # num_bits -> type
    TYPE_MAP = {
        4: scalar_types.uint4,
        8: scalar_types.uint8,
    }

    def __init__(
        self,
        weight_bits: int,
        group_size: int,
        zero_point: bool,
        lm_head_quantized: bool,
        modules_to_not_convert: Optional[list[str]],
        full_config: dict[str, Any],
    ) -> None:
        super().__init__()
        if _is_hip:
            warnings.warn(f"HIP does not support fused_marlin_moe currently.")
        self.pack_factor = 32 // weight_bits  # packed into int32
        self.group_size = group_size
        self.zero_point = zero_point
        self.lm_head_quantized = lm_head_quantized
        self.weight_bits = weight_bits
        self.modules_to_not_convert = modules_to_not_convert or []
        self.full_config = full_config

        if self.weight_bits not in self.TYPE_MAP:
            raise ValueError(
                f"Unsupported num_bits = {self.weight_bits}. "
                f"Supported num_bits = {self.TYPE_MAP.keys()}"
            )

        self.quant_type = self.TYPE_MAP[self.weight_bits]

        verify_marlin_supported(
            self.quant_type, group_size=self.group_size, has_zp=self.zero_point

View on GitHub (pinned to 0132848349)

Solutions

1. Use a non-Marlin AWQ variant or different quantization format on ROCm
2. Check sglang release notes for fused_marlin_moe ROCm support and upgrade
3. Alternatively force a different MoE quant method via quantization config override

Example fix

# before
--quantization awq_marlin  # on ROCm with MoE model
# after
--quantization awq  # or a ROCm-supported format

from sglang.srt.utils import is_hip
if is_hip():
    # avoid fused_marlin_moe AWQ checkpoints
    choose_non_marlin_checkpoint()

When it happens

Trigger: Loading an AWQ-Marlin quantized MoE model (e.g. Mixtral/Meta-Llama MoE AWQ checkpoints) on an AMD ROCm (HIP) system; _is_hip is true when awq.py is imported on ROCm.

Common situations: Running AWQ Marlin MoE models on MI200/MI300 GPUs; porting a CUDA deployment to ROCm without checking kernel coverage.

Related errors

• Decode context parallel (decode_context_parallel_size > 1) i
• num_bits must be 4 or 8, got {}
• experimental_sgl_marlin LoRA requires --lora-use-virtual-exp
• experimental_sgl_marlin EP requires --moe-a2a-backend none
• experimental_sgl_marlin EP requires trivial expert placement

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d83eeb9e9d7f9fe4. Report an issue: GitHub.