sgl-project/sglang · error · ValueError

The quantization method moe_wna16 + awq is not supported for

Error message

The quantization method moe_wna16 + awq is not supported for the current GPU. Minimum capability: {awq_min_capability}. Current capability: {device_capability}.

What it means

The moe_wna16 + AWQ path uses the AWQ marlin kernel, which requires a minimum GPU compute capability (typically SM 75 / Turing and above). At init the server compares the current device's capability against AWQConfig.get_min_capability() and refuses to run when the GPU is too old.

Source

Thrown at python/sglang/srt/layers/quantization/moe_wna16.py:103

        self.bit8_pack_factor = 8 // self.weight_bits
        self.lm_head_quantized = lm_head_quantized
        self.linear_quant_method = linear_quant_method
        self.full_config = full_config
        self.use_marlin = False
        # Avoid circular import

        if self.linear_quant_method == "gptq":
            self.use_marlin = GPTQMarlinConfig.is_gptq_marlin_compatible(full_config)
        elif self.linear_quant_method == "awq":
            capability_tuple = get_device_capability()
            device_capability = (
                -1
                if capability_tuple is None
                else capability_tuple[0] * 10 + capability_tuple[1]
            )
            awq_min_capability = AWQConfig.get_min_capability()
            if device_capability < awq_min_capability:
                raise ValueError(
                    "The quantization method moe_wna16 + awq is not supported "
                    "for the current GPU. "
                    f"Minimum capability: {awq_min_capability}. "
                    f"Current capability: {device_capability}."
                )
        else:
            raise ValueError("moe_wna16 only support gptq and awq.")

        if modules_to_not_convert is None:
            self.modules_to_not_convert = []
        else:
            self.modules_to_not_convert = modules_to_not_convert

    @classmethod
    def get_name(cls) -> str:
        return "moe_wna16"

    @classmethod

View on GitHub (pinned to 0132848349)

Solutions

1. Run on a Turing or newer GPU (T4, A100, L4, H100, RTX 20xx+ or newer)
2. If the capability is being reported as -1 because the device couldn't be queried (non-CUDA visible device, MIG misconfiguration), fix GPU visibility (CUDA_VISIBLE_DEVICES) so torch.cuda.get_device_capability succeeds
3. Use a non-AWQ quantization format supported on your GPU (e.g. GPTQ marlin, or unquantized weights)

Example fix

# before: AWQ MoE on GTX 1080
python -m sglang.launch_server --model some-model-awq-moe
# after
CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server --model some-model-awq-moe  # on a T4/A100+ GPU

import torch
cap = torch.cuda.get_device_capability()
major, minor = cap if cap else (-1, -1)
device_cap = major * 10 + minor
if device_cap < 75:  # AWQ marlin minimum
    raise RuntimeError(f"GPU SM {device_cap} too old for AWQ moe_wna16")

try:
    server_args = ServerArgs(model=path, ...)
except ValueError as e:
    if "capability" in str(e):
        # fall back to a GPTQ or unquantized checkpoint on this GPU
        ...
    raise

When it happens

Trigger: Loading an AWQ-quantized MoE model (moe_wna16 with quant_method 'awq') on a GPU with compute capability below the AWQ minimum — e.g. a Pascal (SM 60, e.g. GTX 1080, P100) or Volta (SM 70) card.

Common situations: Running on older datacenter GPUs (P100/V100), some cloud instances (AWS P2/P3, older K80), or a GPU whose capability cannot be detected so it defaults to -1; the latter makes any card fail the comparison.

Related errors

• num_bits must be 4 or 8, got {}
• moe_wna16 only support gptq and awq.
• Only CUDA, HIP and XPU support AWQ currently.
• Kimi expert-pack {role} quant type is unsupported
• The hpc_ops MoE runner backend only supports FP8-quantized M

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a92299306eaf4f2c. Report an issue: GitHub.