sgl-project/sglang · warning

Only CUDA, HIP and XPU support AWQ currently.

Error message

Only CUDA, HIP and XPU support AWQ currently.

What it means

Plain UserWarning emitted at import of sglang.srt.layers.quantization.awq.awq: AWQ quantization kernels are only implemented for CUDA, HIP, XPU (and NPU). On any other platform (e.g. CPU or unrecognized accelerators) the module warns that AWQ is unsupported.

Source

Thrown at python/sglang/srt/layers/quantization/awq/awq.py:53

    AWQMoEScheme,
    AWQXPULinearScheme,
)

if TYPE_CHECKING:
    from sglang.srt.layers.moe.token_dispatcher import (
        CombineInput,
        StandardDispatchOutput,
    )

from sglang.srt.utils import is_cuda, is_hip, is_npu, is_xpu

_is_cuda = is_cuda()
_is_hip = is_hip()
_is_xpu = is_xpu()
_is_npu = is_npu()

if not (_is_cuda or _is_hip or _is_xpu or _is_npu):
    warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

logger = logging.getLogger(__name__)


ScalarType, scalar_types = get_scalar_types()


def is_layer_skipped_awq(prefix: str, modules_to_not_convert: List[str]):
    return any(module_name in prefix for module_name in modules_to_not_convert)


class AWQConfig(QuantizationConfig):
    """Config class for AWQ.

    Reference: https://arxiv.org/abs/2306.00978
    """

    def __init__(

View on GitHub (pinned to 0132848349)

Solutions

1. Run on CUDA/HIP/XPU/NPU hardware for AWQ models
2. Use a non-AWQ checkpoint (e.g. unquantized or a supported quant format) on unsupported platforms
3. Suppress only if you knowingly import the module without executing AWQ paths

Example fix

# before
model = AutoModelForCausalLM.from_pretrained("x/TheBloke-7B-AWQ")  # on CPU
# after
model = AutoModelForCausalLM.from_pretrained("x/TheBloke-7B-GPTQ")  # or run on CUDA

from sglang.utils import is_cuda, is_hip, is_xpu  # or sglang.srt.utils
if not (is_cuda() or is_hip() or is_xpu()):
    raise RuntimeError("AWQ requires CUDA/HIP/XPU; pick another quantization")

When it happens

Trigger: Importing the AWQ quantization module on a machine where is_cuda(), is_hip(), is_xpu(), and is_npu() are all false — e.g. CPU-only dev boxes or unsupported accelerators.

Common situations: Installing sglang on CPU-only hardware; running unit tests or imports on a laptop/CI runner without GPUs; later attempting to load an AWQ checkpoint which will then fail at kernel dispatch.

Related errors

• The quantization method moe_wna16 + awq is not supported for
• HIP does not support fused_marlin_moe currently.
• Only CUDA, MUSA and NPU support GGUF quantization currently.
• Only CUDA and MUSA support GGUF quantization currently.
• Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs (Amp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bacf4cf71dc42e0a. Report an issue: GitHub.