sgl-project/sglang · error · ValueError

{self.quantization} quantization is currently not supported

Error message

{self.quantization} quantization is currently not supported in ROCm.

What it means

The requested quantization method is supported on CUDA but not in the rocm_supported_quantization list, and is_hip() is true (AMD GPU). _verify_quantization blocks the launch rather than failing later in a kernel.

Source

Thrown at python/sglang/srt/configs/model_config.py:1652

                    logger.info_once(
                        f"Requantizing from quant_method='{quant_method}' to the requested online quantization='{self.quantization}'. Beware that requantization may incur a loss in accuracy, the requantized model should be re-validated/re-evaluated. More details at https://docs.sglang.io/advanced_features/quantization.html#online-quantization."
                    )
                else:
                    raise ValueError(
                        "Quantization method specified in the model config "
                        f"({quant_method}) does not match the quantization "
                        f"method specified in the `quantization` argument "
                        f"({self.quantization})."
                    )

        if self.quantization is not None:
            if self.quantization not in supported_quantization:
                raise ValueError(
                    f"Unknown quantization method: {self.quantization}. Must "
                    f"be one of {supported_quantization}."
                )
            if is_hip() and self.quantization not in rocm_supported_quantization:
                raise ValueError(
                    f"{self.quantization} quantization is currently not "
                    f"supported in ROCm."
                )
            if self.quantization not in optimized_quantization_methods:
                # Don't warn for MXFP4/MXFP8 on SM100 since they have optimized kernels
                if not (
                    self.quantization in ["mxfp4", "mxfp8"] and is_sm100_supported()
                ):
                    logger.warning(
                        "%s quantization is not fully "
                        "optimized yet. The speed can be slower than "
                        "non-quantized models.",
                        self.quantization,
                    )

    def _verify_dual_chunk_attention_config(self) -> None:
        if hasattr(self.hf_config, "dual_chunk_attention_config"):
            # Try loading the sparse attention config

View on GitHub (pinned to 0132848349)

Solutions

1. Switch to a ROCm-supported quantization method listed in rocm_supported_quantization (or drop --quantization to use the native one)
2. Run on NVIDIA hardware if that specific quantization is required
3. Check sglang ROCm docs for the current per-method support matrix

Example fix

# before (on ROCm)
python -m sglang.launch_server --model MODEL --quantization awq_marlin
# after
python -m sglang.launch_server --model MODEL --quantization awq

import torch
if torch.version.hip:
    assert requested_quant in ROCM_SUPPORTED, f"{requested_quant} not available on ROCm"

When it happens

Trigger: Running on ROCm/AMD hardware with --quantization set to a CUDA-only method (e.g. many marlin/fp4 kernels).

Common situations: Running containerized sglang on MI2xx/MI3xx GPUs with configs borrowed from NVIDIA recipes, or CI on AMD runners using default quantization args.

Related errors

• Online MXFP4 quantization for MoE layers requires an AMD ROC
• Only CUDA and MUSA support GGUF quantization currently.
• --enable-hisparse is not supported with the unified-KV path
• --quantization nvfp4_online is supported only on NVIDIA Blac
• --quantization nvfp4_online supports only --moe-runner-backe

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/30ee6b3782bbbea0. Report an issue: GitHub.