sgl-project/sglang · error · ValueError

Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs (Amp

Error message

Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs (Ampere SM8x or SM12x).

What it means

Nunchaku SVDQuant quantized inference requires an NVIDIA CUDA platform; the config validator checks current_platform.is_cuda() when enable_svdquant is on and raises if the runtime is not CUDA (CPU, ROCm/HIP, etc.).

Source

Thrown at python/sglang/multimodal_gen/configs/quantization/nunchaku.py:107

                    f"from --transformer-weights-path: {self.transformer_weights_path}"
                )

        if self.quantization_rank is None and inferred_rank:
            if inferred_rank:
                logger.info(
                    f"inferred --quantization-rank: {normalized.quantization_rank} "
                    f"from --transformer-weights-path: {self.transformer_weights_path}"
                )

        return normalized

    def _validate(self) -> None:
        # TODO: warn if the served model doesn't support nunchaku
        if not self.enable_svdquant:
            return

        if not current_platform.is_cuda():
            raise ValueError(
                "Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs "
                "(Ampere SM8x or SM12x)."
            )

        device_count = torch.cuda.device_count()

        unsupported: list[str] = []
        for i in range(device_count):
            major, minor = torch.cuda.get_device_capability(i)
            if major == 9:
                unsupported.append(f"cuda:{i} (SM{major}{minor}, Hopper)")
            elif major not in (8, 12):
                unsupported.append(f"cuda:{i} (SM{major}{minor})")

        if unsupported:
            raise ValueError(
                "Nunchaku SVDQuant is currently only supported on Ampere (SM8x) or SM12x GPUs; "
                f"Unsupported devices: {', '.join(unsupported)}. "

View on GitHub (pinned to 0132848349)

Solutions

1. Disable the flag: pass --enable-svdquant false (or omit it) on non-CUDA machines
2. Run on an NVIDIA CUDA machine if SVDQuant acceleration is required
3. Gate the config by hardware: only enable svdquant when torch.cuda.is_available() and the platform is CUDA

Example fix

# before
quant_cfg.enable_svdquant = True  # on CPU/ROCm box

# after
import torch
quant_cfg.enable_svdquant = torch.cuda.is_available() and torch.version.cuda is not None

import torch
from sglang.srt.utils import current_platform
can_svdquant = current_platform.is_cuda()

def svdquant_supported_here() -> bool:
    import torch
    try:
        from sglang.srt.utils import current_platform
        return current_platform.is_cuda()
    except Exception:
        return False

When it happens

Trigger: Launching with --enable-svdquant (enable_svdquant=True) on a non-CUDA platform: CPU-only machine, AMD GPU via ROCm/HIP, or a MPS/macOS build; resolve_runtime_config -> _validate raises during startup.

Common situations: Developing/testing configs on laptops or CPU-only containers that will later deploy to NVIDIA; running on AMD clusters where the flag was copied from an NVIDIA config; CI runners without GPUs.

Related errors

• Nunchaku SVDQuant is currently only supported on Ampere (SM8
• --enable-svdquant requires --transformer-weights-path to be
• Serialized W4A4 checkpoints are not supported on MPS
• Serialized W4A4 checkpoints require CUDA compute capability
• Serialized W4A8 checkpoints are not supported on MPS

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b3677cc18d8a14e3. Report an issue: GitHub.