sgl-project/sglang · error · ValueError

Nunchaku SVDQuant is currently only supported on Ampere (SM8

Error message

Nunchaku SVDQuant is currently only supported on Ampere (SM8x) or SM12x GPUs; Unsupported devices: {', '.join(unsupported)}. Disable it with --enable-svdquant false.

What it means

Even on CUDA, Nunchaku SVDQuant kernels only support Ampere (SM8x) and SM12x GPUs. The validator enumerates all visible devices and raises with the list of unsupported ones — Hopper SM9x is explicitly unsupported, as is any other major version outside 8 and 12.

Source

Thrown at python/sglang/multimodal_gen/configs/quantization/nunchaku.py:123

        if not current_platform.is_cuda():
            raise ValueError(
                "Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs "
                "(Ampere SM8x or SM12x)."
            )

        device_count = torch.cuda.device_count()

        unsupported: list[str] = []
        for i in range(device_count):
            major, minor = torch.cuda.get_device_capability(i)
            if major == 9:
                unsupported.append(f"cuda:{i} (SM{major}{minor}, Hopper)")
            elif major not in (8, 12):
                unsupported.append(f"cuda:{i} (SM{major}{minor})")

        if unsupported:
            raise ValueError(
                "Nunchaku SVDQuant is currently only supported on Ampere (SM8x) or SM12x GPUs; "
                f"Unsupported devices: {', '.join(unsupported)}. "
                "Disable it with --enable-svdquant false."
            )

        if not self.transformer_weights_path:
            raise ValueError(
                "--enable-svdquant requires --transformer-weights-path to be set"
            )

        if not is_nunchaku_available():
            raise ValueError(
                "Nunchaku is enabled, but not installed. Please refer to https://nunchaku.tech/docs/nunchaku/installation/installation.html for detailed installation methods."
            )

        if self.quantization_precision not in ("int4", "nvfp4"):
            raise ValueError(
                f"Invalid --quantization-precision: {self.quantization_precision}. "

View on GitHub (pinned to 0132848349)

Solutions

Disable the feature: --enable-svdquant false
Restrict visibility to supported GPUs, e.g. CUDA_VISIBLE_DEVICES pointing only at A100/Ampere or SM12x devices
Move the workload to an Ampere (A100/30xx) or SM12x machine
Check torch.cuda.get_device_capability(i) per device to confirm the SM major/minor before enabling

Example fix

# before
# all GPUs visible, includes H100 (SM90)
cfg.enable_svdquant = True

# after
# run on / restrict to Ampere devices
# CUDA_VISIBLE_DEVICES=0,1  (A100s only)
cfg.enable_svdquant = True

Defensive patterns

Strategy: validation

Validate before calling

import torch
unsupported = [i for i in range(torch.cuda.device_count())
               if torch.cuda.get_device_capability(i)[0] not in (8, 12)]
assert not unsupported, f"unsupported GPUs: {unsupported}"

Type guard

def all_gpus_svdquant_capable() -> bool:
    import torch
    return torch.cuda.is_available() and all(
        torch.cuda.get_device_capability(i)[0] in (8, 12)
        for i in range(torch.cuda.device_count())
    )

Prevention

Check torch.cuda.get_device_capability before enabling svdquant
Set CUDA_VISIBLE_DEVICES to only Ampere/SM12x devices on mixed nodes
Note H100/H200 (SM9x) are explicitly unsupported by nunchaku SVDQuant

When it happens

Trigger: Starting with --enable-svdquant on a machine whose visible devices include H100/H200 (SM9x, listed as 'Hopper') or pre-Ampere (e.g. SM7x Turing) or Blackwell-non-SM12x GPUs; resolve_runtime_config -> _validate raises listing each offending cuda:i.

Common situations: H100/H200 clusters where the config was authored on A100; mixed-GPU nodes where one device is unsupported; CUDA_VISIBLE_ORDER/visibility changes exposing an unexpected device; older Turing boxes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1f1ccd99bc58252f. Report an issue: GitHub.