{"record":{"id":"b3677cc18d8a14e3","repo":"sgl-project/sglang","slug":"nunchaku-svdquant-is-only-supported-on-nvidia-cuda","errorCode":null,"errorMessage":"Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs (Ampere SM8x or SM12x).","messagePattern":"Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs \\(Ampere SM8x or SM12x\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/configs/quantization/nunchaku.py","lineNumber":107,"sourceCode":"                    f\"from --transformer-weights-path: {self.transformer_weights_path}\"\n                )\n\n        if self.quantization_rank is None and inferred_rank:\n            if inferred_rank:\n                logger.info(\n                    f\"inferred --quantization-rank: {normalized.quantization_rank} \"\n                    f\"from --transformer-weights-path: {self.transformer_weights_path}\"\n                )\n\n        return normalized\n\n    def _validate(self) -> None:\n        # TODO: warn if the served model doesn't support nunchaku\n        if not self.enable_svdquant:\n            return\n\n        if not current_platform.is_cuda():\n            raise ValueError(\n                \"Nunchaku SVDQuant is only supported on NVIDIA CUDA GPUs \"\n                \"(Ampere SM8x or SM12x).\"\n            )\n\n        device_count = torch.cuda.device_count()\n\n        unsupported: list[str] = []\n        for i in range(device_count):\n            major, minor = torch.cuda.get_device_capability(i)\n            if major == 9:\n                unsupported.append(f\"cuda:{i} (SM{major}{minor}, Hopper)\")\n            elif major not in (8, 12):\n                unsupported.append(f\"cuda:{i} (SM{major}{minor})\")\n\n        if unsupported:\n            raise ValueError(\n                \"Nunchaku SVDQuant is currently only supported on Ampere (SM8x) or SM12x GPUs; \"\n                f\"Unsupported devices: {', '.join(unsupported)}. \"","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/configs/quantization/nunchaku.py#L89-L125","documentation":"Nunchaku SVDQuant quantized inference requires an NVIDIA CUDA platform; the config validator checks current_platform.is_cuda() when enable_svdquant is on and raises if the runtime is not CUDA (CPU, ROCm/HIP, etc.).","triggerScenarios":"Launching with --enable-svdquant (enable_svdquant=True) on a non-CUDA platform: CPU-only machine, AMD GPU via ROCm/HIP, or a MPS/macOS build; resolve_runtime_config -> _validate raises during startup.","commonSituations":"Developing/testing configs on laptops or CPU-only containers that will later deploy to NVIDIA; running on AMD clusters where the flag was copied from an NVIDIA config; CI runners without GPUs.","solutions":["Disable the flag: pass --enable-svdquant false (or omit it) on non-CUDA machines","Run on an NVIDIA CUDA machine if SVDQuant acceleration is required","Gate the config by hardware: only enable svdquant when torch.cuda.is_available() and the platform is CUDA"],"exampleFix":"# before\nquant_cfg.enable_svdquant = True  # on CPU/ROCm box\n\n# after\nimport torch\nquant_cfg.enable_svdquant = torch.cuda.is_available() and torch.version.cuda is not None","handlingStrategy":"type-guard","validationCode":"import torch\nfrom sglang.srt.utils import current_platform\ncan_svdquant = current_platform.is_cuda()","typeGuard":"def svdquant_supported_here() -> bool:\n    import torch\n    try:\n        from sglang.srt.utils import current_platform\n        return current_platform.is_cuda()\n    except Exception:\n        return False","tryCatchPattern":null,"preventionTips":["Gate --enable-svdquant on platform detection in launch scripts","Keep CPU/ROCm dev configs separate from production NVIDIA configs"],"tags":["nunchaku","svdquant","cuda","platform-support","quantization"],"backgroundTag":"unsupported-hardware-platform","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}