{"record":{"id":"fe865727f5652487","repo":"sgl-project/sglang","slug":"only-cuda-and-musa-support-gguf-quantization-curre","errorCode":null,"errorMessage":"Only CUDA and MUSA support GGUF quantization currently.","messagePattern":"Only CUDA and MUSA support GGUF quantization currently\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/srt/layers/quantization/gguf.py","lineNumber":91,"sourceCode":"\ndef _ordered_gguf_shard_ids(shard_ids: list) -> list:\n    \"\"\"Return checkpoint shards in the fused layer's logical output order.\"\"\"\n    if len(shard_ids) == 3 and set(shard_ids) == {\"q\", \"k\", \"v\"}:\n        return [\"q\", \"k\", \"v\"]\n    if all(isinstance(shard_id, int) for shard_id in shard_ids) and set(\n        shard_ids\n    ) == set(range(len(shard_ids))):\n        return sorted(shard_ids)\n    return list(shard_ids)\n\n\nclass GGUFConfig(QuantizationConfig):\n    \"\"\"Config class for GGUF.\"\"\"\n\n    def __init__(self, modules_to_not_convert: list[str] | None = None) -> None:\n        super().__init__()\n        if _is_hip:\n            warnings.warn(f\"Only CUDA and MUSA support GGUF quantization currently.\")\n        self.modules_to_not_convert = modules_to_not_convert or []\n\n    def __repr__(self) -> str:\n        return \"GGUFConfig()\"\n\n    def get_scaled_act_names(self) -> List[str]:\n        return []\n\n    def get_name(self) -> str:\n        return \"gguf\"\n\n    def get_supported_act_dtypes(self) -> list[torch.dtype]:\n        return [torch.half, torch.bfloat16, torch.float32]\n\n    @classmethod\n    def get_min_capability(cls) -> int:\n        return 60 if not _is_musa else 21\n","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gguf.py#L73-L109","documentation":"Warning from GGUFConfig.__init__: when the GGUF quantization config is constructed on HIP (ROCm), SGLang warns that only CUDA and MUSA support GGUF currently. Unlike the import-time warning (which exempts HIP), constructing GGUFConfig on ROCm explicitly warns, indicating the runtime path is not expected to work.","triggerScenarios":"Loading a model whose quantization_config in config.json is \"gguf\" on a ROCm system, causing GGUFConfig() instantiation during quantization method resolution.","commonSituations":"Serving GGUF checkpoints on AMD GPUs; reusing a CUDA deployment recipe on ROCm.","solutions":["Use CUDA or MUSA hardware for GGUF models","Convert the GGUF checkpoint to safetensors with a ROCm-supported quantization","Catch/acknowledge the warning and expect degraded or failing behavior on ROCm"],"exampleFix":"# before\npython -m sglang.launch_server --model model.Q8_0.gguf  # on ROCm\n# after\npython -m sglang.launch_server --model model-fp16  # converted for ROCm","handlingStrategy":"validation","validationCode":"from sglang.srt.utils import is_hip\nif is_hip():\n    assert not is_gguf_model(config), \"GGUF not supported on ROCm\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Gate GGUF serving on platform checks","Watch SGLang ROCm support matrices per release"],"tags":["sglang","gguf","rocm","quantization","platform-support"],"backgroundTag":"unsupported-platform","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}