{"record":{"id":"92fe0eeed0058fdf","repo":"sgl-project/sglang","slug":"only-cuda-musa-and-npu-support-gguf-quantization","errorCode":null,"errorMessage":"Only CUDA, MUSA and NPU support GGUF quantization currently.","messagePattern":"Only CUDA, MUSA and NPU support GGUF quantization currently\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/srt/layers/quantization/gguf.py","lineNumber":69,"sourceCode":"        ggml_mul_mat_vec_a8,\n    )\n\n    from sglang.kernels.ops.activation.activation import gelu_and_mul, silu_and_mul\nelif _is_musa:\n    from sgl_kernel import gelu_and_mul, moe_align_block_size, moe_sum, silu_and_mul\n    from sgl_kernel.quantization import (\n        ggml_dequantize,\n        ggml_moe_a8,\n        ggml_moe_a8_vec,\n        ggml_moe_get_block_size,\n        ggml_mul_mat_a8,\n        ggml_mul_mat_vec_a8,\n    )\nelif _is_npu:\n    from gguf import dequantize as gguf_dequantize\nelse:\n    if not _is_hip:\n        warnings.warn(f\"Only CUDA, MUSA and NPU support GGUF quantization currently.\")\n\nlogger = logging.getLogger(__name__)\n\n\ndef _ordered_gguf_shard_ids(shard_ids: list) -> list:\n    \"\"\"Return checkpoint shards in the fused layer's logical output order.\"\"\"\n    if len(shard_ids) == 3 and set(shard_ids) == {\"q\", \"k\", \"v\"}:\n        return [\"q\", \"k\", \"v\"]\n    if all(isinstance(shard_id, int) for shard_id in shard_ids) and set(\n        shard_ids\n    ) == set(range(len(shard_ids))):\n        return sorted(shard_ids)\n    return list(shard_ids)\n\n\nclass GGUFConfig(QuantizationConfig):\n    \"\"\"Config class for GGUF.\"\"\"\n","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gguf.py#L51-L87","documentation":"Import-time warning from the GGUF quantization module: GGUF support requires CUDA, MUSA, or NPU (with a special gguf dequant path on NPU). On other platforms — except HIP, which is deliberately exempted — the module warns that GGUF is unsupported.","triggerScenarios":"Importing sglang.srt.layers.quantization.gguf on a platform that is not CUDA, MUSA, NPU, or HIP (e.g. CPU-only or XPU machines).","commonSituations":"Trying to serve .gguf llama.cpp-style checkpoints on unsupported hardware; CI imports on CPU runners.","solutions":["Run on CUDA/MUSA/NPU to use GGUF checkpoints","Convert the GGUF model to a supported format (e.g. safetensors FP16/GPTQ) for other platforms","Skip loading GGUF models in platform-conditional code"],"exampleFix":"# before\n--model-path model-Q4_K_M.gguf  # on CPU/XPU\n# after\n--model-path model-fp16-safetensors  # converted checkpoint","handlingStrategy":"validation","validationCode":"from sglang.srt.utils import is_cuda, is_hip, is_npu\nif not (is_cuda() or is_hip() or is_npu()):\n    raise RuntimeError(\"GGUF requires CUDA/MUSA/NPU\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Detect platform before selecting GGUF","Convert GGUF to safetensors for portable deployments"],"tags":["sglang","gguf","quantization","platform-support"],"backgroundTag":"unsupported-platform","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}