Only CUDA, MUSA and NPU support GGUF quantization currently.

Error message

Only CUDA, MUSA and NPU support GGUF quantization currently.

What it means

Import-time warning from the GGUF quantization module: GGUF support requires CUDA, MUSA, or NPU (with a special gguf dequant path on NPU). On other platforms — except HIP, which is deliberately exempted — the module warns that GGUF is unsupported.

Source

Thrown at python/sglang/srt/layers/quantization/gguf.py:69

        ggml_mul_mat_vec_a8,
    )

    from sglang.kernels.ops.activation.activation import gelu_and_mul, silu_and_mul
elif _is_musa:
    from sgl_kernel import gelu_and_mul, moe_align_block_size, moe_sum, silu_and_mul
    from sgl_kernel.quantization import (
        ggml_dequantize,
        ggml_moe_a8,
        ggml_moe_a8_vec,
        ggml_moe_get_block_size,
        ggml_mul_mat_a8,
        ggml_mul_mat_vec_a8,
    )
elif _is_npu:
    from gguf import dequantize as gguf_dequantize
else:
    if not _is_hip:
        warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")

logger = logging.getLogger(__name__)


def _ordered_gguf_shard_ids(shard_ids: list) -> list:
    """Return checkpoint shards in the fused layer's logical output order."""
    if len(shard_ids) == 3 and set(shard_ids) == {"q", "k", "v"}:
        return ["q", "k", "v"]
    if all(isinstance(shard_id, int) for shard_id in shard_ids) and set(
        shard_ids
    ) == set(range(len(shard_ids))):
        return sorted(shard_ids)
    return list(shard_ids)


class GGUFConfig(QuantizationConfig):
    """Config class for GGUF."""

View on GitHub (pinned to 0132848349)

Solutions

Run on CUDA/MUSA/NPU to use GGUF checkpoints
Convert the GGUF model to a supported format (e.g. safetensors FP16/GPTQ) for other platforms
Skip loading GGUF models in platform-conditional code

Example fix

# before
--model-path model-Q4_K_M.gguf  # on CPU/XPU
# after
--model-path model-fp16-safetensors  # converted checkpoint

Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.utils import is_cuda, is_hip, is_npu
if not (is_cuda() or is_hip() or is_npu()):
    raise RuntimeError("GGUF requires CUDA/MUSA/NPU")

Prevention

Detect platform before selecting GGUF
Convert GGUF to safetensors for portable deployments

When it happens

Trigger: Importing sglang.srt.layers.quantization.gguf on a platform that is not CUDA, MUSA, NPU, or HIP (e.g. CPU-only or XPU machines).

Common situations: Trying to serve .gguf llama.cpp-style checkpoints on unsupported hardware; CI imports on CPU runners.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/92fe0eeed0058fdf. Report an issue: GitHub.