sgl-project/sglang · warning

Only CUDA and MUSA support GGUF quantization currently.

Error message

Only CUDA and MUSA support GGUF quantization currently.

What it means

Warning from GGUFConfig.__init__: when the GGUF quantization config is constructed on HIP (ROCm), SGLang warns that only CUDA and MUSA support GGUF currently. Unlike the import-time warning (which exempts HIP), constructing GGUFConfig on ROCm explicitly warns, indicating the runtime path is not expected to work.

Solutions

1. Use CUDA or MUSA hardware for GGUF models
2. Convert the GGUF checkpoint to safetensors with a ROCm-supported quantization
3. Catch/acknowledge the warning and expect degraded or failing behavior on ROCm

Example fix

# before
python -m sglang.launch_server --model model.Q8_0.gguf  # on ROCm
# after
python -m sglang.launch_server --model model-fp16  # converted for ROCm

from sglang.srt.utils import is_hip
if is_hip():
    assert not is_gguf_model(config), "GGUF not supported on ROCm"

When it happens

Trigger: Loading a model whose quantization_config in config.json is "gguf" on a ROCm system, causing GGUFConfig() instantiation during quantization method resolution.

Common situations: Serving GGUF checkpoints on AMD GPUs; reusing a CUDA deployment recipe on ROCm.

Understand the failure class

Background: "unsupported platform" / "not supported on this platform" errors: what they mean and how to fix them — this error's family across 47 libraries.

Related errors

• Only CUDA, MUSA and NPU support GGUF quantization currently.
• Decode context parallel (decode_context_parallel_size > 1)…
• HIP does not support fused_marlin_moe currently.
• KV4 is not tested on non-CUDA platforms.
• Only CUDA, HIP and XPU support AWQ currently.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fe865727f5652487. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/layers/quantization/gguf.py:91

def _ordered_gguf_shard_ids(shard_ids: list) -> list:
    """Return checkpoint shards in the fused layer's logical output order."""
    if len(shard_ids) == 3 and set(shard_ids) == {"q", "k", "v"}:
        return ["q", "k", "v"]
    if all(isinstance(shard_id, int) for shard_id in shard_ids) and set(
        shard_ids
    ) == set(range(len(shard_ids))):
        return sorted(shard_ids)
    return list(shard_ids)


class GGUFConfig(QuantizationConfig):
    """Config class for GGUF."""

    def __init__(self, modules_to_not_convert: list[str] | None = None) -> None:
        super().__init__()
        if _is_hip:
            warnings.warn(f"Only CUDA and MUSA support GGUF quantization currently.")
        self.modules_to_not_convert = modules_to_not_convert or []

    def __repr__(self) -> str:
        return "GGUFConfig()"

    def get_scaled_act_names(self) -> List[str]:
        return []

    def get_name(self) -> str:
        return "gguf"

    def get_supported_act_dtypes(self) -> list[torch.dtype]:
        return [torch.half, torch.bfloat16, torch.float32]

    @classmethod
    def get_min_capability(cls) -> int:
        return 60 if not _is_musa else 21

View on GitHub (pinned to 0132848349)