sgl-project/sglang · warning

Only CUDA and MUSA support GGUF quantization currently.

Error message

Only CUDA and MUSA support GGUF quantization currently.

What it means

Warning from GGUFConfig.__init__: when the GGUF quantization config is constructed on HIP (ROCm), SGLang warns that only CUDA and MUSA support GGUF currently. Unlike the import-time warning (which exempts HIP), constructing GGUFConfig on ROCm explicitly warns, indicating the runtime path is not expected to work.

Source

Thrown at python/sglang/srt/layers/quantization/gguf.py:91

def _ordered_gguf_shard_ids(shard_ids: list) -> list:
    """Return checkpoint shards in the fused layer's logical output order."""
    if len(shard_ids) == 3 and set(shard_ids) == {"q", "k", "v"}:
        return ["q", "k", "v"]
    if all(isinstance(shard_id, int) for shard_id in shard_ids) and set(
        shard_ids
    ) == set(range(len(shard_ids))):
        return sorted(shard_ids)
    return list(shard_ids)


class GGUFConfig(QuantizationConfig):
    """Config class for GGUF."""

    def __init__(self, modules_to_not_convert: list[str] | None = None) -> None:
        super().__init__()
        if _is_hip:
            warnings.warn(f"Only CUDA and MUSA support GGUF quantization currently.")
        self.modules_to_not_convert = modules_to_not_convert or []

    def __repr__(self) -> str:
        return "GGUFConfig()"

    def get_scaled_act_names(self) -> List[str]:
        return []

    def get_name(self) -> str:
        return "gguf"

    def get_supported_act_dtypes(self) -> list[torch.dtype]:
        return [torch.half, torch.bfloat16, torch.float32]

    @classmethod
    def get_min_capability(cls) -> int:
        return 60 if not _is_musa else 21

View on GitHub (pinned to 0132848349)

Solutions

1. Use CUDA or MUSA hardware for GGUF models
2. Convert the GGUF checkpoint to safetensors with a ROCm-supported quantization
3. Catch/acknowledge the warning and expect degraded or failing behavior on ROCm

Example fix

# before
python -m sglang.launch_server --model model.Q8_0.gguf  # on ROCm
# after
python -m sglang.launch_server --model model-fp16  # converted for ROCm

from sglang.srt.utils import is_hip
if is_hip():
    assert not is_gguf_model(config), "GGUF not supported on ROCm"

When it happens

Trigger: Loading a model whose quantization_config in config.json is "gguf" on a ROCm system, causing GGUFConfig() instantiation during quantization method resolution.

Common situations: Serving GGUF checkpoints on AMD GPUs; reusing a CUDA deployment recipe on ROCm.

Related errors

• Only CUDA, MUSA and NPU support GGUF quantization currently.
• {self.quantization} quantization is currently not supported
• Decode context parallel (decode_context_parallel_size > 1) i
• KV4 is not tested on non-CUDA platforms.
• Only CUDA, HIP and XPU support AWQ currently.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fe865727f5652487. Report an issue: GitHub.