sgl-project/sglang · error · ValueError

GGUFConfig must be constructed from a GGUF checkpoint

Error message

GGUFConfig must be constructed from a GGUF checkpoint

What it means

GGUFConfig.from_config unconditionally raises because GGUF checkpoints are not loaded through the standard quantization-config JSON path. GGUF models are constructed directly from the .gguf file, so building the config from a plain dict is unsupported.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/gguf.py:80

    @classmethod
    def get_name(cls) -> str:
        return "gguf"

    @classmethod
    def get_supported_act_dtypes(cls) -> list[torch.dtype]:
        return [torch.float32, torch.float16, torch.bfloat16]

    @classmethod
    def get_min_capability(cls) -> int:
        return 60

    @staticmethod
    def get_config_filenames() -> list[str]:
        return []

    @classmethod
    def from_config(cls, config: dict[str, Any]) -> GGUFConfig:
        raise ValueError("GGUFConfig must be constructed from a GGUF checkpoint")

    def get_quant_method(
        self, layer: nn.Module, prefix: str
    ) -> QuantizeMethodBase | None:
        if isinstance(layer, LinearBase):
            unquantized_method = UnquantizedLinearMethod
        elif isinstance(layer, VocabParallelEmbedding):
            unquantized_method = None
        else:
            return None

        metadata = self.tensor_meta.get(f"{prefix}.weight")
        if metadata is None:
            raise ValueError(
                f"Linear layer {prefix!r} has no weight in the GGUF checkpoint "
                f"{self.gguf_file!r}"
            )
        weight_type = metadata.weight_type

View on GitHub (pinned to 0132848349)

Solutions

1. Load GGUF models via the dedicated GGUF loading path (construct GGUFConfig directly from the .gguf checkpoint)
2. Remove/fix the quantization config that routes to GGUFConfig.from_config
3. Convert the checkpoint to a supported format if GGUF loading is not intended

if isinstance(config, dict) and 'gguf' in str(config.get('quant_method','')).lower():
    raise SystemExit('Load GGUF models via the GGUF loading path, not from_config')

def is_gguf_checkpoint(path: str) -> bool:
    return path.endswith('.gguf')

When it happens

Trigger: Calling GGUFConfig.from_config(config_dict), or a generic config loader auto-detecting a quant method named GGUF in hf_quant_config.json and trying to instantiate it via from_config.

Common situations: Passing a GGUF file through a loader designed for HuggingFace quantization configs (e.g. transformers-style CLI), or misconfigured quantization: gguf in a config JSON.

Related errors

• The quantization method `{quantization}` is already exists.
• The quantization config must be a subclass of `QuantizationC
• Invalid quantization method: {quantization}
• kitchen_w4a4 is inferred from per-layer checkpoint metadata;
• f"Cannot parse checkpoint quantization for {component_name!r

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/069569e00479f268. Report an issue: GitHub.