sgl-project/sglang · error · ValueError

GGUF is selected by passing the checkpoint itself, not `--qu

Error message

GGUF is selected by passing the checkpoint itself, not `--quantization gguf`. Use `--transformer-weights-path <file.gguf>` (or a Hub reference such as owner/repo:Q4_K_M).

What it means

GGUF checkpoints are not a --quantization mode; their quantization config must be parsed from the GGUF file header itself. The loader therefore rejects the explicit flag and tells you to point at the .gguf file directly.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:1007

    """
    resolve quant config from checkpoints' metadata
    priority: explicit --quantization flag -> model config.json -> safetensors metadata -> format-specific fallback
    """
    # priority: explicit --quantization flag (e.g. mxfp8, mxfp4_npu, modelslim)
    if server_args.quantization is not None:
        from sglang.multimodal_gen.runtime.layers.quantization import (
            get_quantization_config,
        )

        # modelslim requires a per-layer quant description file; load it from
        # the component directory rather than constructing an empty config.
        if server_args.quantization == "modelslim":
            return get_quant_config(hf_config, component_model_path)

        # GGUF is selected by pointing at the file, not by this flag: the config
        # has to be built from that file's header.
        if server_args.quantization == "gguf":
            raise ValueError(
                "GGUF is selected by passing the checkpoint itself, not "
                "`--quantization gguf`. Use "
                "`--transformer-weights-path <file.gguf>` (or a Hub reference "
                "such as owner/repo:Q4_K_M)."
            )

        # Online-quant convention: for `fp8`, `mxfp4` and `kitchen_int8`, a
        # no-arg QuantizationConfig() selects the post-load path -- weights
        # load in source dtype and are quantized in
        # process_weights_after_loading.
        quant_cls = get_quantization_config(server_args.quantization)
        quant_kwargs = {}
        if server_args.quantization in {"fp8", "mxfp4", "kitchen_int8"}:
            quant_kwargs["ignored_layers"] = getattr(
                server_args, "quantization_ignored_layers", None
            )
        return quant_cls(**quant_kwargs)

View on GitHub (pinned to 0132848349)

Solutions

1. Remove --quantization gguf and pass --transformer-weights-path /path/to/model.gguf
2. Or use a Hub reference like owner/repo:Q4_K_M as the weights path

Example fix

# before
--quantization gguf --model some/model
# after
--transformer-weights-path model.gguf --model some/model

def gguf_args_ok(quantization: str | None, weights_path: str | None) -> bool:
    if weights_path and weights_path.endswith(".gguf"):
        return quantization is None
    return quantization != "gguf"

try:
    _resolve_quant_config(...)
except ValueError as e:
    if "--quantization gguf" in str(e):
        server_args.quantization = None  # route via weights path instead

When it happens

Trigger: _resolve_quant_config (via resolve_transformer_quant_load_spec or load_customized) sees server_args.quantization == "gguf".

Common situations: User familiar with llama.cpp-style tooling assumes --quantization gguf selects the format; or a script enumerates quantization names and passes gguf as one of them.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• GGUFConfig must be constructed from a GGUF checkpoint
• A GGUF encoder checkpoint cannot be combined with a second q
• Cannot parse checkpoint quantization for {component_name!r}:
• GGUF tensor {tensor.name} is quantized, but diffusion GGUF c
• GGUF tensor {tensor.name} has inner dimension {inner_dim}, w

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9256eac54c45a945. Report an issue: GitHub.