sgl-project/sglang · error · ValueError

--quantization {server_args.quantization} cannot be combined

Error message

--quantization {server_args.quantization} cannot be combined with a GGUF transformer, whose quantization is fixed by the checkpoint. Drop the flag, or use an unquantized checkpoint to quantize online.

What it means

The loader refuses to load a GGUF transformer checkpoint when --quantization is also set, because a GGUF file's quantization format is baked into the checkpoint itself and cannot be changed or supplemented by an online quantization method.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:530

    Called before the checkpoint is downloaded or read, so an unsupported
    combination costs a second rather than a multi-gigabyte fetch.

    ``component_name`` selects the FSDP decision to check. FSDP is resolved per
    component, so a globally enabled ``--use-fsdp-inference`` does not shard a
    transformer that is offloaded; only the component actually holding the
    packed weights matters.
    """
    # The quantization comes from the file, so an explicit --quantization is
    # either redundant (gguf) or a conflicting request that would otherwise be
    # dropped without a word.
    if server_args.quantization == "gguf":
        raise ValueError(
            "GGUF is selected by passing the checkpoint itself, not "
            "`--quantization gguf`. Drop the flag; "
            "`--transformer-weights-path <file.gguf>` is what enables it."
        )
    if server_args.quantization is not None:
        raise ValueError(
            f"--quantization {server_args.quantization} cannot be combined with "
            "a GGUF transformer, whose quantization is fixed by the checkpoint. "
            "Drop the flag, or use an unquantized checkpoint to quantize online."
        )
    # Nunchaku shares --transformer-weights-path with GGUF, and the GGUF plan is
    # resolved first, so without this the SVDQuant request would be dropped in
    # silence rather than refused.
    if server_args.nunchaku_config is not None:
        raise ValueError(
            "--enable-svdquant cannot be combined with a GGUF transformer: both "
            "supply the transformer weights. Point "
            "--transformer-weights-path at either an SVDQuant checkpoint or a "
            ".gguf, not one while requesting the other."
        )
    if not current_platform.is_cuda():
        raise ValueError(
            "GGUF diffusion checkpoints require CUDA; the GGML kernels have no "
            f"{current_platform.device_type} implementation."

View on GitHub (pinned to 0132848349)

Solutions

1. Remove the --quantization flag; GGUF quantization is fixed by the checkpoint
2. If you want online quantization with --quantization, use an unquantized (safetensors) checkpoint instead of GGUF

Example fix

# before
--transformer-weights-path model.gguf --quantization awq
# after
--transformer-weights-path model.gguf

if server_args.quantization is not None and str(server_args.transformer_weights_path or '').endswith('.gguf'):
    raise SystemExit('Drop --quantization: GGUF quantization is fixed by the checkpoint')

When it happens

Trigger: Passing server_args.quantization (e.g. --quantization awq) together with a .gguf file via --transformer-weights-path; _validate_gguf_runtime_support raises whenever server_args.quantization is not None on a GGUF load.

Common situations: Copy-pasting a launch command from an unquantized/safetensors deployment and just swapping the model path to a .gguf file, leaving the old --quantization flag in place.

Related errors

• GGUFConfig must be constructed from a GGUF checkpoint
• A GGUF encoder checkpoint cannot be combined with a second q
• Cannot parse checkpoint quantization for {component_name!r}:
• GGUF tensor {tensor.name} is quantized, but diffusion GGUF c
• GGUF tensor {tensor.name} has inner dimension {inner_dim}, w

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9ddff4518c7532d8. Report an issue: GitHub.