sgl-project/sglang · error · ValueError

Checkpoint quantization is encoded in per-layer metadata; do

Error message

Checkpoint quantization is encoded in per-layer metadata; do not also set --quantization

What it means

The checkpoint on disk already carries per-layer quantization metadata, so the loader derives the quantization config from the checkpoint itself. Passing --quantization on the CLI would create two competing sources of truth for how to quantize the model, so the resolver refuses to proceed.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:788

    model_cls: type[nn.Module],
    cls_name: str,
    component_name: str | None = None,
    gguf_file: str | None = None,
    checkpoint_quant_config: QuantizationConfig | None = None,
) -> TransformerQuantLoadSpec:
    if gguf_file is not None:
        if checkpoint_quant_config is not None:
            raise ValueError("GGUF and safetensors quantization metadata conflict")
        return _resolve_gguf_quant_load_spec(
            gguf_file=gguf_file,
            server_args=server_args,
            model_cls=model_cls,
            component_name=component_name,
        )

    if checkpoint_quant_config is not None:
        if server_args.quantization is not None:
            raise ValueError(
                "Checkpoint quantization is encoded in per-layer metadata; do not "
                "also set --quantization"
            )
        if server_args.nunchaku_config is not None:
            raise ValueError(
                "Per-layer checkpoint quantization and Nunchaku are mutually "
                "exclusive"
            )
        quant_config = checkpoint_quant_config
    elif getattr(model_cls, "handles_checkpoint_quantization", False):
        quant_config = None
    else:
        quant_config = _resolve_quant_config(
            hf_config=hf_config,
            server_args=server_args,
            safetensors_list=safetensors_list,
            component_model_path=component_model_path,
        )

View on GitHub (pinned to 0132848349)

Solutions

Remove the --quantization flag from the server args / launch command; the checkpoint metadata drives quantization automatically
If you intentionally want CLI-driven quantization, point at a non-quantized checkpoint instead
Re-check server_args.quantization is None before calling load_customized in programmatic use

Example fix

# before
python -m sglang.launch_server --model quantized-model --quantization fp8
# after
python -m sglang.launch_server --model quantized-model

Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.server_args import ServerArgs

def can_load_checkpoint_quant(server_args, checkpoint_quant_config) -> bool:
    return checkpoint_quant_config is None or server_args.quantization is None

Try / catch

try:
    resolve_transformer_quant_load_spec(...)
except ValueError as e:
    if "do not also set --quantization" in str(e):
        server_args.quantization = None  # let checkpoint metadata win
        resolve_transformer_quant_load_spec(...)

Prevention

Never pass --quantization when serving a checkpoint that shipped pre-quantized
Wrap model launches in a helper that asserts server_args.quantization is None for quantized checkpoints

When it happens

Trigger: Calling resolve_transformer_quant_load_spec (directly or via load_customized) with server_args.quantization set to any non-None value while the model checkpoint contains a quantization config (checkpoint_quant_config is not None).

Common situations: User downloads a pre-quantized (e.g. fp8/int8 per-layer) checkpoint but keeps a --quantization fp8 flag in their launch script from a previous bf16 setup; or a config template hard-codes --quantization.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e329535a2501b2eb. Report an issue: GitHub.