sgl-project/sglang · error · ValueError

Per-layer checkpoint quantization and Nunchaku are mutually

Error message

Per-layer checkpoint quantization and Nunchaku are mutually exclusive

What it means

Per-layer checkpoint quantization and the Nunchaku backend are implemented as mutually exclusive weight-loading paths; combining them has no defined semantics. The resolver detects both and aborts before any weights are loaded.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:793

) -> TransformerQuantLoadSpec:
    if gguf_file is not None:
        if checkpoint_quant_config is not None:
            raise ValueError("GGUF and safetensors quantization metadata conflict")
        return _resolve_gguf_quant_load_spec(
            gguf_file=gguf_file,
            server_args=server_args,
            model_cls=model_cls,
            component_name=component_name,
        )

    if checkpoint_quant_config is not None:
        if server_args.quantization is not None:
            raise ValueError(
                "Checkpoint quantization is encoded in per-layer metadata; do not "
                "also set --quantization"
            )
        if server_args.nunchaku_config is not None:
            raise ValueError(
                "Per-layer checkpoint quantization and Nunchaku are mutually "
                "exclusive"
            )
        quant_config = checkpoint_quant_config
    elif getattr(model_cls, "handles_checkpoint_quantization", False):
        quant_config = None
    else:
        quant_config = _resolve_quant_config(
            hf_config=hf_config,
            server_args=server_args,
            safetensors_list=safetensors_list,
            component_model_path=component_model_path,
        )

    if quant_config is not None:
        packed = getattr(model_cls, "packed_modules_mapping", None)
        if packed and hasattr(quant_config, "packed_modules_mapping"):
            quant_config.packed_modules_mapping = packed

View on GitHub (pinned to 0132848349)

Solutions

Remove the --nunchaku-config flag so the checkpoint's per-layer quantization is used
Alternatively, use a Nunchaku-compatible (unquantized or Nunchaku-format) checkpoint and drop the per-layer quantized one
Programmatically, assert server_args.nunchaku_config is None before loading a checkpoint with quant metadata

Example fix

# before
--nunchaku-config ... --model per-layer-quantized-model
# after
--model per-layer-quantized-model

Defensive patterns

Strategy: validation

Validate before calling

def nunchaku_compatible(server_args, checkpoint_quant_config) -> bool:
    return checkpoint_quant_config is None or server_args.nunchaku_config is None

Try / catch

try:
    resolve_transformer_quant_load_spec(...)
except ValueError as e:
    if "Nunchaku are mutually exclusive" in str(e):
        server_args.nunchaku_config = None
        resolve_transformer_quant_load_spec(...)

Prevention

Pick one quantization path per deployment: Nunchaku backend OR per-layer quantized checkpoint
Fail fast at config-parse time by rejecting the combination in your own arg validation

When it happens

Trigger: resolve_transformer_quant_load_spec / load_customized is invoked with a checkpoint that has quantization metadata (checkpoint_quant_config is not None) while server_args.nunchaku_config is set.

Common situations: User switches to a per-layer quantized checkpoint but leaves --nunchaku-config in their launch config; or copies a Nunchaku-serving command line onto a new quantized model.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5b20d05ad8162ccd. Report an issue: GitHub.