{"record":{"id":"e329535a2501b2eb","repo":"sgl-project/sglang","slug":"checkpoint-quantization-is-encoded-in-per-layer-me","errorCode":null,"errorMessage":"Checkpoint quantization is encoded in per-layer metadata; do not also set --quantization","messagePattern":"Checkpoint quantization is encoded in per-layer metadata; do not also set --quantization","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py","lineNumber":788,"sourceCode":"    model_cls: type[nn.Module],\n    cls_name: str,\n    component_name: str | None = None,\n    gguf_file: str | None = None,\n    checkpoint_quant_config: QuantizationConfig | None = None,\n) -> TransformerQuantLoadSpec:\n    if gguf_file is not None:\n        if checkpoint_quant_config is not None:\n            raise ValueError(\"GGUF and safetensors quantization metadata conflict\")\n        return _resolve_gguf_quant_load_spec(\n            gguf_file=gguf_file,\n            server_args=server_args,\n            model_cls=model_cls,\n            component_name=component_name,\n        )\n\n    if checkpoint_quant_config is not None:\n        if server_args.quantization is not None:\n            raise ValueError(\n                \"Checkpoint quantization is encoded in per-layer metadata; do not \"\n                \"also set --quantization\"\n            )\n        if server_args.nunchaku_config is not None:\n            raise ValueError(\n                \"Per-layer checkpoint quantization and Nunchaku are mutually \"\n                \"exclusive\"\n            )\n        quant_config = checkpoint_quant_config\n    elif getattr(model_cls, \"handles_checkpoint_quantization\", False):\n        quant_config = None\n    else:\n        quant_config = _resolve_quant_config(\n            hf_config=hf_config,\n            server_args=server_args,\n            safetensors_list=safetensors_list,\n            component_model_path=component_model_path,\n        )","sourceCodeStart":770,"sourceCodeEnd":806,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py#L770-L806","documentation":"The checkpoint on disk already carries per-layer quantization metadata, so the loader derives the quantization config from the checkpoint itself. Passing --quantization on the CLI would create two competing sources of truth for how to quantize the model, so the resolver refuses to proceed.","triggerScenarios":"Calling resolve_transformer_quant_load_spec (directly or via load_customized) with server_args.quantization set to any non-None value while the model checkpoint contains a quantization config (checkpoint_quant_config is not None).","commonSituations":"User downloads a pre-quantized (e.g. fp8/int8 per-layer) checkpoint but keeps a --quantization fp8 flag in their launch script from a previous bf16 setup; or a config template hard-codes --quantization.","solutions":["Remove the --quantization flag from the server args / launch command; the checkpoint metadata drives quantization automatically","If you intentionally want CLI-driven quantization, point at a non-quantized checkpoint instead","Re-check server_args.quantization is None before calling load_customized in programmatic use"],"exampleFix":"# before\npython -m sglang.launch_server --model quantized-model --quantization fp8\n# after\npython -m sglang.launch_server --model quantized-model","handlingStrategy":"validation","validationCode":"from sglang.srt.server_args import ServerArgs\n\ndef can_load_checkpoint_quant(server_args, checkpoint_quant_config) -> bool:\n    return checkpoint_quant_config is None or server_args.quantization is None","typeGuard":null,"tryCatchPattern":"try:\n    resolve_transformer_quant_load_spec(...)\nexcept ValueError as e:\n    if \"do not also set --quantization\" in str(e):\n        server_args.quantization = None  # let checkpoint metadata win\n        resolve_transformer_quant_load_spec(...)","preventionTips":["Never pass --quantization when serving a checkpoint that shipped pre-quantized","Wrap model launches in a helper that asserts server_args.quantization is None for quantized checkpoints"],"tags":["quantization","config-conflict","checkpoint","server-args"],"backgroundTag":"conflicting-configuration-options","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}