{"record":{"id":"53d0cd23e0c0f720","repo":"sgl-project/sglang","slug":"quantization-method-specified-in-the-model-config","errorCode":null,"errorMessage":"Quantization method specified in the model config ({quant_method}) does not match the quantization method specified in the `quantization` argument ({self.quantization}).","messagePattern":"Quantization method specified in the model config \\((.+?)\\) does not match the quantization method specified in the `quantization` argument \\((.+?)\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/configs/model_config.py","lineNumber":1638,"sourceCode":"                    logger.info(\n                        f\"Using CLI-specified quantization ({self.quantization}) which is \"\n                        f\"compatible with HF config quant_method ({quant_method}).\"\n                    )\n                elif self.is_draft_model:\n                    # Allow auto-detection of quantization from checkpoint for draft model\n                    # only if the CLI quantization is not compatible\n                    logger.info(\n                        f\"Draft model quantization ({quant_method}) differs from \"\n                        f\"main model quantization ({self.quantization}). \"\n                        f\"Using draft model's detected quantization: {quant_method}\"\n                    )\n                    self.quantization = quant_method\n                elif self.quantization in REQUANTIZATION_METHODS:\n                    logger.info_once(\n                        f\"Requantizing from quant_method='{quant_method}' to the requested online quantization='{self.quantization}'. Beware that requantization may incur a loss in accuracy, the requantized model should be re-validated/re-evaluated. More details at https://docs.sglang.io/advanced_features/quantization.html#online-quantization.\"\n                    )\n                else:\n                    raise ValueError(\n                        \"Quantization method specified in the model config \"\n                        f\"({quant_method}) does not match the quantization \"\n                        f\"method specified in the `quantization` argument \"\n                        f\"({self.quantization}).\"\n                    )\n\n        if self.quantization is not None:\n            if self.quantization not in supported_quantization:\n                raise ValueError(\n                    f\"Unknown quantization method: {self.quantization}. Must \"\n                    f\"be one of {supported_quantization}.\"\n                )\n            if is_hip() and self.quantization not in rocm_supported_quantization:\n                raise ValueError(\n                    f\"{self.quantization} quantization is currently not \"\n                    f\"supported in ROCm.\"\n                )\n            if self.quantization not in optimized_quantization_methods:","sourceCodeStart":1620,"sourceCodeEnd":1656,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/configs/model_config.py#L1620-L1656","documentation":"The quantization method recorded in the model's quantization_config (quant_method) disagrees with the --quantization CLI value, and the requested method is not in REQUANTIZATION_METHODS, so sglang refuses the mismatched combination.","triggerScenarios":"ModelConfig.__init__ -> _verify_quantization finds quant_cfg['quant_method'] == X while self.quantization == Y, and Y not in REQUANTIZATION_METHODS. Typical: serving an fp8 checkpoint with --quantization awq, or a ModelOpt-fp4 config with --quantization fp8.","commonSituations":"Copy-pasting launch commands from a different model's recipe, reusing a serve script after swapping checkpoints, or inheriting a draft model config whose quant method differs from the target's.","solutions":["Drop the --quantization flag and let sglang infer it from the checkpoint's quantization_config","Change --quantization to match the checkpoint's quant_method exactly","If you truly want requantization, use one of the REQUANTIZATION_METHODS supported for online requantizing"],"exampleFix":"# before (fp8 checkpoint)\npython -m sglang.launch_server --model MODEL --quantization awq\n# after\npython -m sglang.launch_server --model MODEL  # inferred as fp8\n# or\npython -m sglang.launch_server --model MODEL --quantization fp8","handlingStrategy":"validation","validationCode":"import json\nq = json.load(open(f\"{model_dir}/config.json\")).get(\"quantization_config\", {})\ncheckpoint_method = q.get(\"quant_method\")\nif checkpoint_method and cli_quantization and checkpoint_method != cli_quantization:\n    # either align or omit the CLI flag\n    del cli_quantization","typeGuard":null,"tryCatchPattern":"try:\n    ModelConfig(model_path=p, quantization=requested)\nexcept ValueError as e:\n    if \"does not match\" in str(e):\n        ModelConfig(model_path=p)  # let checkpoint decide\n    else:\n        raise","preventionTips":["Default to omitting --quantization for pre-quantized checkpoints","Keep one serve script per checkpoint rather than a shared flag soup"],"tags":["quantization","config-mismatch","cli-args","sglang"],"backgroundTag":"config-value-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}