sgl-project/sglang · error · ValueError

Quantization method specified in the model config ({quant_me

Error message

Quantization method specified in the model config ({quant_method}) does not match the quantization method specified in the `quantization` argument ({self.quantization}).

What it means

The quantization method recorded in the model's quantization_config (quant_method) disagrees with the --quantization CLI value, and the requested method is not in REQUANTIZATION_METHODS, so sglang refuses the mismatched combination.

Source

Thrown at python/sglang/srt/configs/model_config.py:1638

                    logger.info(
                        f"Using CLI-specified quantization ({self.quantization}) which is "
                        f"compatible with HF config quant_method ({quant_method})."
                    )
                elif self.is_draft_model:
                    # Allow auto-detection of quantization from checkpoint for draft model
                    # only if the CLI quantization is not compatible
                    logger.info(
                        f"Draft model quantization ({quant_method}) differs from "
                        f"main model quantization ({self.quantization}). "
                        f"Using draft model's detected quantization: {quant_method}"
                    )
                    self.quantization = quant_method
                elif self.quantization in REQUANTIZATION_METHODS:
                    logger.info_once(
                        f"Requantizing from quant_method='{quant_method}' to the requested online quantization='{self.quantization}'. Beware that requantization may incur a loss in accuracy, the requantized model should be re-validated/re-evaluated. More details at https://docs.sglang.io/advanced_features/quantization.html#online-quantization."
                    )
                else:
                    raise ValueError(
                        "Quantization method specified in the model config "
                        f"({quant_method}) does not match the quantization "
                        f"method specified in the `quantization` argument "
                        f"({self.quantization})."
                    )

        if self.quantization is not None:
            if self.quantization not in supported_quantization:
                raise ValueError(
                    f"Unknown quantization method: {self.quantization}. Must "
                    f"be one of {supported_quantization}."
                )
            if is_hip() and self.quantization not in rocm_supported_quantization:
                raise ValueError(
                    f"{self.quantization} quantization is currently not "
                    f"supported in ROCm."
                )
            if self.quantization not in optimized_quantization_methods:

View on GitHub (pinned to 0132848349)

Solutions

1. Drop the --quantization flag and let sglang infer it from the checkpoint's quantization_config
2. Change --quantization to match the checkpoint's quant_method exactly
3. If you truly want requantization, use one of the REQUANTIZATION_METHODS supported for online requantizing

Example fix

# before (fp8 checkpoint)
python -m sglang.launch_server --model MODEL --quantization awq
# after
python -m sglang.launch_server --model MODEL  # inferred as fp8
# or
python -m sglang.launch_server --model MODEL --quantization fp8

import json
q = json.load(open(f"{model_dir}/config.json")).get("quantization_config", {})
checkpoint_method = q.get("quant_method")
if checkpoint_method and cli_quantization and checkpoint_method != cli_quantization:
    # either align or omit the CLI flag
    del cli_quantization

try:
    ModelConfig(model_path=p, quantization=requested)
except ValueError as e:
    if "does not match" in str(e):
        ModelConfig(model_path=p)  # let checkpoint decide
    else:
        raise

When it happens

Trigger: ModelConfig.__init__ -> _verify_quantization finds quant_cfg['quant_method'] == X while self.quantization == Y, and Y not in REQUANTIZATION_METHODS. Typical: serving an fp8 checkpoint with --quantization awq, or a ModelOpt-fp4 config with --quantization fp8.

Common situations: Copy-pasting launch commands from a different model's recipe, reusing a serve script after swapping checkpoints, or inheriting a draft model config whose quant method differs from the target's.

Related errors

• --quantization nvfp4_online is supported only on NVIDIA Blac
• --quantization nvfp4_online supports only --moe-runner-backe
• quantize_and_serve requires ModelOpt quantization (set with
• quantize_and_serve functionality is currently disabled due t
• Config list contains configs from 2 methods, must be only 1

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/53d0cd23e0c0f720. Report an issue: GitHub.