hiyouga/LlamaFactory · error · ValueError

AutoGPTQ only accepts 2/3/4/8-bit quantization.

Error message

AutoGPTQ only accepts 2/3/4/8-bit quantization.

What it means

During export-time GPTQ quantization via gptqmodel/optimum, LlamaFactory validates export_quantization_bit against the only widths AutoGPTQ supports: 2, 3, 4, or 8 bits. Any other value raises ValueError before calibration starts.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:145

        if quant_method == QuantizationMethod.GPTQ:
            check_version("gptqmodel>=2.0.0", mandatory=True)
            quantization_config.pop("disable_exllama", None)  # remove deprecated args
            quantization_config["use_exllama"] = False  # disable exllama

        if quant_method == QuantizationMethod.AWQ:
            check_version("autoawq", mandatory=True)

        if quant_method == QuantizationMethod.AQLM:
            check_version("aqlm>=1.1.0", mandatory=True)
            quantization_config["bits"] = 2

        quant_bits = quantization_config.get("bits", "?")
        logger.info_rank0(f"Loading {quant_bits}-bit {quant_method.upper()}-quantized model.")

    elif model_args.export_quantization_bit is not None:  # gptqmodel
        if model_args.export_quantization_bit not in [8, 4, 3, 2]:
            raise ValueError("AutoGPTQ only accepts 2/3/4/8-bit quantization.")

        check_version("optimum>=1.24.0", mandatory=True)
        check_version("gptqmodel>=2.0.0", mandatory=True)
        from accelerate.utils import get_max_memory

        if getattr(config, "model_type", None) == "chatglm":
            raise ValueError("ChatGLM model is not supported yet.")

        try:
            from optimum.gptq import utils as gq_utils

            if "language_model.model.layers" not in gq_utils.BLOCK_PATTERNS:
                gq_utils.BLOCK_PATTERNS.insert(0, "language_model.model.layers")
        except ImportError:
            pass

        block_name_to_quantize = None
        if getattr(config, "model_type", None) in ["gemma3", "paligemma"]:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set export_quantization_bit to 2, 3, 4, or 8 (4 is the common default).
2. If you need other widths, use export_quantization_method hqq-style paths or quantize outside LlamaFactory with a library that supports it.
3. Double-check the key: this is export_quantization_bit (export), not quantization_bit (train-time).

Example fix

# before (export yaml)
export_quantization_bit: 6

# after
export_quantization_bit: 4

assert export_quantization_bit in (2, 3, 4, 8), (
    f"export_quantization_bit must be 2/3/4/8 for GPTQ; got {export_quantization_bit}"
)

When it happens

Trigger: llamafactory-cli export with export_quantization_bit set to a value outside [2,3,4,8] — e.g. 16, 6, 5, or 1.

Common situations: Users assuming arbitrary bit widths (6-bit to hit a size target); typos like export_quantization_bit: 44; confusing this setting with HQQ (which allows 1-6 bits) or with quantization_bit.

Related errors

• Cannot find satisfying example, considering decrease `export
• Quantization dataset is necessary for exporting.
• Cannot use device map for quantized models in training.
• FP8 training is not compatible with quantization. Please dis
• vLLM engine does not support bnb quantization (GPTQ and AWQ

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/358f6be0feeb23f2. Report an issue: GitHub.