{"record":{"id":"358f6be0feeb23f2","repo":"hiyouga/LlamaFactory","slug":"autogptq-only-accepts-2-3-4-8-bit-quantization","errorCode":null,"errorMessage":"AutoGPTQ only accepts 2/3/4/8-bit quantization.","messagePattern":"AutoGPTQ only accepts 2/3/4/8-bit quantization\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":145,"sourceCode":"\n        if quant_method == QuantizationMethod.GPTQ:\n            check_version(\"gptqmodel>=2.0.0\", mandatory=True)\n            quantization_config.pop(\"disable_exllama\", None)  # remove deprecated args\n            quantization_config[\"use_exllama\"] = False  # disable exllama\n\n        if quant_method == QuantizationMethod.AWQ:\n            check_version(\"autoawq\", mandatory=True)\n\n        if quant_method == QuantizationMethod.AQLM:\n            check_version(\"aqlm>=1.1.0\", mandatory=True)\n            quantization_config[\"bits\"] = 2\n\n        quant_bits = quantization_config.get(\"bits\", \"?\")\n        logger.info_rank0(f\"Loading {quant_bits}-bit {quant_method.upper()}-quantized model.\")\n\n    elif model_args.export_quantization_bit is not None:  # gptqmodel\n        if model_args.export_quantization_bit not in [8, 4, 3, 2]:\n            raise ValueError(\"AutoGPTQ only accepts 2/3/4/8-bit quantization.\")\n\n        check_version(\"optimum>=1.24.0\", mandatory=True)\n        check_version(\"gptqmodel>=2.0.0\", mandatory=True)\n        from accelerate.utils import get_max_memory\n\n        if getattr(config, \"model_type\", None) == \"chatglm\":\n            raise ValueError(\"ChatGLM model is not supported yet.\")\n\n        try:\n            from optimum.gptq import utils as gq_utils\n\n            if \"language_model.model.layers\" not in gq_utils.BLOCK_PATTERNS:\n                gq_utils.BLOCK_PATTERNS.insert(0, \"language_model.model.layers\")\n        except ImportError:\n            pass\n\n        block_name_to_quantize = None\n        if getattr(config, \"model_type\", None) in [\"gemma3\", \"paligemma\"]:","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L127-L163","documentation":"During export-time GPTQ quantization via gptqmodel/optimum, LlamaFactory validates export_quantization_bit against the only widths AutoGPTQ supports: 2, 3, 4, or 8 bits. Any other value raises ValueError before calibration starts.","triggerScenarios":"llamafactory-cli export with export_quantization_bit set to a value outside [2,3,4,8] — e.g. 16, 6, 5, or 1.","commonSituations":"Users assuming arbitrary bit widths (6-bit to hit a size target); typos like export_quantization_bit: 44; confusing this setting with HQQ (which allows 1-6 bits) or with quantization_bit.","solutions":["Set export_quantization_bit to 2, 3, 4, or 8 (4 is the common default).","If you need other widths, use export_quantization_method hqq-style paths or quantize outside LlamaFactory with a library that supports it.","Double-check the key: this is export_quantization_bit (export), not quantization_bit (train-time)."],"exampleFix":"# before (export yaml)\nexport_quantization_bit: 6\n\n# after\nexport_quantization_bit: 4","handlingStrategy":"validation","validationCode":"assert export_quantization_bit in (2, 3, 4, 8), (\n    f\"export_quantization_bit must be 2/3/4/8 for GPTQ; got {export_quantization_bit}\"\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Remember the per-method bit matrices: GPTQ 2/3/4/8, bnb 4/8, HQQ 1-6/8, EETQ 8.","Validate export configs in a lint step before queuing long export jobs."],"tags":["gptq","export","quantization","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}