{"record":{"id":"e38063f050209d9a","repo":"hiyouga/LlamaFactory","slug":"hqq-only-accepts-1-2-3-4-5-6-8-bit-quantization","errorCode":null,"errorMessage":"HQQ only accepts 1/2/3/4/5/6/8-bit quantization.","messagePattern":"HQQ only accepts 1/2/3/4/5/6/8-bit quantization\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":208,"sourceCode":"                )\n            else:\n                raise ValueError(\"Bitsandbytes only accepts 4-bit or 8-bit quantization.\")\n\n            # Do not assign device map if:\n            # 1. deepspeed zero3 or fsdp (train)\n            # 2. auto quantization device map (inference)\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == \"auto\":\n                if model_args.quantization_bit != 4:\n                    raise ValueError(\"Only 4-bit quantized model can use fsdp+qlora or auto device map.\")\n\n                check_version(\"bitsandbytes>=0.43.0\", mandatory=True)\n            else:\n                init_kwargs[\"device_map\"] = {\"\": get_current_device()}  # change auto device map for inference\n\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.\")\n        elif model_args.quantization_method == QuantizationMethod.HQQ:\n            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:\n                raise ValueError(\"HQQ only accepts 1/2/3/4/5/6/8-bit quantization.\")\n\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():\n                raise ValueError(\"HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.\")\n\n            check_version(\"hqq\", mandatory=True)\n            init_kwargs[\"quantization_config\"] = HqqConfig(\n                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0\n            )  # use ATEN kernel (axis=0) for performance\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with HQQ.\")\n        elif model_args.quantization_method == QuantizationMethod.EETQ:\n            if model_args.quantization_bit != 8:\n                raise ValueError(\"EETQ only accepts 8-bit quantization.\")\n\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():\n                raise ValueError(\"EETQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.\")\n\n            check_version(\"eetq\", mandatory=True)\n            init_kwargs[\"quantization_config\"] = EetqConfig()","sourceCodeStart":190,"sourceCodeEnd":226,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L190-L226","documentation":"For on-the-fly HQQ quantization, LlamaFactory validates quantization_bit against HqqConfig's supported widths [8,6,5,4,3,2,1]. Note 7-bit is absent from the list even though the message says 1/2/3/4/5/6/8. Invalid values raise ValueError before the hqq import/version check matters.","triggerScenarios":"quantization_method: hqq with quantization_bit outside [8,6,5,4,3,2,1] — e.g. 7, 12, or 16.","commonSituations":"Assuming all 1-8 widths exist and picking 7; copying a 16-bit 'half-precision' intent into quantization_bit; typos.","solutions":["Choose one of 1,2,3,4,5,6,8 for quantization_bit with quantization_method: hqq.","For 4-bit with broader ecosystem support, prefer bitsandbytes; for export, GPTQ 2/3/4/8.","Remove quantization_bit if you actually wanted plain fp16/bf16 (no quantization)."],"exampleFix":"# before (yaml)\nquantization_method: hqq\nquantization_bit: 7\n\n# after (yaml)\nquantization_method: hqq\nquantization_bit: 4","handlingStrategy":"validation","validationCode":"if model_args.quantization_method == \"hqq\":\n    assert model_args.quantization_bit in (1, 2, 3, 4, 5, 6, 8), (\n        \"HQQ supports 1/2/3/4/5/6/8 bits\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Note 7-bit is not valid for HQQ despite the 1-8 range intuition.","Prefer 4-bit defaults unless you specifically need HQQ's extreme low-bit modes."],"tags":["hqq","quantization","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}