hiyouga/LlamaFactory · error · ValueError

HQQ only accepts 1/2/3/4/5/6/8-bit quantization.

Error message

HQQ only accepts 1/2/3/4/5/6/8-bit quantization.

What it means

For on-the-fly HQQ quantization, LlamaFactory validates quantization_bit against HqqConfig's supported widths [8,6,5,4,3,2,1]. Note 7-bit is absent from the list even though the message says 1/2/3/4/5/6/8. Invalid values raise ValueError before the hqq import/version check matters.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:208

                )
            else:
                raise ValueError("Bitsandbytes only accepts 4-bit or 8-bit quantization.")

            # Do not assign device map if:
            # 1. deepspeed zero3 or fsdp (train)
            # 2. auto quantization device map (inference)
            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == "auto":
                if model_args.quantization_bit != 4:
                    raise ValueError("Only 4-bit quantized model can use fsdp+qlora or auto device map.")

                check_version("bitsandbytes>=0.43.0", mandatory=True)
            else:
                init_kwargs["device_map"] = {"": get_current_device()}  # change auto device map for inference

            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.")
        elif model_args.quantization_method == QuantizationMethod.HQQ:
            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:
                raise ValueError("HQQ only accepts 1/2/3/4/5/6/8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("hqq", mandatory=True)
            init_kwargs["quantization_config"] = HqqConfig(
                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0
            )  # use ATEN kernel (axis=0) for performance
            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with HQQ.")
        elif model_args.quantization_method == QuantizationMethod.EETQ:
            if model_args.quantization_bit != 8:
                raise ValueError("EETQ only accepts 8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("EETQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("eetq", mandatory=True)
            init_kwargs["quantization_config"] = EetqConfig()

View on GitHub (pinned to f28afaf635)

Solutions

1. Choose one of 1,2,3,4,5,6,8 for quantization_bit with quantization_method: hqq.
2. For 4-bit with broader ecosystem support, prefer bitsandbytes; for export, GPTQ 2/3/4/8.
3. Remove quantization_bit if you actually wanted plain fp16/bf16 (no quantization).

Example fix

# before (yaml)
quantization_method: hqq
quantization_bit: 7

# after (yaml)
quantization_method: hqq
quantization_bit: 4

if model_args.quantization_method == "hqq":
    assert model_args.quantization_bit in (1, 2, 3, 4, 5, 6, 8), (
        "HQQ supports 1/2/3/4/5/6/8 bits"
    )

When it happens

Trigger: quantization_method: hqq with quantization_bit outside [8,6,5,4,3,2,1] — e.g. 7, 12, or 16.

Common situations: Assuming all 1-8 widths exist and picking 7; copying a 16-bit 'half-precision' intent into quantization_bit; typos.

Related errors

• Cannot use device map for quantized models in training.
• FP8 training is not compatible with quantization. Please dis
• AutoGPTQ only accepts 2/3/4/8-bit quantization.
• Bitsandbytes only accepts 4-bit or 8-bit quantization.
• HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FS

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e38063f050209d9a. Report an issue: GitHub.