hiyouga/LlamaFactory · error · ValueError

Bitsandbytes only accepts 4-bit or 8-bit quantization.

Error message

Bitsandbytes only accepts 4-bit or 8-bit quantization.

What it means

For on-the-fly bitsandbytes quantization, LlamaFactory only builds configs for quantization_bit == 8 (load_in_8bit) and == 4 (load_in_4bit with QLoRA options). Any other bit value falls to the else and raises ValueError naming the bitsandbytes limitation.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:192

        model_args.compute_dtype = torch.float16  # force fp16 for gptqmodel
        logger.info_rank0(f"Quantizing model to {model_args.export_quantization_bit} bit with GPTQModel.")

    elif model_args.quantization_bit is not None:  # on-the-fly
        if model_args.quantization_method == QuantizationMethod.BNB:
            if model_args.quantization_bit == 8:
                check_version("bitsandbytes>=0.37.0", mandatory=True)
                init_kwargs["quantization_config"] = BitsAndBytesConfig(load_in_8bit=True)
            elif model_args.quantization_bit == 4:
                check_version("bitsandbytes>=0.39.0", mandatory=True)
                init_kwargs["quantization_config"] = BitsAndBytesConfig(
                    load_in_4bit=True,
                    bnb_4bit_compute_dtype=model_args.compute_dtype,
                    bnb_4bit_use_double_quant=model_args.double_quantization,
                    bnb_4bit_quant_type=model_args.quantization_type,
                    bnb_4bit_quant_storage=model_args.compute_dtype,  # crucial for fsdp+qlora
                )
            else:
                raise ValueError("Bitsandbytes only accepts 4-bit or 8-bit quantization.")

            # Do not assign device map if:
            # 1. deepspeed zero3 or fsdp (train)
            # 2. auto quantization device map (inference)
            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == "auto":
                if model_args.quantization_bit != 4:
                    raise ValueError("Only 4-bit quantized model can use fsdp+qlora or auto device map.")

                check_version("bitsandbytes>=0.43.0", mandatory=True)
            else:
                init_kwargs["device_map"] = {"": get_current_device()}  # change auto device map for inference

            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.")
        elif model_args.quantization_method == QuantizationMethod.HQQ:
            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:
                raise ValueError("HQQ only accepts 1/2/3/4/5/6/8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():

View on GitHub (pinned to f28afaf635)

Solutions

1. Set quantization_bit: 4 (QLoRA) or quantization_bit: 8.
2. If you need lower bits, switch quantization_method: hqq (1-6/8 bits) or export-time GPTQ instead.
3. Confirm you are not confusing train-time quantization_bit with export-time export_quantization_bit.

Example fix

# before (yaml)
quantization_bit: 2

# after (yaml)
quantization_bit: 4
# or: quantization_method: hqq + hqq-compatible bit width

if model_args.quantization_method == "bitsandbytes":
    assert model_args.quantization_bit in (4, 8), (
        "bitsandbytes supports only 4-bit or 8-bit"
    )

When it happens

Trigger: Setting quantization_bit to anything other than 4 or 8 while quantization_method is bitsandbytes (the default method) — e.g. quantization_bit: 2 or 16.

Common situations: Assuming bitsandbytes supports 2/3-bit like GPTQ/HQQ; copying export_quantization_bit values into quantization_bit; typos.

Related errors

• Cannot use device map for quantized models in training.
• FP8 training is not compatible with quantization. Please dis
• AutoGPTQ only accepts 2/3/4/8-bit quantization.
• HQQ only accepts 1/2/3/4/5/6/8-bit quantization.
• EETQ only accepts 8-bit quantization.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/16c24830c1a2912c. Report an issue: GitHub.