hiyouga/LlamaFactory · error · ValueError

Bitsandbytes only accepts 4-bit or 8-bit quantization.

Error message

Bitsandbytes only accepts 4-bit or 8-bit quantization.

What it means

The bnb quantization plugin configures transformers' BitsAndBytesConfig, which only implements load_in_4bit and load_in_8bit. Any other quantization_bit is rejected before constructing the config. If quantization_bit is omitted the plugin defaults to 4-bit with a warning, so this error specifically means an explicit invalid value was supplied.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/quantization.py:92

@QuantizationPlugin("bnb").register()
def quantization_with_bnb(
    init_kwargs: dict[str, Any],
    quant_config: dict | BnbParams,
    is_trainable: bool = False,
) -> dict[str, Any]:
    from transformers import BitsAndBytesConfig

    from ...accelerator.helper import get_current_device
    from ...utils.packages import check_version

    quant_config = QuantizationPlugin.parse_params(quant_config, BnbParams)
    quantization_bit = quant_config.quantization_bit
    if quantization_bit is None:
        logger.warning_rank0("quantization_bit is not specified, default to 4-bit quantization.")
        quantization_bit = 4
    if quantization_bit not in (4, 8):
        raise ValueError("Bitsandbytes only accepts 4-bit or 8-bit quantization.")

    logger.info_rank0("Using Bitsandbytes quantization.")
    if quantization_bit == 8:
        check_version("bitsandbytes>=0.37.0", mandatory=True)
        init_kwargs["quantization_config"] = BitsAndBytesConfig(load_in_8bit=True)
    else:
        check_version("bitsandbytes>=0.39.0", mandatory=True)
        init_kwargs["quantization_config"] = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=quant_config.compute_dtype,
            bnb_4bit_use_double_quant=quant_config.double_quantization,
            bnb_4bit_quant_type=quant_config.quantization_type,
            bnb_4bit_quant_storage=quant_config.compute_dtype,
        )

    if is_trainable:
        logger.info_rank0("Detected inference mode, setting device_map for bitsandbytes quantization.")
        init_kwargs["device_map"] = {"": get_current_device()}

View on GitHub (pinned to f28afaf635)

Solutions

1. Use quantization_bit: 4 (QLoRA-capable) or quantization_bit: 8
2. For 2/3-bit, switch to a GPTQ or AWQ quantized model rather than bnb
3. Remove the key entirely if you want the 4-bit default

Example fix

# before
quantization:
  name: bnb
  quantization_bit: 2

# after
quantization:
  name: bnb
  quantization_bit: 4

bit = quant_config.get("quantization_bit", 4)
assert bit in (4, 8), f"bitsandbytes supports 4/8 bits only, got {bit}"

When it happens

Trigger: Explicitly setting quantization_bit to a value outside {4, 8} with the 'bnb' quantization plugin.

Common situations: Attempting sub-4-bit quantization with bitsandbytes; mixing up quantization_bit with bits-per-component settings of other backends; stale configs from tools that accepted arbitrary bit widths.

Related errors

• compute_dtype={self.compute_dtype!r} is not a torch dtype na
• Unsupported quantization bit: {quant_config.quantization_bit
• Bitsandbytes only accepts 4-bit or 8-bit quantization.
• compute_dtype must be str or torch.dtype, got {type(self.com
• Only 4-bit quantized model can use fsdp+qlora or auto device

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/43ccd1286ea31572. Report an issue: GitHub.