hiyouga/LlamaFactory · error · ValueError

Unsupported quantization bit: {quant_config.quantization_bit

Error message

Unsupported quantization bit: {quant_config.quantization_bit} for auto quantization.

What it means

The 'auto' quantization plugin dispatches to bitsandbytes and only supports 4-bit or 8-bit widths; quantization_bit values like 2, 3, or 16 are rejected immediately. This mirrors bitsandbytes' own capability, since the auto route forwards to the bnb plugin after validating the bit width.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/quantization.py:69

            if not isinstance(dtype, torch.dtype):
                raise ValueError(f"compute_dtype={self.compute_dtype!r} is not a torch dtype name.")
            self.compute_dtype = dtype
        elif not isinstance(self.compute_dtype, torch.dtype):
            raise TypeError(f"compute_dtype must be str or torch.dtype, got {type(self.compute_dtype).__name__}.")


@QuantizationPlugin("auto").register()
def quantization_auto(
    init_kwargs: dict[str, Any],
    quant_config: dict | BnbParams,
    is_trainable: bool = False,
) -> dict[str, Any]:
    quant_config = QuantizationPlugin.parse_params(quant_config, BnbParams)
    if quant_config.quantization_bit is None:
        logger.warning_rank0("No quantization method applied.")
        return init_kwargs
    if quant_config.quantization_bit not in (4, 8):
        raise ValueError(f"Unsupported quantization bit: {quant_config.quantization_bit} for auto quantization.")

    logger.info_rank0(f"Loading {quant_config.quantization_bit}-bit quantized model.")
    return QuantizationPlugin("bnb")(init_kwargs, quant_config=quant_config, is_trainable=is_trainable)


@QuantizationPlugin("bnb").register()
def quantization_with_bnb(
    init_kwargs: dict[str, Any],
    quant_config: dict | BnbParams,
    is_trainable: bool = False,
) -> dict[str, Any]:
    from transformers import BitsAndBytesConfig

    from ...accelerator.helper import get_current_device
    from ...utils.packages import check_version

    quant_config = QuantizationPlugin.parse_params(quant_config, BnbParams)
    quantization_bit = quant_config.quantization_bit

View on GitHub (pinned to f28afaf635)

Solutions

1. Set quantization_bit to 4 or 8
2. For lower bit widths, use a different quantization method/plugin (e.g. a GPTQ/AWQ pipeline) or pre-quantized checkpoints
3. Omit quantization_bit to get the logged no-quantization path if quantization was unintentional

Example fix

# before
quantization:
  name: auto
  quantization_bit: 3

# after
quantization:
  name: auto
  quantization_bit: 4

bit = quant_config.get("quantization_bit")
assert bit is None or bit in (4, 8), f"auto quantization supports 4/8 bits, got {bit}"

When it happens

Trigger: quantization name 'auto' (or 'bnb') with quantization_bit set to a value other than 4 or 8 in the quantization config.

Common situations: User tries 2-bit/3-bit extreme quantization (only available via GPTQ/AWQ-style tools, not bnb); typo such as quantization_bit: 4.0 or 44; copy-paste from a config for a different quant backend.

Related errors

• compute_dtype={self.compute_dtype!r} is not a torch dtype na
• Bitsandbytes only accepts 4-bit or 8-bit quantization.
• Bitsandbytes only accepts 4-bit or 8-bit quantization.
• compute_dtype must be str or torch.dtype, got {type(self.com
• Only 4-bit quantized model can use fsdp+qlora or auto device

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/dca5751c53ffdd1f. Report an issue: GitHub.