{"record":{"id":"dca5751c53ffdd1f","repo":"hiyouga/LlamaFactory","slug":"unsupported-quantization-bit-quant-config-quanti","errorCode":null,"errorMessage":"Unsupported quantization bit: {quant_config.quantization_bit} for auto quantization.","messagePattern":"Unsupported quantization bit: (.+?) for auto quantization\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/model_plugins/quantization.py","lineNumber":69,"sourceCode":"            if not isinstance(dtype, torch.dtype):\n                raise ValueError(f\"compute_dtype={self.compute_dtype!r} is not a torch dtype name.\")\n            self.compute_dtype = dtype\n        elif not isinstance(self.compute_dtype, torch.dtype):\n            raise TypeError(f\"compute_dtype must be str or torch.dtype, got {type(self.compute_dtype).__name__}.\")\n\n\n@QuantizationPlugin(\"auto\").register()\ndef quantization_auto(\n    init_kwargs: dict[str, Any],\n    quant_config: dict | BnbParams,\n    is_trainable: bool = False,\n) -> dict[str, Any]:\n    quant_config = QuantizationPlugin.parse_params(quant_config, BnbParams)\n    if quant_config.quantization_bit is None:\n        logger.warning_rank0(\"No quantization method applied.\")\n        return init_kwargs\n    if quant_config.quantization_bit not in (4, 8):\n        raise ValueError(f\"Unsupported quantization bit: {quant_config.quantization_bit} for auto quantization.\")\n\n    logger.info_rank0(f\"Loading {quant_config.quantization_bit}-bit quantized model.\")\n    return QuantizationPlugin(\"bnb\")(init_kwargs, quant_config=quant_config, is_trainable=is_trainable)\n\n\n@QuantizationPlugin(\"bnb\").register()\ndef quantization_with_bnb(\n    init_kwargs: dict[str, Any],\n    quant_config: dict | BnbParams,\n    is_trainable: bool = False,\n) -> dict[str, Any]:\n    from transformers import BitsAndBytesConfig\n\n    from ...accelerator.helper import get_current_device\n    from ...utils.packages import check_version\n\n    quant_config = QuantizationPlugin.parse_params(quant_config, BnbParams)\n    quantization_bit = quant_config.quantization_bit","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/model_plugins/quantization.py#L51-L87","documentation":"The 'auto' quantization plugin dispatches to bitsandbytes and only supports 4-bit or 8-bit widths; quantization_bit values like 2, 3, or 16 are rejected immediately. This mirrors bitsandbytes' own capability, since the auto route forwards to the bnb plugin after validating the bit width.","triggerScenarios":"quantization name 'auto' (or 'bnb') with quantization_bit set to a value other than 4 or 8 in the quantization config.","commonSituations":"User tries 2-bit/3-bit extreme quantization (only available via GPTQ/AWQ-style tools, not bnb); typo such as quantization_bit: 4.0 or 44; copy-paste from a config for a different quant backend.","solutions":["Set quantization_bit to 4 or 8","For lower bit widths, use a different quantization method/plugin (e.g. a GPTQ/AWQ pipeline) or pre-quantized checkpoints","Omit quantization_bit to get the logged no-quantization path if quantization was unintentional"],"exampleFix":"# before\nquantization:\n  name: auto\n  quantization_bit: 3\n\n# after\nquantization:\n  name: auto\n  quantization_bit: 4","handlingStrategy":"validation","validationCode":"bit = quant_config.get(\"quantization_bit\")\nassert bit is None or bit in (4, 8), f\"auto quantization supports 4/8 bits, got {bit}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat {4, 8} as the only valid bnb bit widths in config schemas","Use pre-quantized GPTQ/AWQ checkpoints for lower widths"],"tags":["quantization","bitsandbytes","configuration"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}