{"record":{"id":"43ccd1286ea31572","repo":"hiyouga/LlamaFactory","slug":"bitsandbytes-only-accepts-4-bit-or-8-bit-quantizat-43ccd1","errorCode":null,"errorMessage":"Bitsandbytes only accepts 4-bit or 8-bit quantization.","messagePattern":"Bitsandbytes only accepts 4-bit or 8-bit quantization\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/model_plugins/quantization.py","lineNumber":92,"sourceCode":"\n@QuantizationPlugin(\"bnb\").register()\ndef quantization_with_bnb(\n    init_kwargs: dict[str, Any],\n    quant_config: dict | BnbParams,\n    is_trainable: bool = False,\n) -> dict[str, Any]:\n    from transformers import BitsAndBytesConfig\n\n    from ...accelerator.helper import get_current_device\n    from ...utils.packages import check_version\n\n    quant_config = QuantizationPlugin.parse_params(quant_config, BnbParams)\n    quantization_bit = quant_config.quantization_bit\n    if quantization_bit is None:\n        logger.warning_rank0(\"quantization_bit is not specified, default to 4-bit quantization.\")\n        quantization_bit = 4\n    if quantization_bit not in (4, 8):\n        raise ValueError(\"Bitsandbytes only accepts 4-bit or 8-bit quantization.\")\n\n    logger.info_rank0(\"Using Bitsandbytes quantization.\")\n    if quantization_bit == 8:\n        check_version(\"bitsandbytes>=0.37.0\", mandatory=True)\n        init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(load_in_8bit=True)\n    else:\n        check_version(\"bitsandbytes>=0.39.0\", mandatory=True)\n        init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(\n            load_in_4bit=True,\n            bnb_4bit_compute_dtype=quant_config.compute_dtype,\n            bnb_4bit_use_double_quant=quant_config.double_quantization,\n            bnb_4bit_quant_type=quant_config.quantization_type,\n            bnb_4bit_quant_storage=quant_config.compute_dtype,\n        )\n\n    if is_trainable:\n        logger.info_rank0(\"Detected inference mode, setting device_map for bitsandbytes quantization.\")\n        init_kwargs[\"device_map\"] = {\"\": get_current_device()}","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/model_plugins/quantization.py#L74-L110","documentation":"The bnb quantization plugin configures transformers' BitsAndBytesConfig, which only implements load_in_4bit and load_in_8bit. Any other quantization_bit is rejected before constructing the config. If quantization_bit is omitted the plugin defaults to 4-bit with a warning, so this error specifically means an explicit invalid value was supplied.","triggerScenarios":"Explicitly setting quantization_bit to a value outside {4, 8} with the 'bnb' quantization plugin.","commonSituations":"Attempting sub-4-bit quantization with bitsandbytes; mixing up quantization_bit with bits-per-component settings of other backends; stale configs from tools that accepted arbitrary bit widths.","solutions":["Use quantization_bit: 4 (QLoRA-capable) or quantization_bit: 8","For 2/3-bit, switch to a GPTQ or AWQ quantized model rather than bnb","Remove the key entirely if you want the 4-bit default"],"exampleFix":"# before\nquantization:\n  name: bnb\n  quantization_bit: 2\n\n# after\nquantization:\n  name: bnb\n  quantization_bit: 4","handlingStrategy":"validation","validationCode":"bit = quant_config.get(\"quantization_bit\", 4)\nassert bit in (4, 8), f\"bitsandbytes supports 4/8 bits only, got {bit}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Lint quantization configs against the bnb 4/8-bit contract","Omit quantization_bit when the 4-bit default is intended"],"tags":["quantization","bitsandbytes","configuration"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}