{"record":{"id":"16c24830c1a2912c","repo":"hiyouga/LlamaFactory","slug":"bitsandbytes-only-accepts-4-bit-or-8-bit-quantizat","errorCode":null,"errorMessage":"Bitsandbytes only accepts 4-bit or 8-bit quantization.","messagePattern":"Bitsandbytes only accepts 4-bit or 8-bit quantization\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":192,"sourceCode":"        model_args.compute_dtype = torch.float16  # force fp16 for gptqmodel\n        logger.info_rank0(f\"Quantizing model to {model_args.export_quantization_bit} bit with GPTQModel.\")\n\n    elif model_args.quantization_bit is not None:  # on-the-fly\n        if model_args.quantization_method == QuantizationMethod.BNB:\n            if model_args.quantization_bit == 8:\n                check_version(\"bitsandbytes>=0.37.0\", mandatory=True)\n                init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(load_in_8bit=True)\n            elif model_args.quantization_bit == 4:\n                check_version(\"bitsandbytes>=0.39.0\", mandatory=True)\n                init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(\n                    load_in_4bit=True,\n                    bnb_4bit_compute_dtype=model_args.compute_dtype,\n                    bnb_4bit_use_double_quant=model_args.double_quantization,\n                    bnb_4bit_quant_type=model_args.quantization_type,\n                    bnb_4bit_quant_storage=model_args.compute_dtype,  # crucial for fsdp+qlora\n                )\n            else:\n                raise ValueError(\"Bitsandbytes only accepts 4-bit or 8-bit quantization.\")\n\n            # Do not assign device map if:\n            # 1. deepspeed zero3 or fsdp (train)\n            # 2. auto quantization device map (inference)\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == \"auto\":\n                if model_args.quantization_bit != 4:\n                    raise ValueError(\"Only 4-bit quantized model can use fsdp+qlora or auto device map.\")\n\n                check_version(\"bitsandbytes>=0.43.0\", mandatory=True)\n            else:\n                init_kwargs[\"device_map\"] = {\"\": get_current_device()}  # change auto device map for inference\n\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.\")\n        elif model_args.quantization_method == QuantizationMethod.HQQ:\n            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:\n                raise ValueError(\"HQQ only accepts 1/2/3/4/5/6/8-bit quantization.\")\n\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L174-L210","documentation":"For on-the-fly bitsandbytes quantization, LlamaFactory only builds configs for quantization_bit == 8 (load_in_8bit) and == 4 (load_in_4bit with QLoRA options). Any other bit value falls to the else and raises ValueError naming the bitsandbytes limitation.","triggerScenarios":"Setting quantization_bit to anything other than 4 or 8 while quantization_method is bitsandbytes (the default method) — e.g. quantization_bit: 2 or 16.","commonSituations":"Assuming bitsandbytes supports 2/3-bit like GPTQ/HQQ; copying export_quantization_bit values into quantization_bit; typos.","solutions":["Set quantization_bit: 4 (QLoRA) or quantization_bit: 8.","If you need lower bits, switch quantization_method: hqq (1-6/8 bits) or export-time GPTQ instead.","Confirm you are not confusing train-time quantization_bit with export-time export_quantization_bit."],"exampleFix":"# before (yaml)\nquantization_bit: 2\n\n# after (yaml)\nquantization_bit: 4\n# or: quantization_method: hqq + hqq-compatible bit width","handlingStrategy":"validation","validationCode":"if model_args.quantization_method == \"bitsandbytes\":\n    assert model_args.quantization_bit in (4, 8), (\n        \"bitsandbytes supports only 4-bit or 8-bit\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use 4 (QLoRA) or 8 for bnb; other widths belong to hqq/gptq methods.","Keep train-time quantization_bit and export-time export_quantization_bit clearly separated in configs."],"tags":["bitsandbytes","quantization","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}