{"record":{"id":"8727aead998c80b5","repo":"hiyouga/LlamaFactory","slug":"only-4-bit-quantized-model-can-use-fsdp-qlora-or-a","errorCode":null,"errorMessage":"Only 4-bit quantized model can use fsdp+qlora or auto device map.","messagePattern":"Only 4-bit quantized model can use fsdp\\+qlora or auto device map\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":199,"sourceCode":"                init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(load_in_8bit=True)\n            elif model_args.quantization_bit == 4:\n                check_version(\"bitsandbytes>=0.39.0\", mandatory=True)\n                init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(\n                    load_in_4bit=True,\n                    bnb_4bit_compute_dtype=model_args.compute_dtype,\n                    bnb_4bit_use_double_quant=model_args.double_quantization,\n                    bnb_4bit_quant_type=model_args.quantization_type,\n                    bnb_4bit_quant_storage=model_args.compute_dtype,  # crucial for fsdp+qlora\n                )\n            else:\n                raise ValueError(\"Bitsandbytes only accepts 4-bit or 8-bit quantization.\")\n\n            # Do not assign device map if:\n            # 1. deepspeed zero3 or fsdp (train)\n            # 2. auto quantization device map (inference)\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == \"auto\":\n                if model_args.quantization_bit != 4:\n                    raise ValueError(\"Only 4-bit quantized model can use fsdp+qlora or auto device map.\")\n\n                check_version(\"bitsandbytes>=0.43.0\", mandatory=True)\n            else:\n                init_kwargs[\"device_map\"] = {\"\": get_current_device()}  # change auto device map for inference\n\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.\")\n        elif model_args.quantization_method == QuantizationMethod.HQQ:\n            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:\n                raise ValueError(\"HQQ only accepts 1/2/3/4/5/6/8-bit quantization.\")\n\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():\n                raise ValueError(\"HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.\")\n\n            check_version(\"hqq\", mandatory=True)\n            init_kwargs[\"quantization_config\"] = HqqConfig(\n                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0\n            )  # use ATEN kernel (axis=0) for performance\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with HQQ.\")","sourceCodeStart":181,"sourceCodeEnd":217,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L181-L217","documentation":"When bitsandbytes quantization runs under DeepSpeed ZeRO-3/FSDP or with quantization_device_map: auto, only the 4-bit (QLoRA) path is compatible — the bnb_4bit_quant_storage sharding trick requires 4-bit. The code raises ValueError if quantization_bit != 4 in that branch, and additionally enforces bitsandbytes>=0.43.0 for sharded 4-bit.","triggerScenarios":"quantization_method: bitsandbytes with quantization_bit: 8 combined with a ZeRO-3 deepspeed config, FSDP enabled, or quantization_device_map: auto in the YAML.","commonSituations":"Multi-GPU users trying 8-bit BNB + ZeRO-3 for memory; inference setups requesting auto device mapping with 8-bit loads.","solutions":["Switch to quantization_bit: 4 (QLoRA) which is the supported sharded path.","Or disable ZeRO-3/FSDP (use ZeRO-2) and remove quantization_device_map: auto for 8-bit.","Ensure bitsandbytes>=0.43.0 once on the 4-bit sharded path."],"exampleFix":"# before (yaml)\nquantization_bit: 8\ndeepspeed: examples/deepspeed/ds_z3_config.json\n\n# after (yaml)\nquantization_bit: 4   # QLoRA + zero3 supported\ndeepspeed: examples/deepspeed/ds_z3_config.json","handlingStrategy":"validation","validationCode":"if model_args.quantization_method == \"bitsandbytes\" and (\n    is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == \"auto\"\n):\n    assert model_args.quantization_bit == 4, (\n        \"zero3/FSDP/auto device map requires 4-bit (QLoRA) bnb quantization\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Standardize on 4-bit QLoRA for sharded multi-GPU quantized training.","Remove quantization_device_map: auto from training configs; it is an inference-only convenience."],"tags":["bitsandbytes","qlora","deepspeed","fsdp","zero3","device-map"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}