{"record":{"id":"f028e356d710094e","repo":"hiyouga/LlamaFactory","slug":"only-4-bit-quantized-model-can-use-fsdp-qlora-or-a-f028e3","errorCode":null,"errorMessage":"Only 4-bit quantized model can use fsdp+qlora or auto device map.","messagePattern":"Only 4-bit quantized model can use fsdp\\+qlora or auto device map\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/model_plugins/quantization.py","lineNumber":114,"sourceCode":"        check_version(\"bitsandbytes>=0.37.0\", mandatory=True)\n        init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(load_in_8bit=True)\n    else:\n        check_version(\"bitsandbytes>=0.39.0\", mandatory=True)\n        init_kwargs[\"quantization_config\"] = BitsAndBytesConfig(\n            load_in_4bit=True,\n            bnb_4bit_compute_dtype=quant_config.compute_dtype,\n            bnb_4bit_use_double_quant=quant_config.double_quantization,\n            bnb_4bit_quant_type=quant_config.quantization_type,\n            bnb_4bit_quant_storage=quant_config.compute_dtype,\n        )\n\n    if is_trainable:\n        logger.info_rank0(\"Detected inference mode, setting device_map for bitsandbytes quantization.\")\n        init_kwargs[\"device_map\"] = {\"\": get_current_device()}\n    else:\n        logger.info_rank0(\"Detected training mode, skip setting device_map for bitsandbytes quantization.\")\n        if quantization_bit != 4:\n            raise ValueError(\"Only 4-bit quantized model can use fsdp+qlora or auto device map.\")\n        check_version(\"bitsandbytes>=0.43.0\", mandatory=True)\n\n    logger.info_rank0(f\"Quantizing model to {quantization_bit} bit with bitsandbytes.\")\n    return init_kwargs\n","sourceCodeStart":96,"sourceCodeEnd":119,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/model_plugins/quantization.py#L96-L119","documentation":"In the bnb plugin, the non-trainable path (is_trainable=False) enforces 4-bit, because only 4-bit QLoRA is compatible with FSDP gathering / auto device map in this code path. NOTE: the surrounding code's log messages are swapped ('Detected inference mode' prints under is_trainable=True and vice versa), but the check itself fires when the model is loaded for inference/export with quantization_bit=8. So 8-bit + is_trainable=False is the actual trigger.","triggerScenarios":"Loading a model with bnb 8-bit quantization where is_trainable is False (inference/export load), i.e. the else-branch of the is_trainable check with quantization_bit == 8.","commonSituations":"User trains successfully with 8-bit LoRA, then runs export or chat/inference with the same quantization config; the inverted log text makes the resulting error message ('fsdp+qlora') misleading in inference contexts.","solutions":["Switch the export/inference run to quantization_bit: 4, or drop bnb quantization for the merged export (merging into fp16/bf16 is the usual route)","If you need 8-bit inference, load without the bnb plugin path that enforces this rule (plain load_in_8bit via transformers)","Report/fix the swapped is_trainable log branches upstream so the message matches the actual mode"],"exampleFix":"# before (export run)\nquantization:\n  name: bnb\n  quantization_bit: 8\n\n# after (export run)\nquantization: null  # merge in native dtype\n# or\nquantization:\n  name: bnb\n  quantization_bit: 4","handlingStrategy":"validation","validationCode":"if not is_trainable:\n    assert quant_config.get(\"quantization_bit\", 4) == 4, \"non-trainable bnb loads must be 4-bit in this path; drop quantization for export\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use separate YAMLs for training (8-bit ok) and export (unquantized or 4-bit)","Note the is_trainable log messages in this region are swapped; trust the branch, not the log text","Merge LoRA into the base model in native dtype for exports"],"tags":["quantization","bitsandbytes","qlora","export","bug"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}