hiyouga/LlamaFactory · error · ValueError

Only 4-bit quantized model can use fsdp+qlora or auto device

Error message

Only 4-bit quantized model can use fsdp+qlora or auto device map.

What it means

When bitsandbytes quantization runs under DeepSpeed ZeRO-3/FSDP or with quantization_device_map: auto, only the 4-bit (QLoRA) path is compatible — the bnb_4bit_quant_storage sharding trick requires 4-bit. The code raises ValueError if quantization_bit != 4 in that branch, and additionally enforces bitsandbytes>=0.43.0 for sharded 4-bit.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:199

                init_kwargs["quantization_config"] = BitsAndBytesConfig(load_in_8bit=True)
            elif model_args.quantization_bit == 4:
                check_version("bitsandbytes>=0.39.0", mandatory=True)
                init_kwargs["quantization_config"] = BitsAndBytesConfig(
                    load_in_4bit=True,
                    bnb_4bit_compute_dtype=model_args.compute_dtype,
                    bnb_4bit_use_double_quant=model_args.double_quantization,
                    bnb_4bit_quant_type=model_args.quantization_type,
                    bnb_4bit_quant_storage=model_args.compute_dtype,  # crucial for fsdp+qlora
                )
            else:
                raise ValueError("Bitsandbytes only accepts 4-bit or 8-bit quantization.")

            # Do not assign device map if:
            # 1. deepspeed zero3 or fsdp (train)
            # 2. auto quantization device map (inference)
            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == "auto":
                if model_args.quantization_bit != 4:
                    raise ValueError("Only 4-bit quantized model can use fsdp+qlora or auto device map.")

                check_version("bitsandbytes>=0.43.0", mandatory=True)
            else:
                init_kwargs["device_map"] = {"": get_current_device()}  # change auto device map for inference

            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.")
        elif model_args.quantization_method == QuantizationMethod.HQQ:
            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:
                raise ValueError("HQQ only accepts 1/2/3/4/5/6/8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("hqq", mandatory=True)
            init_kwargs["quantization_config"] = HqqConfig(
                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0
            )  # use ATEN kernel (axis=0) for performance
            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with HQQ.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Switch to quantization_bit: 4 (QLoRA) which is the supported sharded path.
2. Or disable ZeRO-3/FSDP (use ZeRO-2) and remove quantization_device_map: auto for 8-bit.
3. Ensure bitsandbytes>=0.43.0 once on the 4-bit sharded path.

Example fix

# before (yaml)
quantization_bit: 8
deepspeed: examples/deepspeed/ds_z3_config.json

# after (yaml)
quantization_bit: 4   # QLoRA + zero3 supported
deepspeed: examples/deepspeed/ds_z3_config.json

if model_args.quantization_method == "bitsandbytes" and (
    is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == "auto"
):
    assert model_args.quantization_bit == 4, (
        "zero3/FSDP/auto device map requires 4-bit (QLoRA) bnb quantization"
    )

When it happens

Trigger: quantization_method: bitsandbytes with quantization_bit: 8 combined with a ZeRO-3 deepspeed config, FSDP enabled, or quantization_device_map: auto in the YAML.

Common situations: Multi-GPU users trying 8-bit BNB + ZeRO-3 for memory; inference setups requesting auto device mapping with 8-bit loads.

Related errors

• DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized
• HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FS
• EETQ quantization is incompatible with DeepSpeed ZeRO-3 or F
• Cannot use device map for quantized models in training.
• KTransformers is incompatible with DeepSpeed ZeRO-3.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/8727aead998c80b5. Report an issue: GitHub.