hiyouga/LlamaFactory · error · ValueError

Only 4-bit quantized model can use fsdp+qlora or auto device

Error message

Only 4-bit quantized model can use fsdp+qlora or auto device map.

What it means

In the bnb plugin, the non-trainable path (is_trainable=False) enforces 4-bit, because only 4-bit QLoRA is compatible with FSDP gathering / auto device map in this code path. NOTE: the surrounding code's log messages are swapped ('Detected inference mode' prints under is_trainable=True and vice versa), but the check itself fires when the model is loaded for inference/export with quantization_bit=8. So 8-bit + is_trainable=False is the actual trigger.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/quantization.py:114

        check_version("bitsandbytes>=0.37.0", mandatory=True)
        init_kwargs["quantization_config"] = BitsAndBytesConfig(load_in_8bit=True)
    else:
        check_version("bitsandbytes>=0.39.0", mandatory=True)
        init_kwargs["quantization_config"] = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=quant_config.compute_dtype,
            bnb_4bit_use_double_quant=quant_config.double_quantization,
            bnb_4bit_quant_type=quant_config.quantization_type,
            bnb_4bit_quant_storage=quant_config.compute_dtype,
        )

    if is_trainable:
        logger.info_rank0("Detected inference mode, setting device_map for bitsandbytes quantization.")
        init_kwargs["device_map"] = {"": get_current_device()}
    else:
        logger.info_rank0("Detected training mode, skip setting device_map for bitsandbytes quantization.")
        if quantization_bit != 4:
            raise ValueError("Only 4-bit quantized model can use fsdp+qlora or auto device map.")
        check_version("bitsandbytes>=0.43.0", mandatory=True)

    logger.info_rank0(f"Quantizing model to {quantization_bit} bit with bitsandbytes.")
    return init_kwargs

View on GitHub (pinned to f28afaf635)

Solutions

1. Switch the export/inference run to quantization_bit: 4, or drop bnb quantization for the merged export (merging into fp16/bf16 is the usual route)
2. If you need 8-bit inference, load without the bnb plugin path that enforces this rule (plain load_in_8bit via transformers)
3. Report/fix the swapped is_trainable log branches upstream so the message matches the actual mode

Example fix

# before (export run)
quantization:
  name: bnb
  quantization_bit: 8

# after (export run)
quantization: null  # merge in native dtype
# or
quantization:
  name: bnb
  quantization_bit: 4

if not is_trainable:
    assert quant_config.get("quantization_bit", 4) == 4, "non-trainable bnb loads must be 4-bit in this path; drop quantization for export"

When it happens

Trigger: Loading a model with bnb 8-bit quantization where is_trainable is False (inference/export load), i.e. the else-branch of the is_trainable check with quantization_bit == 8.

Common situations: User trains successfully with 8-bit LoRA, then runs export or chat/inference with the same quantization config; the inverted log text makes the resulting error message ('fsdp+qlora') misleading in inference contexts.

Related errors

• Quantization dataset is necessary for exporting.
• Quantization is only compatible with the LoRA or OFT method.
• Cannot use device map for quantized models in training.
• Quantized models can only be used for the LoRA or OFT tuning
• Cannot initialize PiSSA adapter on quantized models.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f028e356d710094e. Report an issue: GitHub.