hiyouga/LlamaFactory · error · ValueError

Cannot use LoRA with GaLore, APOLLO or BAdam together.

Error message

Cannot use LoRA with GaLore, APOLLO or BAdam together.

What it means

GaLore, APOLLO, and BAdam are full-parameter training optimizers that project/update the base weights; LoRA freezes the base and trains low-rank adapters, which conflicts with those optimizers at the implementation level. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:616) forbids combining finetuning_type: lora with use_galore, use_apollo, or use_badam.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:616

        assert self.hyper_parallel_cp_size > 0, "`hyper_parallel_cp_size` must be greater than 0."

        if self.stage == "ppo" and self.reward_model is None:
            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:
                raise ValueError("`loraplus_lr_ratio` is only valid for LoRA training.")

            if self.use_rslora:
                raise ValueError("`use_rslora` is only valid for LoRA training.")

            if self.use_dora:
                raise ValueError("`use_dora` is only valid for LoRA training.")

            if self.pissa_init:

View on GitHub (pinned to f28afaf635)

Solutions

1. Choose one memory-efficient strategy: keep finetuning_type: lora and remove use_galore/use_apollo/use_badam.
2. Or keep GaLore/APOLLO/BAdam and set finetuning_type: full (these optimizers are designed for full training within limited VRAM).
3. If VRAM is the constraint, prefer LoRA with a higher lora_rank or 4-bit quantization (quantization_bit) rather than mixing methods.

Example fix

# before (yaml)
finetuning_type: lora
use_galore: true

# after (yaml)
finetuning_type: full
use_galore: true  # GaLore does memory-efficient FULL fine-tuning

def check_lora_optimizers(finetuning_type: str, use_galore: bool, use_apollo: bool, use_badam: bool) -> None:
    if finetuning_type == "lora" and (use_galore or use_apollo or use_badam):
        raise ValueError("LoRA cannot be combined with GaLore/APOLLO/BAdam; pick one strategy")

When it happens

Trigger: A config with finetuning_type: lora and at least one of use_galore: true, use_apollo: true, or use_badam: true.

Common situations: Memory-constrained users stacking every memory-saving flag (LoRA plus GaLore) hoping they compose. Also copy-paste from optimizer-comparison configs.

Related errors

• Cannot use GaLore, APOLLO or BAdam together.
• `loraplus_lr_ratio` is only valid for LoRA training.
• GaLore and APOLLO are incompatible with DeepSpeed yet.
• Unknown optim: {training_args.optim}.
• `reward_model_type` cannot be lora for Freeze/Full PPO train

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0910590b3133805d. Report an issue: GitHub.