{"record":{"id":"0910590b3133805d","repo":"hiyouga/LlamaFactory","slug":"cannot-use-lora-with-galore-apollo-or-badam-toget","errorCode":null,"errorMessage":"Cannot use LoRA with GaLore, APOLLO or BAdam together.","messagePattern":"Cannot use LoRA with GaLore, APOLLO or BAdam together\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":616,"sourceCode":"        assert self.hyper_parallel_cp_size > 0, \"`hyper_parallel_cp_size` must be greater than 0.\"\n\n        if self.stage == \"ppo\" and self.reward_model is None:\n            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")\n\n        if self.finetuning_type != \"lora\":\n            if self.loraplus_lr_ratio is not None:\n                raise ValueError(\"`loraplus_lr_ratio` is only valid for LoRA training.\")\n\n            if self.use_rslora:\n                raise ValueError(\"`use_rslora` is only valid for LoRA training.\")\n\n            if self.use_dora:\n                raise ValueError(\"`use_dora` is only valid for LoRA training.\")\n\n            if self.pissa_init:","sourceCodeStart":598,"sourceCodeEnd":634,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L598-L634","documentation":"GaLore, APOLLO, and BAdam are full-parameter training optimizers that project/update the base weights; LoRA freezes the base and trains low-rank adapters, which conflicts with those optimizers at the implementation level. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:616) forbids combining finetuning_type: lora with use_galore, use_apollo, or use_badam.","triggerScenarios":"A config with finetuning_type: lora and at least one of use_galore: true, use_apollo: true, or use_badam: true.","commonSituations":"Memory-constrained users stacking every memory-saving flag (LoRA plus GaLore) hoping they compose. Also copy-paste from optimizer-comparison configs.","solutions":["Choose one memory-efficient strategy: keep finetuning_type: lora and remove use_galore/use_apollo/use_badam.","Or keep GaLore/APOLLO/BAdam and set finetuning_type: full (these optimizers are designed for full training within limited VRAM).","If VRAM is the constraint, prefer LoRA with a higher lora_rank or 4-bit quantization (quantization_bit) rather than mixing methods."],"exampleFix":"# before (yaml)\nfinetuning_type: lora\nuse_galore: true\n\n# after (yaml)\nfinetuning_type: full\nuse_galore: true  # GaLore does memory-efficient FULL fine-tuning","handlingStrategy":"validation","validationCode":"def check_lora_optimizers(finetuning_type: str, use_galore: bool, use_apollo: bool, use_badam: bool) -> None:\n    if finetuning_type == \"lora\" and (use_galore or use_apollo or use_badam):\n        raise ValueError(\"LoRA cannot be combined with GaLore/APOLLO/BAdam; pick one strategy\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Classify memory-saving techniques as either adapter-based (LoRA/DoRA/rsLoRA) or optimizer-based (GaLore/APOLLO/BAdam) and pick one class.","For tight VRAM, combine LoRA with quantization (quantization_bit: 4), not with projection optimizers."],"tags":["config","lora","galore","apollo","badam","optimizer"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}