{"record":{"id":"e37a3a364063a1be","repo":"hiyouga/LlamaFactory","slug":"cannot-use-galore-apollo-or-badam-together","errorCode":null,"errorMessage":"Cannot use GaLore, APOLLO or BAdam together.","messagePattern":"Cannot use GaLore, APOLLO or BAdam together\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":619,"sourceCode":"            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")\n\n        if self.finetuning_type != \"lora\":\n            if self.loraplus_lr_ratio is not None:\n                raise ValueError(\"`loraplus_lr_ratio` is only valid for LoRA training.\")\n\n            if self.use_rslora:\n                raise ValueError(\"`use_rslora` is only valid for LoRA training.\")\n\n            if self.use_dora:\n                raise ValueError(\"`use_dora` is only valid for LoRA training.\")\n\n            if self.pissa_init:\n                raise ValueError(\"`pissa_init` is only valid for LoRA training.\")\n\n    def to_dict(self) -> dict[str, Any]:","sourceCodeStart":601,"sourceCodeEnd":637,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L601-L637","documentation":"GaLore, APOLLO, and BAdam each replace the standard optimizer update rule, so at most one can be active. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:619) counts the enabled flags (int(use_galore) + int(use_apollo) + use_badam) and raises if more than one is set. Note the third term is not wrapped in int(), so a non-boolean truthy string for use_badam also trips this check.","triggerScenarios":"A config enabling two or more of use_galore: true, use_apollo: true, use_badam: true (e.g. GaLore for attention layers plus BAdam elsewhere is not supported).","commonSituations":"Users reading that each optimizer saves memory and enabling several at once; or benchmark configs where multiple optimizer flags were toggled during experiments and not reset.","solutions":["Keep exactly one of use_galore / use_apollo / use_badam and set the others to false or remove them.","Pick the one matching your need: GaLore/APOLLO for projected full training, BAdam for block-wise updates.","If you need layer-specific optimizers, that is not supported by these flags — file a feature request instead of combining."],"exampleFix":"# before (yaml)\nuse_galore: true\nuse_badam: true\n\n# after (yaml)\nuse_galore: true\n# use_badam removed","handlingStrategy":"validation","validationCode":"def check_single_projection_optimizer(use_galore: bool, use_apollo: bool, use_badam: bool) -> None:\n    if int(use_galore) + int(use_apollo) + int(use_badam) > 1:\n        raise ValueError(\"Enable at most one of use_galore / use_apollo / use_badam\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep exactly one experimental-optimizer flag per config; reset the others to false explicitly.","Use a config linter that asserts mutual exclusion for known-incompatible boolean flags."],"tags":["config","galore","apollo","badam","optimizer"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}