hiyouga/LlamaFactory · error · ValueError

Cannot use GaLore, APOLLO or BAdam together.

Error message

Cannot use GaLore, APOLLO or BAdam together.

What it means

GaLore, APOLLO, and BAdam each replace the standard optimizer update rule, so at most one can be active. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:619) counts the enabled flags (int(use_galore) + int(use_apollo) + use_badam) and raises if more than one is set. Note the third term is not wrapped in int(), so a non-boolean truthy string for use_badam also trips this check.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:619

            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:
                raise ValueError("`loraplus_lr_ratio` is only valid for LoRA training.")

            if self.use_rslora:
                raise ValueError("`use_rslora` is only valid for LoRA training.")

            if self.use_dora:
                raise ValueError("`use_dora` is only valid for LoRA training.")

            if self.pissa_init:
                raise ValueError("`pissa_init` is only valid for LoRA training.")

    def to_dict(self) -> dict[str, Any]:

View on GitHub (pinned to f28afaf635)

Solutions

1. Keep exactly one of use_galore / use_apollo / use_badam and set the others to false or remove them.
2. Pick the one matching your need: GaLore/APOLLO for projected full training, BAdam for block-wise updates.
3. If you need layer-specific optimizers, that is not supported by these flags — file a feature request instead of combining.

Example fix

# before (yaml)
use_galore: true
use_badam: true

# after (yaml)
use_galore: true
# use_badam removed

def check_single_projection_optimizer(use_galore: bool, use_apollo: bool, use_badam: bool) -> None:
    if int(use_galore) + int(use_apollo) + int(use_badam) > 1:
        raise ValueError("Enable at most one of use_galore / use_apollo / use_badam")

When it happens

Trigger: A config enabling two or more of use_galore: true, use_apollo: true, use_badam: true (e.g. GaLore for attention layers plus BAdam elsewhere is not supported).

Common situations: Users reading that each optimizer saves memory and enabling several at once; or benchmark configs where multiple optimizer flags were toggled during experiments and not reset.

Related errors

• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• GaLore and APOLLO are incompatible with DeepSpeed yet.
• Unknown optim: {training_args.optim}.
• `loraplus_lr_ratio` is only valid for LoRA training.
• Distributed training does not support layer-wise GaLore.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e37a3a364063a1be. Report an issue: GitHub.