hiyouga/LlamaFactory · error · ValueError

`loraplus_lr_ratio` is only valid for LoRA training.

Error message

`loraplus_lr_ratio` is only valid for LoRA training.

What it means

LoRA+ assigns a higher learning rate to the B matrix than the A matrix (ratio loraplus_lr_ratio), which only exists when LoRA adapters are being trained. With freeze/full fine-tuning there is no A/B split. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:626) rejects a non-None loraplus_lr_ratio when finetuning_type != lora.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:626

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:
                raise ValueError("`loraplus_lr_ratio` is only valid for LoRA training.")

            if self.use_rslora:
                raise ValueError("`use_rslora` is only valid for LoRA training.")

            if self.use_dora:
                raise ValueError("`use_dora` is only valid for LoRA training.")

            if self.pissa_init:
                raise ValueError("`pissa_init` is only valid for LoRA training.")

    def to_dict(self) -> dict[str, Any]:
        args = asdict(self)
        args = {k: f"<{k.upper()}>" if k.endswith("api_key") else v for k, v in args.items()}
        return args

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove loraplus_lr_ratio if you are doing full or freeze fine-tuning.
2. Set finetuning_type: lora if the LoRA+ trick is what you want.
3. Audit the config for other LoRA-only flags (use_rslora, use_dora, pissa_init) — they fail next in the same block.

Example fix

# before (yaml)
finetuning_type: full
loraplus_lr_ratio: 8

# after (yaml)
finetuning_type: full
# loraplus_lr_ratio removed

def check_lora_only_flags(finetuning_type: str, **flags) -> None:
    if finetuning_type != "lora":
        for name, val in flags.items():
            if val:
                raise ValueError(f"{name} is only valid for LoRA training")

When it happens

Trigger: loraplus_lr_ratio: 8 (any non-null value) with finetuning_type: full or freeze in the YAML.

Common situations: Copying LoRA+ hyperparameters from a paper/blog into a full fine-tuning config, or switching a config from lora to full without cleaning LoRA-specific flags.

Related errors

• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• `reward_model_type` cannot be lora for Freeze/Full PPO train
• Cannot use GaLore, APOLLO or BAdam together.
• Cannot use PiSSA for current training stage.
• `use_rslora` is only valid for LoRA training.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/74735632b003996e. Report an issue: GitHub.