{"record":{"id":"74735632b003996e","repo":"hiyouga/LlamaFactory","slug":"loraplus-lr-ratio-is-only-valid-for-lora-trainin","errorCode":null,"errorMessage":"`loraplus_lr_ratio` is only valid for LoRA training.","messagePattern":"`loraplus_lr_ratio` is only valid for LoRA training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":626,"sourceCode":"\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")\n\n        if self.finetuning_type != \"lora\":\n            if self.loraplus_lr_ratio is not None:\n                raise ValueError(\"`loraplus_lr_ratio` is only valid for LoRA training.\")\n\n            if self.use_rslora:\n                raise ValueError(\"`use_rslora` is only valid for LoRA training.\")\n\n            if self.use_dora:\n                raise ValueError(\"`use_dora` is only valid for LoRA training.\")\n\n            if self.pissa_init:\n                raise ValueError(\"`pissa_init` is only valid for LoRA training.\")\n\n    def to_dict(self) -> dict[str, Any]:\n        args = asdict(self)\n        args = {k: f\"<{k.upper()}>\" if k.endswith(\"api_key\") else v for k, v in args.items()}\n        return args\n","sourceCodeStart":608,"sourceCodeEnd":641,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L608-L641","documentation":"LoRA+ assigns a higher learning rate to the B matrix than the A matrix (ratio loraplus_lr_ratio), which only exists when LoRA adapters are being trained. With freeze/full fine-tuning there is no A/B split. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:626) rejects a non-None loraplus_lr_ratio when finetuning_type != lora.","triggerScenarios":"loraplus_lr_ratio: 8 (any non-null value) with finetuning_type: full or freeze in the YAML.","commonSituations":"Copying LoRA+ hyperparameters from a paper/blog into a full fine-tuning config, or switching a config from lora to full without cleaning LoRA-specific flags.","solutions":["Remove loraplus_lr_ratio if you are doing full or freeze fine-tuning.","Set finetuning_type: lora if the LoRA+ trick is what you want.","Audit the config for other LoRA-only flags (use_rslora, use_dora, pissa_init) — they fail next in the same block."],"exampleFix":"# before (yaml)\nfinetuning_type: full\nloraplus_lr_ratio: 8\n\n# after (yaml)\nfinetuning_type: full\n# loraplus_lr_ratio removed","handlingStrategy":"validation","validationCode":"def check_lora_only_flags(finetuning_type: str, **flags) -> None:\n    if finetuning_type != \"lora\":\n        for name, val in flags.items():\n            if val:\n                raise ValueError(f\"{name} is only valid for LoRA training\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Group LoRA-only flags in the YAML under a comment; delete the whole group when switching to full/freeze.","CI lint: if finetuning_type != lora, reject loraplus_lr_ratio/use_rslora/use_dora/pissa_init."],"tags":["config","lora","loraplus","optimizer"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}