{"record":{"id":"4d0da05fab6fdb21","repo":"hiyouga/LlamaFactory","slug":"reward-model-type-cannot-be-lora-for-freeze-full","errorCode":null,"errorMessage":"`reward_model_type` cannot be lora for Freeze/Full PPO training.","messagePattern":"`reward_model_type` cannot be lora for Freeze/Full PPO training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":604,"sourceCode":"        self.freeze_extra_modules: list[str] | None = split_arg(self.freeze_extra_modules)\n        self.lora_alpha: int = self.lora_alpha or self.lora_rank * 2\n        self.lora_target: list[str] = split_arg(self.lora_target)\n        self.oft_target: list[str] = split_arg(self.oft_target)\n        self.additional_target: list[str] | None = split_arg(self.additional_target)\n        self.galore_target: list[str] = split_arg(self.galore_target)\n        self.apollo_target: list[str] = split_arg(self.apollo_target)\n        self.use_ref_model = self.stage == \"dpo\" and self.pref_loss not in [\"orpo\", \"simpo\"]\n\n        assert self.finetuning_type in [\"lora\", \"oft\", \"freeze\", \"full\"], \"Invalid fine-tuning method.\"\n        assert self.ref_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.reward_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.hyper_parallel_cp_size > 0, \"`hyper_parallel_cp_size` must be greater than 0.\"\n\n        if self.stage == \"ppo\" and self.reward_model is None:\n            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")","sourceCodeStart":586,"sourceCodeEnd":622,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L586-L622","documentation":"A LoRA reward model in PPO is loaded by attaching the adapter to the policy model's weights. When the policy is trained with freeze or full fine-tuning, the LoRA adapter's base weights no longer match a separable frozen actor, so this combination is unsupported. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:604) rejects reward_model_type: lora unless finetuning_type is also lora.","triggerScenarios":"stage: ppo with reward_model_type: lora and finetuning_type set to freeze or full (e.g. a config copied from a full-RLHF recipe that then swaps in a LoRA reward model adapter path).","commonSituations":"Mixing recipes: taking a full-parameter PPO config and pointing reward_model at a LoRA adapter checkpoint (reward_model_type: lora) without changing finetuning_type.","solutions":["Set finetuning_type: lora so the policy and the LoRA reward adapter share the same frozen base.","Or use a merged/full reward model checkpoint and remove reward_model_type: lora (or set it to the matching type).","Merge the LoRA reward adapter into its base model (llamafactory-cli export) and reference the merged checkpoint as reward_model."],"exampleFix":"# before (yaml)\nstage: ppo\nfinetuning_type: full\nreward_model: path/to/lora_rm_adapter\nreward_model_type: lora\n\n# after (yaml)\nstage: ppo\nfinetuning_type: lora\nreward_model: path/to/lora_rm_adapter\nreward_model_type: lora","handlingStrategy":"validation","validationCode":"def check_ppo_rm_type(finetuning_type: str, reward_model_type: str) -> None:\n    if reward_model_type == \"lora\" and finetuning_type != \"lora\":\n        raise ValueError(\"reward_model_type=lora requires finetuning_type=lora\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep adapter types consistent across the PPO stack: LoRA policy pairs with LoRA reward adapter on the same base.","Merge adapters (llamafactory-cli export) when you need to mix full training with an adapter reward model."],"tags":["config","ppo","lora","reward-model"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}