{"record":{"id":"6d5f984ad50fe997","repo":"hiyouga/LlamaFactory","slug":"reward-model-type-cannot-be-oft-for-freeze-full","errorCode":null,"errorMessage":"`reward_model_type` cannot be oft for Freeze/Full PPO training.","messagePattern":"`reward_model_type` cannot be oft for Freeze/Full PPO training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":607,"sourceCode":"        self.oft_target: list[str] = split_arg(self.oft_target)\n        self.additional_target: list[str] | None = split_arg(self.additional_target)\n        self.galore_target: list[str] = split_arg(self.galore_target)\n        self.apollo_target: list[str] = split_arg(self.apollo_target)\n        self.use_ref_model = self.stage == \"dpo\" and self.pref_loss not in [\"orpo\", \"simpo\"]\n\n        assert self.finetuning_type in [\"lora\", \"oft\", \"freeze\", \"full\"], \"Invalid fine-tuning method.\"\n        assert self.ref_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.reward_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.hyper_parallel_cp_size > 0, \"`hyper_parallel_cp_size` must be greater than 0.\"\n\n        if self.stage == \"ppo\" and self.reward_model is None:\n            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")\n\n        if self.finetuning_type != \"lora\":\n            if self.loraplus_lr_ratio is not None:","sourceCodeStart":589,"sourceCodeEnd":625,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L589-L625","documentation":"An OFT reward model adapter in PPO must attach to a policy whose base is shared the same way as in OFT training. If the policy uses freeze/full/lora fine-tuning, OFT reward adaptation is unsupported. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:607) rejects reward_model_type: oft unless finetuning_type is also oft.","triggerScenarios":"stage: ppo with reward_model_type: oft and finetuning_type != oft (e.g. lora, freeze, or full).","commonSituations":"Reusing a LoRA or full PPO config but substituting an OFT-trained reward model adapter as the reward model.","solutions":["Set finetuning_type: oft so both the policy and reward adapter use OFT.","Or export the OFT reward adapter merged into its base model and use the merged checkpoint without reward_model_type: oft.","Or switch to a reward model matching your finetuning_type (full checkpoint for freeze/full, LoRA adapter for lora)."],"exampleFix":"# before (yaml)\nstage: ppo\nfinetuning_type: lora\nreward_model: path/to/oft_rm_adapter\nreward_model_type: oft\n\n# after (yaml)\nstage: ppo\nfinetuning_type: oft\nreward_model: path/to/oft_rm_adapter\nreward_model_type: oft","handlingStrategy":"validation","validationCode":"def check_ppo_oft(finetuning_type: str, reward_model_type: str) -> None:\n    if reward_model_type == \"oft\" and finetuning_type != \"oft\":\n        raise ValueError(\"reward_model_type=oft requires finetuning_type=oft\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat reward_model_type as bound to finetuning_type for adapter-based reward models (lora/oft).","Prefer merged reward-model checkpoints unless you specifically need adapter sharing."],"tags":["config","ppo","oft","reward-model"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}