{"record":{"id":"e7221ed2725b39de","repo":"hiyouga/LlamaFactory","slug":"reward-model-is-necessary-for-ppo-training","errorCode":null,"errorMessage":"`reward_model` is necessary for PPO training.","messagePattern":"`reward_model` is necessary for PPO training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":601,"sourceCode":"            return arg\n\n        self.freeze_trainable_modules: list[str] = split_arg(self.freeze_trainable_modules)\n        self.freeze_extra_modules: list[str] | None = split_arg(self.freeze_extra_modules)\n        self.lora_alpha: int = self.lora_alpha or self.lora_rank * 2\n        self.lora_target: list[str] = split_arg(self.lora_target)\n        self.oft_target: list[str] = split_arg(self.oft_target)\n        self.additional_target: list[str] | None = split_arg(self.additional_target)\n        self.galore_target: list[str] = split_arg(self.galore_target)\n        self.apollo_target: list[str] = split_arg(self.apollo_target)\n        self.use_ref_model = self.stage == \"dpo\" and self.pref_loss not in [\"orpo\", \"simpo\"]\n\n        assert self.finetuning_type in [\"lora\", \"oft\", \"freeze\", \"full\"], \"Invalid fine-tuning method.\"\n        assert self.ref_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.reward_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.hyper_parallel_cp_size > 0, \"`hyper_parallel_cp_size` must be greater than 0.\"\n\n        if self.stage == \"ppo\" and self.reward_model is None:\n            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")","sourceCodeStart":583,"sourceCodeEnd":619,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L583-L619","documentation":"PPO (RLHF) training needs a reward model to score generated responses; there is no intrinsic reward signal. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:601) therefore requires a non-None reward_model whenever stage is 'ppo'.","triggerScenarios":"A training config with stage: ppo but no reward_model key (or reward_model: null). Raised during finetuning-argument validation before the trainer starts.","commonSituations":"Users switching a config from stage: sft or dpo to stage: ppo without adding the reward model section. Also occurs when reward_model is set only in model_args-style YAML indentation so the parser never sees it.","solutions":["Add a pretrained reward model to the config, e.g. reward_model: OpenAssistant/reward-model-deberta-v3-large-v2.","If you have no reward model, first train one with stage: rm on preference data, then point reward_model at that checkpoint.","If you actually want preference-based training without a reward model, use stage: dpo instead."],"exampleFix":"# before (yaml)\nstage: ppo\n# no reward_model\n\n# after (yaml)\nstage: ppo\nreward_model: OpenAssistant/reward-model-deberta-v3-large-v2","handlingStrategy":"validation","validationCode":"def check_ppo_reward_model(stage: str, reward_model: str | None) -> None:\n    if stage == \"ppo\" and reward_model is None:\n        raise ValueError(\"stage=ppo requires a reward_model; train one with stage=rm first or use stage=dpo\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Add a pre-flight checklist for RLHF runs: policy model, value model, reward model, and dataset all set.","Consider DPO/ORPO when no reward model is available — they train directly on preferences."],"tags":["config","ppo","rlhf","reward-model"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}