What causes "`reward_model` is necessary for PPO training." in hiyouga/LlamaFactory?

Add a pretrained reward model to the config, e.g. reward_model: OpenAssistant/reward-model-deberta-v3-large-v2.

hiyouga/LlamaFactory · critical · ValueError

`reward_model` is necessary for PPO training.

Error message

`reward_model` is necessary for PPO training.

What it means

PPO (RLHF) training needs a reward model to score generated responses; there is no intrinsic reward signal. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:601) therefore requires a non-None reward_model whenever stage is 'ppo'.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:601

            return arg

        self.freeze_trainable_modules: list[str] = split_arg(self.freeze_trainable_modules)
        self.freeze_extra_modules: list[str] | None = split_arg(self.freeze_extra_modules)
        self.lora_alpha: int = self.lora_alpha or self.lora_rank * 2
        self.lora_target: list[str] = split_arg(self.lora_target)
        self.oft_target: list[str] = split_arg(self.oft_target)
        self.additional_target: list[str] | None = split_arg(self.additional_target)
        self.galore_target: list[str] = split_arg(self.galore_target)
        self.apollo_target: list[str] = split_arg(self.apollo_target)
        self.use_ref_model = self.stage == "dpo" and self.pref_loss not in ["orpo", "simpo"]

        assert self.finetuning_type in ["lora", "oft", "freeze", "full"], "Invalid fine-tuning method."
        assert self.ref_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.reward_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.hyper_parallel_cp_size > 0, "`hyper_parallel_cp_size` must be greater than 0."

        if self.stage == "ppo" and self.reward_model is None:
            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

View on GitHub (pinned to f28afaf635)

Solutions

Add a pretrained reward model to the config, e.g. reward_model: OpenAssistant/reward-model-deberta-v3-large-v2.
If you have no reward model, first train one with stage: rm on preference data, then point reward_model at that checkpoint.
If you actually want preference-based training without a reward model, use stage: dpo instead.

Example fix

# before (yaml)
stage: ppo
# no reward_model

# after (yaml)
stage: ppo
reward_model: OpenAssistant/reward-model-deberta-v3-large-v2

Defensive patterns

Strategy: validation

Validate before calling

def check_ppo_reward_model(stage: str, reward_model: str | None) -> None:
    if stage == "ppo" and reward_model is None:
        raise ValueError("stage=ppo requires a reward_model; train one with stage=rm first or use stage=dpo")

Prevention

Add a pre-flight checklist for RLHF runs: policy model, value model, reward model, and dataset all set.
Consider DPO/ORPO when no reward model is available — they train directly on preferences.

When it happens

Trigger: A training config with stage: ppo but no reward_model key (or reward_model: null). Raised during finetuning-argument validation before the trainer starts.

Common situations: Users switching a config from stage: sft or dpo to stage: ppo without adding the reward model section. Also occurs when reward_model is set only in model_args-style YAML indentation so the parser never sees it.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e7221ed2725b39de. Report an issue: GitHub.