hiyouga/LlamaFactory · error · ValueError

`reward_model_type` cannot be oft for Freeze/Full PPO traini

Error message

`reward_model_type` cannot be oft for Freeze/Full PPO training.

What it means

An OFT reward model adapter in PPO must attach to a policy whose base is shared the same way as in OFT training. If the policy uses freeze/full/lora fine-tuning, OFT reward adaptation is unsupported. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:607) rejects reward_model_type: oft unless finetuning_type is also oft.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:607

        self.oft_target: list[str] = split_arg(self.oft_target)
        self.additional_target: list[str] | None = split_arg(self.additional_target)
        self.galore_target: list[str] = split_arg(self.galore_target)
        self.apollo_target: list[str] = split_arg(self.apollo_target)
        self.use_ref_model = self.stage == "dpo" and self.pref_loss not in ["orpo", "simpo"]

        assert self.finetuning_type in ["lora", "oft", "freeze", "full"], "Invalid fine-tuning method."
        assert self.ref_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.reward_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.hyper_parallel_cp_size > 0, "`hyper_parallel_cp_size` must be greater than 0."

        if self.stage == "ppo" and self.reward_model is None:
            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:

View on GitHub (pinned to f28afaf635)

Solutions

Set finetuning_type: oft so both the policy and reward adapter use OFT.
Or export the OFT reward adapter merged into its base model and use the merged checkpoint without reward_model_type: oft.
Or switch to a reward model matching your finetuning_type (full checkpoint for freeze/full, LoRA adapter for lora).

Example fix

# before (yaml)
stage: ppo
finetuning_type: lora
reward_model: path/to/oft_rm_adapter
reward_model_type: oft

# after (yaml)
stage: ppo
finetuning_type: oft
reward_model: path/to/oft_rm_adapter
reward_model_type: oft

Defensive patterns

Strategy: validation

Validate before calling

def check_ppo_oft(finetuning_type: str, reward_model_type: str) -> None:
    if reward_model_type == "oft" and finetuning_type != "oft":
        raise ValueError("reward_model_type=oft requires finetuning_type=oft")

Prevention

Treat reward_model_type as bound to finetuning_type for adapter-based reward models (lora/oft).
Prefer merged reward-model checkpoints unless you specifically need adapter sharing.

When it happens

Trigger: stage: ppo with reward_model_type: oft and finetuning_type != oft (e.g. lora, freeze, or full).

Common situations: Reusing a LoRA or full PPO config but substituting an OFT-trained reward model adapter as the reward model.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6d5f984ad50fe997. Report an issue: GitHub.