hiyouga/LlamaFactory · error · ValueError

`reward_model_type` cannot be lora for Freeze/Full PPO train

Error message

`reward_model_type` cannot be lora for Freeze/Full PPO training.

What it means

A LoRA reward model in PPO is loaded by attaching the adapter to the policy model's weights. When the policy is trained with freeze or full fine-tuning, the LoRA adapter's base weights no longer match a separable frozen actor, so this combination is unsupported. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:604) rejects reward_model_type: lora unless finetuning_type is also lora.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:604

        self.freeze_extra_modules: list[str] | None = split_arg(self.freeze_extra_modules)
        self.lora_alpha: int = self.lora_alpha or self.lora_rank * 2
        self.lora_target: list[str] = split_arg(self.lora_target)
        self.oft_target: list[str] = split_arg(self.oft_target)
        self.additional_target: list[str] | None = split_arg(self.additional_target)
        self.galore_target: list[str] = split_arg(self.galore_target)
        self.apollo_target: list[str] = split_arg(self.apollo_target)
        self.use_ref_model = self.stage == "dpo" and self.pref_loss not in ["orpo", "simpo"]

        assert self.finetuning_type in ["lora", "oft", "freeze", "full"], "Invalid fine-tuning method."
        assert self.ref_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.reward_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.hyper_parallel_cp_size > 0, "`hyper_parallel_cp_size` must be greater than 0."

        if self.stage == "ppo" and self.reward_model is None:
            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

View on GitHub (pinned to f28afaf635)

Solutions

Set finetuning_type: lora so the policy and the LoRA reward adapter share the same frozen base.
Or use a merged/full reward model checkpoint and remove reward_model_type: lora (or set it to the matching type).
Merge the LoRA reward adapter into its base model (llamafactory-cli export) and reference the merged checkpoint as reward_model.

Example fix

# before (yaml)
stage: ppo
finetuning_type: full
reward_model: path/to/lora_rm_adapter
reward_model_type: lora

# after (yaml)
stage: ppo
finetuning_type: lora
reward_model: path/to/lora_rm_adapter
reward_model_type: lora

Defensive patterns

Strategy: validation

Validate before calling

def check_ppo_rm_type(finetuning_type: str, reward_model_type: str) -> None:
    if reward_model_type == "lora" and finetuning_type != "lora":
        raise ValueError("reward_model_type=lora requires finetuning_type=lora")

Prevention

Keep adapter types consistent across the PPO stack: LoRA policy pairs with LoRA reward adapter on the same base.
Merge adapters (llamafactory-cli export) when you need to mix full training with an adapter reward model.

When it happens

Trigger: stage: ppo with reward_model_type: lora and finetuning_type set to freeze or full (e.g. a config copied from a full-RLHF recipe that then swaps in a LoRA reward model adapter path).

Common situations: Mixing recipes: taking a full-parameter PPO config and pointing reward_model at a LoRA adapter checkpoint (reward_model_type: lora) without changing finetuning_type.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/4d0da05fab6fdb21. Report an issue: GitHub.