hiyouga/LlamaFactory · error · ValueError

`use_llama_pro` is only valid for Freeze or LoRA training.

Error message

`use_llama_pro` is only valid for Freeze or LoRA training.

What it means

LLaMA-Pro adds extra identity blocks to a model so training can update only the new blocks (Block Expansion). This only makes sense when the base weights stay frozen (freeze or LoRA modes); with full fine-tuning every weight is already trainable. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:613) rejects use_llama_pro with finetuning_type: full.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:613

        assert self.finetuning_type in ["lora", "oft", "freeze", "full"], "Invalid fine-tuning method."
        assert self.ref_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.reward_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.hyper_parallel_cp_size > 0, "`hyper_parallel_cp_size` must be greater than 0."

        if self.stage == "ppo" and self.reward_model is None:
            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:
                raise ValueError("`loraplus_lr_ratio` is only valid for LoRA training.")

            if self.use_rslora:
                raise ValueError("`use_rslora` is only valid for LoRA training.")

            if self.use_dora:

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove use_llama_pro: true if you intend full fine-tuning of all weights.
2. Set finetuning_type: freeze or finetuning_type: lora if you want to train only the LLaMA-Pro expanded blocks.
3. Check additional_target/train_layers config if your goal is partial-layer training.

Example fix

# before (yaml)
use_llama_pro: true
finetuning_type: full

# after (yaml)
# use_llama_pro removed
finetuning_type: full

def check_llama_pro(use_llama_pro: bool, finetuning_type: str) -> None:
    if use_llama_pro and finetuning_type == "full":
        raise ValueError("use_llama_pro requires finetuning_type in (freeze, lora)")

When it happens

Trigger: A config with use_llama_pro: true and finetuning_type: full. Raised during argument validation.

Common situations: Users who previously expanded a model with LLaMA-Pro (creating new trainable blocks) then switching the config to full fine-tuning of all parameters, forgetting the expansion flag is still set.

Related errors

• Megatron Bridge only supports `full` and `lora` finetuning.
• Unknown finetuning type: {finetuning_args.finetuning_type}.
• Please upgrade `transformers` to 4.34.0
• Unable to process key {key}
• bf16 and fp16 cannot be both True.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c80f7d0af29a63a2. Report an issue: GitHub.