{"record":{"id":"c80f7d0af29a63a2","repo":"hiyouga/LlamaFactory","slug":"use-llama-pro-is-only-valid-for-freeze-or-lora-t","errorCode":null,"errorMessage":"`use_llama_pro` is only valid for Freeze or LoRA training.","messagePattern":"`use_llama_pro` is only valid for Freeze or LoRA training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":613,"sourceCode":"        assert self.finetuning_type in [\"lora\", \"oft\", \"freeze\", \"full\"], \"Invalid fine-tuning method.\"\n        assert self.ref_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.reward_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.hyper_parallel_cp_size > 0, \"`hyper_parallel_cp_size` must be greater than 0.\"\n\n        if self.stage == \"ppo\" and self.reward_model is None:\n            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")\n\n        if self.finetuning_type != \"lora\":\n            if self.loraplus_lr_ratio is not None:\n                raise ValueError(\"`loraplus_lr_ratio` is only valid for LoRA training.\")\n\n            if self.use_rslora:\n                raise ValueError(\"`use_rslora` is only valid for LoRA training.\")\n\n            if self.use_dora:","sourceCodeStart":595,"sourceCodeEnd":631,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L595-L631","documentation":"LLaMA-Pro adds extra identity blocks to a model so training can update only the new blocks (Block Expansion). This only makes sense when the base weights stay frozen (freeze or LoRA modes); with full fine-tuning every weight is already trainable. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:613) rejects use_llama_pro with finetuning_type: full.","triggerScenarios":"A config with use_llama_pro: true and finetuning_type: full. Raised during argument validation.","commonSituations":"Users who previously expanded a model with LLaMA-Pro (creating new trainable blocks) then switching the config to full fine-tuning of all parameters, forgetting the expansion flag is still set.","solutions":["Remove use_llama_pro: true if you intend full fine-tuning of all weights.","Set finetuning_type: freeze or finetuning_type: lora if you want to train only the LLaMA-Pro expanded blocks.","Check additional_target/train_layers config if your goal is partial-layer training."],"exampleFix":"# before (yaml)\nuse_llama_pro: true\nfinetuning_type: full\n\n# after (yaml)\n# use_llama_pro removed\nfinetuning_type: full","handlingStrategy":"validation","validationCode":"def check_llama_pro(use_llama_pro: bool, finetuning_type: str) -> None:\n    if use_llama_pro and finetuning_type == \"full\":\n        raise ValueError(\"use_llama_pro requires finetuning_type in (freeze, lora)\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Remember LLaMA-Pro means block expansion: the base stays frozen and only new blocks train.","When switching finetuning_type, grep the config for method-specific flags."],"tags":["config","llama-pro","finetuning-type"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}