hiyouga/LlamaFactory · error · ValueError

`use_rslora` is only valid for LoRA training.

Error message

`use_rslora` is only valid for LoRA training.

What it means

Rank-Stabilized LoRA (rsLoRA) rescales the adapter update by sqrt(rank) instead of 1/rank; the scaling only applies to LoRA adapters. FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:629) rejects use_rslora: true when finetuning_type is not lora.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:629

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:
                raise ValueError("`loraplus_lr_ratio` is only valid for LoRA training.")

            if self.use_rslora:
                raise ValueError("`use_rslora` is only valid for LoRA training.")

            if self.use_dora:
                raise ValueError("`use_dora` is only valid for LoRA training.")

            if self.pissa_init:
                raise ValueError("`pissa_init` is only valid for LoRA training.")

    def to_dict(self) -> dict[str, Any]:
        args = asdict(self)
        args = {k: f"<{k.upper()}>" if k.endswith("api_key") else v for k, v in args.items()}
        return args

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove use_rslora: true for full/freeze training.
2. Set finetuning_type: lora to keep using rsLoRA.
3. If high-rank adaptation was the goal, use LoRA with a larger lora_rank plus use_rslora.

Example fix

# before (yaml)
finetuning_type: full
use_rslora: true

# after (yaml)
finetuning_type: full
# use_rslora removed

def check_rslora(finetuning_type: str, use_rslora: bool) -> None:
    if use_rslora and finetuning_type != "lora":
        raise ValueError("use_rslora requires finetuning_type=lora")

When it happens

Trigger: use_rslora: true with finetuning_type: full or freeze.

Common situations: Users migrating a LoRA recipe (with rsLoRA for high-rank stability) to full fine-tuning and leaving the flag; or combining rsLoRA with GaLore-style full training.

Related errors

• `reward_model_type` cannot be lora for Freeze/Full PPO train
• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use PiSSA for current training stage.
• `loraplus_lr_ratio` is only valid for LoRA training.
• `use_dora` is only valid for LoRA training.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/28817687eddd8f4b. Report an issue: GitHub.