hiyouga/LlamaFactory · error · ValueError

Unsloth does not support lora reward model.

Error message

Unsloth does not support lora reward model.

What it means

For PPO with a LoRA reward model (reward_model_type == 'lora'), the Unsloth acceleration backend (model_args.use_unsloth) is unsupported — Unsloth patches the model for training the policy and cannot attach the separate LoRA reward adapter. The parser rejects the combination up front.

Solutions

1. Merge the LoRA reward model into its base (llamafactory-cli export) and use reward_model_type: full with the merged path, keeping use_unsloth for the policy.
2. Or set use_unsloth: false and run standard LoRA PPO with the LoRA reward model.
3. Retrain the RM with full finetuning for a checkpoint that works in either setup.

Example fix

# before
stage: ppo
reward_model: saves/rm_lora
reward_model_type: lora
use_unsloth: true

# after
stage: ppo
reward_model: saves/rm_merged_full
reward_model_type: full
use_unsloth: true

if cfg.get("stage") == "ppo" and cfg.get("reward_model_type") == "lora" and cfg.get("use_unsloth"):
    raise SystemExit("Unsloth cannot serve a LoRA reward model; merge the RM LoRA via export or disable use_unsloth")

When it happens

Trigger: stage: ppo together with reward_model_type: lora and use_unsloth: true in model_args, passed to llamafactory-cli train / run_exp().

Common situations: Enabling Unsloth for memory-efficient single-GPU PPO while reusing a LoRA reward model from an RM run; 'enable all accelerators' configs that set use_unsloth unconditionally.

Related errors

• KTransformers does not support lora reward model.
• `reward_model_type` cannot be lora for Freeze/Full PPO…
• Cannot use PiSSA for current training stage.
• `reward_model` is necessary for PPO training.
• `reward_model_type` cannot be oft for Freeze/Full PPO…

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/2b2fd0b4c8f836e5. Report an issue: GitHub.

Appendix: source

Thrown at src/llamafactory/hparams/parser.py:461

            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

View on GitHub (pinned to f28afaf635)