hiyouga/LlamaFactory · error · ValueError

Unsloth does not support lora reward model.

Error message

Unsloth does not support lora reward model.

What it means

For PPO with a LoRA reward model (reward_model_type == 'lora'), the Unsloth acceleration backend (model_args.use_unsloth) is unsupported — Unsloth patches the model for training the policy and cannot attach the separate LoRA reward adapter. The parser rejects the combination up front.

Source

Thrown at src/llamafactory/hparams/parser.py:461

            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Merge the LoRA reward model into its base (llamafactory-cli export) and use reward_model_type: full with the merged path, keeping use_unsloth for the policy.
2. Or set use_unsloth: false and run standard LoRA PPO with the LoRA reward model.
3. Retrain the RM with full finetuning for a checkpoint that works in either setup.

Example fix

# before
stage: ppo
reward_model: saves/rm_lora
reward_model_type: lora
use_unsloth: true

# after
stage: ppo
reward_model: saves/rm_merged_full
reward_model_type: full
use_unsloth: true

if cfg.get("stage") == "ppo" and cfg.get("reward_model_type") == "lora" and cfg.get("use_unsloth"):
    raise SystemExit("Unsloth cannot serve a LoRA reward model; merge the RM LoRA via export or disable use_unsloth")

When it happens

Trigger: stage: ppo together with reward_model_type: lora and use_unsloth: true in model_args, passed to llamafactory-cli train / run_exp().

Common situations: Enabling Unsloth for memory-efficient single-GPU PPO while reusing a LoRA reward model from an RM run; 'enable all accelerators' configs that set use_unsloth unconditionally.

Related errors

• `reward_model_type` cannot be lora for Freeze/Full PPO train
• KTransformers does not support lora reward model.
• `reward_model` is necessary for PPO training.
• `reward_model_type` cannot be oft for Freeze/Full PPO traini
• Cannot use PiSSA for current training stage.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/2b2fd0b4c8f836e5. Report an issue: GitHub.