hiyouga/LlamaFactory · error · ValueError

KTransformers does not support lora reward model.

Error message

KTransformers does not support lora reward model.

What it means

For PPO with a LoRA-based reward model (finetuning_args.reward_model_type == 'lora'), the KTransformers backend (model_args.use_kt) cannot serve the LoRA reward head, so the combination is rejected. KTransformers offloads layers to CPU/GPU in a way that does not support attaching the lora RM adapter.

Source

Thrown at src/llamafactory/hparams/parser.py:458

        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Use a merged full reward model: export the LoRA RM (llamafactory-cli export) and set reward_model to the merged path with reward_model_type: full, keeping use_kt.
2. Or disable KTransformers (use_kt: false) to use the LoRA reward model directly — requires enough GPU memory.
3. Retrain the reward model with finetuning_type: full for a native full RM checkpoint.

Example fix

# before
stage: ppo
reward_model: saves/rm_lora
reward_model_type: lora
use_kt: true

# after (merge RM LoRA first via export, then)
stage: ppo
reward_model: saves/rm_merged_full
reward_model_type: full
use_kt: true

if cfg.get("stage") == "ppo" and cfg.get("reward_model_type") == "lora" and cfg.get("use_kt"):
    raise SystemExit("KTransformers cannot serve a LoRA reward model; merge the RM LoRA via export and use reward_model_type: full")

When it happens

Trigger: stage: ppo with reward_model: <path>, reward_model_type: lora and use_kt: true in model_args; raised during argument checking before model load.

Common situations: Running PPO on large models (e.g. 70B) with KTransformers offloading while pointing at a LoRA reward model checkpoint saved from RM training; copying a kt chat config into a ppo config.

Related errors

• `reward_model_type` cannot be lora for Freeze/Full PPO train
• Unsloth does not support lora reward model.
• `reward_model` is necessary for PPO training.
• `reward_model_type` cannot be oft for Freeze/Full PPO traini
• Cannot use PiSSA for current training stage.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/62193bf80b45b961. Report an issue: GitHub.